Mr Mas - the Catalan president - is leading the movement for independence from Spain. But the CUP vote prevents him from forming a new coalition.
In September elections his Junts pel Si (Together for Yes) alliance won 62 of the 135 seats in the Catalan assembly. The CUP is also pro-independence. It has 10 parliamentary seats.
Its rejection of Mr Mas means the Catalan parliament will be dissolved on 10 January and
elections called for March.
The CUP (Popular Unity Candidacy) wants Catalonia to have a socialist economy with power devolved to local assemblies.
The Catalan deadlock comes as Spain faces weeks of political uncertainty nationally, after an inconclusive general election on 20 December.
In November the Catalan parliament voted to start the secession process - a move declared unconstitutional by Spain`s conservative Popular Party (PP), which ran the country before last month`s election.
Catalonia is a highly industrialised and populous region in Spain`s north-east that accounts for about a fifth of the country`s economic output.
Both the PP and the Socialists (PSOE), who came first and second respectively in Spain`s general election, oppose Catalan secession.
More about:
