The centre-right Christian Social Union (CSU) is set to lose its absolute majority in the state parliament.
The Greens look set to surge into second place and the anti-immigration AfD is on track to enter the state parliament for the first time.
The CSU has ruled Bavaria almost single-handedly since 1957.
However, it has lost support as opinion becomes polarised over issues like migration.
Party leader Horst Seehofer said it was "not a nice day", but added it was only "one side of the coin" as the vote gives "a clear mandate" allowing the CSU to form a new government.
Preliminary official results showed the CSU won about 37% of the vote, down 10 points on four years ago, with the left-leaning pro-immigration Greens just short of 18%.
The Alternative for Germany (AfD) came fourth with about 10%, behind a collective of independent candidates known as the Free Voters.
Mrs Merkel's national coalition partners, the centre-left Social Democrats (SPD), trailed in fifth with less than 10%, their vote halved.
There is speculation that it could pull out of the German federal government in an attempt to survive as a party.
SPD leader Andrea Nahles blamed the poor performance in Bavaria on squabbling within the coalition.
More about: Germany Angela-Merkel
