The CDU won about a quarter of the vote in Baden-Württemberg and Rhineland-Palatinate, preliminary results show.
The two western states were once conservative strongholds.
Mrs Merkel is due to step down as German chancellor in September, after 16 years in office.
The slump in the CDU vote comes amid anger in Germany at the slow pace of the coronavirus vaccine rollout and a mask procurement scandal.
Several conservative lawmakers have quit over allegations they received huge commissions for arranging government deals to buy face masks.
