Robert Abela won the Labour Party leadership contest with 57.9% of the vote, reports say.
His predecessor Mr Muscat said he would quit last month over the scandal surrounding the murder of Daphne Caruana Galizia.
She was killed by a car bomb in 2017 as she investigated corruption in Malta.
Mr Muscat's handling of the murder inquiry caused widespread anger. He was accused of protecting allies implicated in the investigation, which he denies.
His replacement - the son of Malta's former President George Abela - will be sworn in on Monday.
Early counts suggest Mr Abela beat his closest rival Chris Fearne, a 56-year-old surgeon, by more than 2,500 votes.
Around 17,500 Labour Party members were expected to cast their ballots in the leadership contest.
In his victory speech, Mr Abela, a 42-year-old lawyer, called on his party to "work together for unity", Malta Today reported.
Seen as a continuity candidate, Mr Abela avoided criticising Mr Muscat or his handling of the inquiry into Caruana Galizia's death during the campaign.
BBC
More about: Malta
-1780923278.jpg&h=190&w=280&zc=1&q=100)
