Parliament voted by 195 votes to 154 to reject the Speaker's plan for a coalition of the center-right Moderates and the small Christian Democrats, with Moderates leader Ulf Kristersson as prime minister.
"It's a pity. It's a lost opportunity to start tackling Sweden's problems," Kristersson told reporters after the vote.
Those opposing the coalition said it would give influence to the anti-immigrant Sweden Democrats — the third-largest party but considered a pariah by many — because the government would be dependent on that party's support in parliament.
Wednesday's vote was the first of a possible four before Speaker Andreas Norlen must call new elections. It was the first time that a candidate for prime minister has ever been rejected by the 349-seat Riksdagen.
Norlen said he would continue talks with party leaders on Thursday, without giving details.
The September election produced a hung parliament with the left-leaning side and the center-right bloc securing about 40 percent of the vote each, leaving neither with a majority and paving the way for months of uncertainty and complex coalition talks.
Both blocs have refused to cooperate with the Sweden Democrats, who made great strides in the election.
Annie Loof, the leader of the Center Party that voted against Kristersson, said he had broken up the 14-year-old center-right alliance by suggesting a two-party government, leaving out her party and the Liberals.
Loof said Wednesday her party did not oppose Kristersson but wanted to avoid giving the anti-immigration Sweden Democrats "a decisive say" over government policies.
Jan Bjorklund, leader of the smaller Liberal Party, voiced a similar view.
Kristersson said that that although the alliance has suffered "its worst crisis," it is not dead.
"We have a common platform, but we cannot agree when it comes to taking power."
Caretaker Prime Minister Stefan Lofven, who heads Sweden's largest party — the Social Democrats — said he was willing to discuss with Loof should she be tapped to try to form a coalition.
Loof earlier said that she doesn't want to be prime minister herself but is willing to try form a government.
Pundits say either Loof or Lofven would be next in line to try to form a government.
The emergence of Sweden Democrats — with roots in a neo-Nazi movement — reflects how deeply that Sweden is being transformed by migration.
In 2015, Sweden took in a record 163,000 migrants — the highest per capita of any European country. Since then, the country has sharply restricted immigration, but many Swedes complain that society cannot cope with integrating so many newcomers, many of them Muslims from Africa and the Middle East.
Ahead of Wednesday's vote, Jimmie Akesson, the leader of the Sweden Democrats, said he sees "a new political landscape mushroom in Sweden, exactly like in the rest of the western world."
Akesson told parliament that his party "expected concessions in proportion to our size" but would be "pragmatic."
His party is Sweden's third-largest with almost 18 percent of the vote, but other parties have ruled out working with the party.
Norlen observed it was "the first time that parliament has rejected a speaker's proposal for prime minister" since 1971 when Sweden introduced its current one-chamber parliament.
Liberal leader Bjorklund called for talks with the Greens, part of Lofven's outgoing government. But that has been rejected by the Greens as well as the Moderates, and a coalition of the Alliance and the Greens would still lack a parliamentary majority.
With no sign of compromise, a snap election is looking more likely. But that could easily return a similar result and simply delay forming a new government.
The impasse is likely to have negative effects the longer it lingers. A budget to be presented by acting Finance Minister Magdalena Andersson on Thursday will freeze new spending and is likely to hit some local authorities hard.
Analysts also say the deadlock could damage faith in the democratic process, potentially boosting the Sweden Democrats.
Even when a new government is in place, it will struggle to push through major policy initiatives, meaning urgent reforms, for example in the housing market, are likely to be delayed.
More about: Sweden