The lawmakers voted 448-197 in favor of a report recommending the launch of a so-called Article 7 procedure, which could lead to the suspension of Hungary's European Union voting rights.
It is the first time in EU history that the European Parliament had initiated and approved such a motion, which needed a two-thirds majority to pass and was approved by 69.4 percent of the lawmakers.
For years, Orban had been able to deflect much of the international condemnation thrown his way. Critics say that Hungary's electoral system is disproportionate; media freedoms and judicial independence are dwindling; asylum-seekers and refugees are mistreated and there are limits placed on non-governmental organizations.
Hungarian Foreign Minister Peter Szijjarto, echoing Orban's longtime position, called the Wednesday's vote "petty revenge" against Hungary for its tough anti-migration policies.
He also claimed that the vote involved "massive fraud" since abstentions weren't counted into the final tally, which made it easier to reach the needed majority.
There were 48 abstentions, so the 448 in favor exceeded the two-thirds needed only because it was based on 645 votes. If the abstentions were counted into the final tally, there would have been a total 693 votes, so the 448 in favor wouldn't have reached two-thirds.
Szijjarto said Hungary was considering legal options to appeal the result because of the way the vote was tallied.
But Judith Sargentini, who presented the report prepared by the European Parliament's Committee on Civil Liberties, Justice and Home Affairs, welcomed the outcome.
"Viktor Orban's government has been leading the charge against European values by silencing independent media, replacing critical judges, and putting academia on a leash," Sargentini said. "The Hungarian people deserve better. They deserve freedom of speech, non-discrimination, tolerance, justice and equality, all of which are enshrined in the European treaties."
"This is a historic result for Hungarian citizens and for European citizens everywhere, that the European Parliament has voted by a large majority to stand up for the values we all hold dear," Sargentini said.
The move saw some members of the European People's Party bloc — of which Orban's Fidesz movement is a member — vote against their ally in Budapest.
Even EPP leader Mandred Weber, who earlier was supportive of Orban and is seeking to become the European Commission president next year, said he had voted for triggering Article 7.
"I have always been in favor of building bridges and I want to continue to do so, but yesterday (Tuesday) I didn't see any readiness from the Hungarian PM to make a move towards his EU partners and address our concerns," Manfred tweeted.
While Weber had called on Orban to show a willingness to compromise on some of the most high-profile issues — like an agreement being delayed by the Hungarian government for the Central European University, founded by George Soros, Orban's ideological opponent, to remain in Budapest and recent laws criminalizing the work of civic groups working with asylum-seekers and refugees — Orban remained steadfast that his policies wouldn't change.
"I have nothing to compromise about since the questions they objected to were decided by the Hungarian people," Orban said Tuesday in Strasbourg, France, after the debate in the European Parliament on the report on Hungary. "There is nothing to talk about."
Orban, who was re-elected in April to his third consecutive term in office, fourth overall also said Tuesday that he expected lawmakers to approve the motion with the support of some EPP lawmakers.
"The order has arrived from Berlin and they will vote accordingly," Orban said, in reference to German Chancellor Angela Merkel, whose governing Christian Democratic Union is the largest party in the EPP.
Orban has insisted that all of the criticism against his government is based on Hungary's tough anti-immigration policies, which include fences built in 2015 on Hungary's southern borders with Serbian and Croatia to divert the flow of migrants and very restrictive asylum rules. He has also expressed his desire to remain within the EPP, which he said was "deeply divided" on the issue of migration.
The vote, however, has little chance of ending up with the ultimate penalty of Hungary being suspended from voting in the EU. If nothing else, its Polish ally would veto that.
The 197 votes cast against the parliament's first bid to launch the punitive process of the EU treaty's Article 7 also highlighted the substantial minority of European opinion who see Orban as a crusader for the rights of nation states and ethnic majorities against rules of civic behavior agreed in Brussels.
Since sweeping to power in 2010, Orban, once a campaigner against Hungary's Soviet Communist overlords, has used his parliamentary majority to pressure courts, media and non-government groups in ways his opponents say breach EU rules.
He has also led opposition to German Chancellor Angela Merkel and others who want Europe to take in more Muslim refugees.
With Britain about to leave the bloc altogether in March and Europeans voting in European Parliament elections in May, the row over Hungary -- and Poland, which faces a similar sanctions procedure launched by the executive European Commission in 2017 -- highlights tensions between nationalist and federalist camps.
All the applause for the Hungarian leader on Tuesday came from the far right of the chamber, notably among supporters of France's Marine Le Pen and the UK Independence Party.
Geert Wilders, the leader of the Dutch anti-immigration Freedom Party, called the vote - which also weighed on Hungarian assets - "a bloody shame".
"Orban is a hero and deserves the Nobel Prize. He closed the borders for Islamic fortune seekers. He protects his citizens against terror and defends the identity of his country," he said.
The Commission has preferred to pressure Budapest through standard legal powers, but the head of the EU executive, Jean-Claude Juncker, also an EPP member, said he would have voted for the move if he were a lawmaker.
More about: Hungary