The prime minister failed to win support for her EU strategy after the European Research Group (ERG), led by Jacob Rees-Mogg, abstained on a government motion because it appeared to rule out a no-deal Brexit.

The defeat marks the end of a temporary truce over Brexit among Conservative MPs, who had voted last month in favour of May’s strategy if she could obtain some concessions from Brussels on the issue of the Northern Ireland backstop.

The prime minister was not present for the House of Commons defeat, by 303 votes to 258, in which she again lost control of her party in the crucial final weeks before Britain is due to leave the EU on 29 March.

It is likely to cause fresh doubt in Brussels that May has the power to win parliament’s support for an amended EU withdrawal agreement. The Eurosceptic refusal to cooperate with No 10 underlined the difficulty the prime minister is likely to face in securing concessions from the EU that will satisfy them.

In Brussels, diplomats said the result confirmed that the prime minister was incapable of commanding the support of her party on key votes, and that she needed to work cross-party. “No one can take any good from this,” said one diplomat.

The vote is not binding but it appeared to be a show of strength by around 60 MPs in the ERG, which included Boris Johnson, the former foreign secretary; Dominic Raab, the former Brexit secretary; and Steve Baker, the former Brexit minister. A small number of pro-EU MPs also refused to back the motion.

Senior figures in the ERG used the result to increase pressure on the government to adopt the so-called Malthouse compromise – a proposal to use unspecified technology to avoid customs checks at the Irish border – which the EU appears likely to reject. Many in the ERG would be equally happy to see a no-deal Brexit.

No 10 played down the significance of the vote and insisted that May understood the concerns of the ERG. However, that appeared only to infuriate many remain-supporting Tories who are determined to block a no-deal Brexit.

The government offered a separate concession to remain-supporting Tory MP Anna Soubry that it would publish some papers relating to the impact of a no-deal Brexit. However, one ERG source said the group was unconcerned about this because the public paid no attention to “project fear” at the referendum.

Nick Boles, a former government minister, said the vote should be a wake-up call to May that she cannot rely on the ERG’s support.

“Maybe, just maybe, the penny will now drop with prime minister and her chief whip that the hardliners in the ERG want a no-deal Brexit and will stop at nothing to get it,” he said. “Responsible MPs of all parties must come together on 27 and 28 March and stop them.”

In an escalation of tensions, Richard Harrington, one of May’s business ministers, even suggested MPs in the ERG should join Ukip.

“The prime minister has done a pretty good job of standing up to them up till now, but they were drinking champagne to celebrate her losing her deal and I regard that as being treachery,” he told the House magazine.

“I read that Nigel Farage is setting up a new party called ‘Brexit’ and if I were them I’d be looking at that, because that seems to reflect their views more than the Conservative party does. They should read carefully what that party’s got to offer, because in my view they’re not Conservatives.

“There are people who are very solid and stringent in their views and if I were they I would be looking at a party that seems designed for them – Nigel Farage’s party.”

He also dismissed the Malthouse compromise as “fanciful nonsense”.

May had previously said that she had “a substantial and sustainable majority” among MPs for her approach, after parliament voted in January for her to seek “alternative arrangements” to the Northern Ireland backstopfrom the EU and separately against a no-deal Brexit.

The government motion simply restated parliament’s position, but the hardline Eurosceptics balked at the idea of endorsing a strategy that takes a no-deal Brexit off the table. MPs will next get a chance to debate and vote on May’s EU strategy on 27 February.

The Labour leader, Jeremy Corbyn, demanded that May come to the House of Commons to explain her Brexit plan in the absence of parliamentary support for her approach.

“Tonight’s vote shows there is no majority for the prime minister’s course of action in dealing with Brexit,” he said. “Yet again her government has been defeated. The government cannot keen on ignoring parliament or ploughing on towards 29 March without a coherent plan.”

No 10, however, simply issued a statement saying that May understood the concerns of her Conservative colleagues. Downing Street had always insisted that it never intended to remove the option of pursing a no deal Brexit.

“While we didn’t secure the support of the Commons this evening, the prime minister continues to believe, and the debate itself indicated, that far from objecting to securing changes to the backstop that will allow us to leave with a deal, there was a concern from some Conservative colleagues about taking no deal off the table at this stage,” the statement said.

“The motion on 29 January remains the only one the House of Commons has passed expressing what it does want, and that is legally binding changes to address concerns about the backstop. The government will continue to pursue this with the EU to ensure we leave on time on 29 March.”

May also attempted to shift some blame on to Labour, saying its failure to support her motion made the prospect of leaving the EU without a deal more likely.

Labour sources said panicking Conservative whips had discussed the possibility with them of accepting a Labour motion and voting with the opposition in order to avoid an embarrassing defeat. Ultimately, however, they decided to accept a humiliating loss rather than appear to join forces with the opposition.

Labour also suffered a split as 41 backbench rebels voted with the SNP to delay Brexit. Those defying the party whip included Chuka Umunna, Chris Leslie, David Lammy, Luciana Berger and Margaret Hodge. They were joined by two pro-EU Conservatives – Ken Clarke and Sarah Wollaston.

More about: #TheresaMay