The Tory peer said the ex-foreign secretary had also “let the genie out of the bottle” after he suggested veiled Muslim women resembled “letter boxes” and “bank robbers”.
He told BBC Newsnight he had received dozens of “vile“ emails, with “obscene language” since calling for the Conservatives to withdraw the whip from Mr Johnson.
His remarks came after the Conservative Party’s decision to consider disciplinary proceedings against Mr Johnson – a move that has led to criticism of Theresa May from Brexiteer MPs who claim the complaints are politically driven.
One backbencher told The Independent the handling of the affair by the prime minister had been so poor that it has increased the chances of a leadership challenge later this year.
The internal process was automatically triggered on Thursday after party HQ received several complaints about Mr Johnson’s remarks and a party spokesperson said the process “is strictly confidential”.
Despite criticism from across the political spectrum for the remarks Mr Johnson won the unexpected support from the comedian Rowan Atkinson – the star of Blackadder and Mr Bean.
“You should really only apologise for a bad joke. On that basis, no apology is required,” Mr Atkinson wrote in a letter to The Times.
Former cabinet minister Iain Duncan Smith said he was “surprised” to hear about the action being taken against Mr Johnson, adding: “I don’t agree that there was anything particularly wrong with what he said – you may not agree with the tone or the jokes.
“But we have a thing called freedom of speech in this country and I don’t believe that just because somebody takes offence that means therefore there has to be an inquiry into whether the individual can be shut down in terms of their ability to say what they believe.”
Asked about Mr Atkinson’s intervention, Mr Duncan Smith replied: “I rather agree with what he said.”
The former Tory chief whip, Andrew Mitchell, also told BBC Newsnight that there was no need for the ex-foreign secretary to apologise for his remarks in a column for the Daily Telegraph on Monday.
But I think what's important is that the procedures of the party now take place,” he said. “This is quite an important issue about free speech and it's got nothing to do with the dreadful events that take place over Enoch Powell and the Rivers of Blood speech.
”Boris was speaking out against a ban of the type that's taken place in Denmark and some other countries.”
Mr Johnson, who is currently on vacation, has not made any personal response to the demands for him to apologise but sources close to him made clear earlier this week that he stands by the article.
More about: Johnson