But a Cabinet Office investigation in December cleared him of wrongdoing and found there was no evidence he had “breached the expected standards of behaviour”.
Mr Garnier said in a Tuesday morning Twitter post he was “very sad” to have lost his job, but was “looking forward” to supporting the Prime Minister from the backbenches.
The father-of-three faced a probe after his former secretary, Caroline Edmondson, told the Mail on Sunday he gave her money to buy two vibrators at a Soho sex shop.
Ms Edmondson claimed the Tory MP told her one of the sex toys was for his wife and the other was for a female assistant in his West Midlands constituency.
On another occasion, he was alleged to have told her: “You are going nowhere, sugar t*ts.”
Mr Garnier admitted the incidents took place, telling the Sunday paper, “I’m not going to be dishonest. I’m going to have to take it on the chin.”
But he denied the 2010 incident constituted sexual harassment, saying the sex toys were bought after a Christmas lunch and the “vibrator shop was high jinks”.
Mr Garnier said he and Ms Edmondson later fell out and that she had been using the incident against him “ever since”.
Mr Garnier’s sacking followed the exits of chairman Patrick McLoughlin and Education Secretary Justine Greening.
Critics said the Prime Minister's January reshuffle showed she lacked authority, with Ms Greening and Health Secretary Jeremy Hunt rebutting decisions to move them into new roles.
More about: #Mark-Garnier