Announcing his resignation, Sir Michael said his conduct had "fallen short" of the required standards.
Mr Williamson has been replaced as chief whip by his former deputy, Julian Smith.
In the role Mr Smith will be in charge of enforcing party discipline in the Commons, and his deputy will be Tatton MP - and former GMTV presenter - Esther McVey.
BBC assistant political editor Norman Smith said Mrs May's decision to move her chief whip at such a difficult time had come as a surprise to many people.
But he said she had avoided "unsettling the equilibrium" and making a full reshuffle ahead of the 22 November Budget.
The appointments come after Sir Michael became the first politician to quit following recently revealed claims of sexual harassment in Parliament.
Gavin Williamson has no military background and has never been a secretary of state. But he is youthful, a rising star and trusted by Number 10.
He arrives at the Ministry of Defence at a difficult time. His first challenge will be to try to stave off more defence cuts. The Cabinet Office is currently carrying out a defence and security review which is due to report by the end of the year.
All three services have been asked to put forward options for cuts. Although there has been a modest rise in the MoD's budget, it has still got to make more than £20bn of efficiency savings.
There's also pressure to fund an increase in pay for the armed forces. They've been struggling with both recruitment and retention. That also hasn't been helped by a political reluctance to put troops in harm's way - something they train for.
There is still unfinished business against so-called Islamic State and Gavin Williamson will be the man who now overseas the RAF airstrikes. Few inside the military or the MoD will know much about him - even fewer on the international stage.
In his resignation letter, Sir Michael said a number of allegations that had surfaced about MPs, including himself, had been false, but added: "I accept that in the past I have fallen below the high standards that we require of the Armed Forces that I have the honour to represent."
He told the BBC that what had been "acceptable 15, 10 years ago is clearly not acceptable now".
He said it "was right" for him to resign, adding: "Parliament now has to look at itself and the prime minister has made very clear that conduct needs to be improved and we need to protect the staff of Westminster against any particular allegations of harassment."
When asked if he thought he should apologise, Sir Michael said: "I think we've all got to look back now at the past, there are always things you regret, you would have done differently."
His replacement, Mr Williamson, has been in Parliament since 2010, representing South Staffordshire.
He was a Parliamentary aide to former Prime Minister David Cameron and became chief whip after Mrs May became PM in July 2016.
More about: #DefenceMinister
