A video recorded at the scene shows several uniformed police detaining the suspect while an armed officer carrying a large gun patrols past.
It comes as London and the UK was placed on high-alert with extra armed police and soldiers deployed on the capital's streets.
A police spokesman said: "At about 10:40hrs on Wednesday, 24 May patrolling offices detained and searched a man in The Mall, SW1.
"A knife was recovered and the man was arrested on suspicion of a possession of an offensive weapon. He has been taken into custody at a central London police station."
The incident is not thought to be related to terrorism, a spokeswoman for the force told the Standard.
The arrest follows Monday night’s attack at an Ariana Grande concert in Manchester Arena when suicide bomber Salman Abedi, 22, killed 22 and left more than 100 people injured.
Children including eight-year-old Saffie Roussos and teenagers Georgina Callander and Olivia Campbell were among the victims killed.
Following the Manchester massacre, security officials raised the threat level to critical – the highest level which means an attack is imminent. Soldiers have been sent to key locations across the capital for extra security.
More about: #BuckinghamPalace
