The 29-year-old British citizen, originally from Sudan, has also been arrested on suspicion of attempted murder, Met Police said.
He came to the UK as a refugee and was granted asylum, the BBC understands.
His brother described him as a "normal person" with no fanatical ideas and no links to any religious group.
Abdullah Khater also said his family - who are originally from Darfur in Sudan - were in "a state of shock".
Magistrates have granted police a warrant of further detention enabling them to question Mr Khater until 20 August.
Three people were injured when the car hit cyclists and pedestrians during Tuesday morning's rush hour.
The silver Ford Fiesta then crashed into security barriers outside the Houses of Parliament just before 07:40 BST.
A man and a woman were taken to hospital after the crash and later discharged, while another man was treated for minor injuries at the scene.
The suspect is not believed to have been known to MI5 or counter-terrorism police, but is understood to have been known to local police.
He did not co-operate with officers after his arrest, Scotland Yard said. The investigation team's priority "continues to be to understand the motivation behind this incident", a spokesman added.
Police have concluded searches of two properties in Birmingham and one in Nottingham, and are currently searching a third address in Birmingham.
BBC home affairs correspondent Daniel Sandford said the reasons for the crash remained "very open" nearly 36 hours on.
He said while detectives were holding Mr Khater on suspicion of terror offences, they had not yet formally declared the crash a terrorist incident, suggesting they had not yet found anything to link him to extremism.
What else is known about the suspect?
Mr Khater came to Britain in 2010 and successfully applied for UK citizenship in the past two years.
His brother said he had been planning a trip back home to see his family, having not seen them for some time.
Mr Khater is believed to have lived in a first-floor flat above a parade of shops in the Sparkbrook area of Birmingham until four months ago, when he moved to the city's Highgate area.
Neighbours called him a quiet man who frequently visited an internet cafe and a nearby shisha lounge, both on Stratford Road.
Local resident Ahmed Abdi described him as a man who "never spoke to anybody", and added: "I recognised his picture from the news and I was shocked."
Having completed his secondary education in Sudan, he studied electrical engineering at university there before moving to the UK.
He studied English as a second or foreign language at City College from 2010 to 2011, later studying a diploma in science at South and City College in Birmingham from 2014 to 2017.
More recently, Mr Khater studied accountancy at Coventry University from September 2017 to May this year, a spokesman for the university said. He is no longer enrolled there.
Members of the Sudanese community have suggested Mr Khater may have had an appointment in London at the Sudan embassy to obtain a visa for his trip.
Birmingham Central Mosque said its inquiries with the community found he did not worship at the mosque and had showed no signs of radicalisation.
In a CV seen by the BBC, Mr Khater described himself as a "hardworking, honest and polite individual" and "mature and responsible, with the ability to stay calm in high pressured situations".
It says he worked in Libya for two years as a farm labourer before coming to the UK. The BBC understands Mr Khater did spend time in Libya before travelling to Britain, but has not confirmed the exact timeframe.
The CV also suggests he spent time as a security guard in Nottingham.
It is understood Mr Khater did obtain a licence from the Security Industry Authority (SIA), although the organisation said it could not comment due to the "ongoing police investigation".
What happened on Tuesday?
A number of witnesses said the car, which was travelling westbound in Parliament Square, appeared to deliberately hit members of the public as it swerved into the opposite lane.
Footage released by the BBC showed the moment of the crash. A police officer can be seen jumping over a barrier to get out of the way.
A passenger in the first car behind the cyclists said they had been "thrown everywhere" after being hit at what she estimated to be 25mph.
The car was driven to London from Birmingham the night before, arriving just after midnight on Tuesday.
It then spent nearly five hours in the Tottenham Court Road area before being driven around Westminster for more than 90 minutes before the crash.
Police have been able to trace the route taken thanks to automatic number plate recognition cameras.
They have released new images of the Fiesta and appealed to anyone who may have seen the car or its driver in the hours leading up to the crash to come forward.
Mr Khater was detained at the scene under the Terrorism Act and taken to a south London police station. No-one else was in the car when it crashed and no weapons were found in it.
Police are treating it as a terror-related incident "given that this appears to be a deliberate act, the method, and this being an iconic site", Scotland Yard's head of counter-terrorism Neil Basu said.
No other arrests have been made, Scotland Yard said.
More about: UK