Spacey, who was late oil tycoon Jean Paul Getty in the film, will be replaced by Christopher Plummer.
The claims against Spacey span from the mid-1980s to last year.
All the Money in the World, directed by Ridley Scott, is about the 1973 kidnapping of Getty's teenage grandson.
Variety said that Spacey had shot about two weeks of footage and there were many scenes in the film where he was the only actor on screen.
He already appears in the film's trailer, which was released in September.
Mark Wahlberg and Michelle Williams, who also star in the film, are expected to take part in the reshooting.
The movie has also been withdrawn from the American Film Institute's (AFI) annual festival in Los Angeles later this month.
Actress Valentina Violo, who plays a secretary in Ridley Scott's film, said she had met Spacey at a party held at the end of shooting this summer.
"He was very nice and kind and very human," she told BBC News. "He was enjoying himself drinking, eating, speaking with people."
Spacey's career has nosedived since the first allegation of sexual advances were made by actor Anthony Rapp on 30 October.
US network Netflix has axed further production of Mr Spacey's House of Cards drama, the International Academy of Television Arts and Sciences announced it would no longer give the actor a special Emmy award, and his agent and publicist dropped him as a client.
Rapp accused Spacey of trying to seduce him when he was aged 14.
In response to Mr Rapp's claims, Mr Spacey said he had no memory of the incident and offered an apology.
Since then several others have come forward accusing him of predatory behaviour, including a woman who said Spacey had sexually assaulted her 18-year-old son last year.
Spacey's representatives say he is seeking unspecified treatment.
More about: #KevinSpacey
