Toback has denied the allegations and said he had never met any of the women in question or, if he did, it "was for five minutes and have no recollection".
The writer and director was nominated for best screenplay for 1991's Bugsy.
The mobster film starred Warren Beatty and Annette Bening. He has also directed Robert Downey Jr in three films, including Black and White and The Pick-Up Artist.
His latest film, The Private Life of a Modern Woman, premiered at the Venice Film Festival last month and stars Sienna Miller.
'Biologically impossible'
The women interviewed by the LA Times have accused Toback, 72, of masturbating in front of them, rubbing himself up against them, asking inappropriate questions of a sexual nature and asking them to perform sexual acts.
One said that after an alleged encounter, she "felt like a prostitute, an utter disappointment to myself, my parents, my friends", adding: "And I deserved not to tell anyone."
Toback told the LA Times that for the past 22 years, it had been "biologically impossible" for him to engage in the behaviour described, saying he had diabetes and a heart condition that required medication.
LA Times writer Glenn Whipp, who broke the story, tweeted that since the story was published on Sunday, the number of women who originally contacted him - 38 - had doubled.
More about: #JamesToback
