Junichi Fukuda denies the accusations reported in a magazine which says it has evidence against him.
Separately, a local governor stepped down over allegations he paid women for relationships.
Japan's socially conservative society has so far been slow to join the worldwide #MeToo movement.
Sparked by the Harvey Weinstein scandal in Hollywood, the movement started a broad pushback against sexual harassment in many countries around the world.
The allegations that Mr Fukuda, the top bureaucrat in Japan's finance ministry, had made sexually suggestive comments to female journalists were first reported by magazine Shukan Shincho.
The publication also released an audio clip earlier this week which it says is the voice of the bureaucrat saying "Can I give you a hug?" and "Can I touch your breasts?".
'Protecting the ministry'
Mr Fukuda said he will fight the allegations against him and was resigning only because he did not want to disrupt work at the ministry which already "faces a severe situation".
Japan's finance ministry is under fire for allegedly having altered documents to shield Prime Minister Shinzo Abe from accusations of cronyism.
The ministry said it plans to appoint a law firm to investigate the accusations.
'Silent suffering'
Despite efforts to boost women in the workforce, Japan still has a big gender gap in politics and companies.
Prior to Mr Fukuda's resignation, Japan's newspaper employees union had issued a statement calling for better protection of women in the workplace.
"Female reporters have had to suffer silently, despite being subjected to humiliating and mortifying treatment," the union said.
"Many female reporters have had to put with sexually abusive language, having hands wrapped around their hips and shoulders, but being able only to take those hands and silently place them on their owner's knees, for fear of undermining the relationship between those that they report on and their organisations."
Political turmoil
In a separate case, Ryuichi Yoneyama, governor of the Niigata prefecture, resigned over media reports about his relations with multiple women, some of which involved money and presents.
"It was hard to find someone to date... I did give gifts and money to get attention," he told reporters at a news conference.
The governor explained his resignation saying those relations might "look to some like prostitution". Prostitution is illegal in Japan but prosecution is rare.
Mr Yoneyama, who is unmarried, said he was stepping down to avoid further political "turmoil".
BBC
More about: Japan