With less than 10 weeks until the Tokyo Games begin and as Japan battles a surge in infections, public opinion remains strongly opposed to the event going ahead this summer.
But Olympic officials say it can be safely held with Covid-19 countermeasures and point to successful test events, including some featuring overseas athletes.
An association of around 6,000 doctors working in Tokyo said they were "struggling with the fourth wave" of virus cases, calling it the largest so far.
"Cancelling an event that has the potential to increase the number of infections and deaths is the right choice," the group said in a statement on Monday.
It urged the government and Games organisers to consult the International Olympic Committee with the aim of cancelling the event.
The statement came after a separate union of Japanese hospital doctors warned last week that holding the 2020 Games safely was "impossible".
Japan's virus outbreak has been relatively small, with around 11,500 deaths, but its vaccine rollout is moving slowly and the latest spike in cases has medics warning of shortages and burnout.
The government on Friday expanded a virus state of emergency that will be in force until the end of May, less than two months before the Games open on July 23.
Tokyo 2020 organisers said Tuesday that nearly double the required number of certified sports medics had applied to volunteer at competition venues and in the athletes' village.
