Only 23.9 percent of respondents in the three-day nationwide poll published Sunday said they wanted to see the 2020 Games held next year.
The survey carried out by Kyodo news agency found 36.4 percent of respondents back a further delay of the Games, while 33.7 think the flagship event should be cancelled altogether.
Most of those backing a delay or cancellation said they simply didn't believe the pandemic could be contained in time for the Games, now scheduled to open on July 23, 2021.
A separate survey carried out over the weekend by the Asahi Shimbun newspaper found 33 percent of respondents backed holding the Games next year, with 61 percent supporting another postponement or cancellation.
Tokyo 2020 was postponed in March as the coronavirus spread across the globe, marking the worst disruption to the Olympics since two editions were cancelled during World War II.
The nationwide polls echo a separate survey carried out last month of Tokyo residents, which found just over half of respondents backed either further delay or cancellation.
Japanese and Olympic officials have made clear that a further postponement is not on the table.
But concerns are rising in Japan about a new wave of infections, with the capital Tokyo registering record numbers of daily new cases in recent days.
International Olympic Committee president Thomas Bach said last week the Tokyo Olympics could be "a unique milestone for the entire world" noting they "will be the first worldwide gathering after coronavirus."
But Bach warned that the unprecedented health situation meant multiple scenarios were being considered in planning the format for Tokyo.
He said the scenario of holding the Games without spectators was one that had been examined, although he stressed that he was opposed to the idea.
The telephone survey conducted by Kyodo News obtained responses from 1,045 people.
The Asahi poll published over the weekend obtained 2,097 responses in a telephone survey on Saturday and Sunday.
