The boycott, driven by Washington and including some of its allies, has become another delicate issue for Japan, a close partner of the United States that also has strong economic ties to China.
Tokyo will not send a government delegation to the Games set to start in February, Chief Cabinet Secretary Hirokazu Matsuno told a news briefing.
Instead, it will send some officials with direct ties to the Games, including politician Seiko Hashimoto, head of the Tokyo 2020 organising committee, as well as the heads of the domestic Olympic and Paralympic committees.
The diplomatic boycott by the United States and other countries stems from concerns about human rights in China, but Japan has taken a softer tone on the issue.
Prime Minister Fumio Kishida has faced rising pressure within his ruling Liberal Democratic Party to take a tougher stance on China, public broadcaster NHK said.
More about: Japan