South Korea's newly elected President Moon Jae-in, who campaigned on a platform of better engagement with the North, said it was a provocation.
US President Donald Trump has called for "stronger sanctions" against North Korea, while China is urging restraint.
A series of North Korean missile tests this year - which are banned by the UN - has sparked international alarm and raised tensions with the US.
Two missile launches last month both failed, with the rockets exploding just minutes into flight.
Is it a new type of missile?
The nature of the launch is still being determined, but analysts have said the test could suggest a longer range than previously tested devices.
The Japanese defence minister said it flew for about 30 minutes before falling in the Sea of Japan and could be a new type of missile, Reuters reported.
Tomomi Inada said it covered a distance of 700km (435 miles), reaching an altitude of more than 2,000km (1,245 miles) - higher than that reached by an intermediate-range missile North Korea fired in February.
The US Pacific Command said in a statement the type was being assessed but that its flight was not consistent with that of an intercontinental ballistic missile [ICBM], which would have the range to reach the US mainland (more than 6,000km).
North Korea is believed to be developing two types of ICBM, but neither has so far been flight tested.
If the Japanese analysis of the trajectory is right (that the missile reached an altitude of 2,000km), North Korea appears to have advanced its technology markedly.
The previous two tests failed, so reliability is not there yet. But last month some experts reckoned that a seemingly new missile on parade in Pyongyang may have been an ICBM (the type President Trump said "won't happen").
Jeffrey Lewis of the James Martin Center for Nonproliferation Studies in California thought at the time that the new missiles on show might be ICBMs. Is this that missile?
One thing is certain: North Korea will certainly trumpet its success if it does now have the capability to strike the US military bases on Guam, 3,400km from Pyongyang in the Western Pacific. Mr Trump would ponder what to do with even greater urgency.
What has been the reaction?
After hosting an emergency meeting of his security council, Mr Moon condemned the latest launch as a "provocation".
"The president said while South Korea remains open to the possibility of dialogue with North Korea, it is only possible when the North shows a change in attitude," his spokesman said.
Image caption South Koreans will be watching closely to see how their new leader's responds to the test
The White House said President Donald Trump "cannot imagine Russia is pleased" because the missile did not land far from Russian territory.
It added that the new launch should serve as a call for stronger sanctions against North Korea.
A Kremlin spokesperson later said Russian President Vladimir Putin was concerned by the test.
China, North Korea's only major ally, called for restraint by "all relevant parties" in the wake of the latest test.
How are US-North Korea relations?
The North has conducted five nuclear tests despite UN sanctions and is also developing long-range missiles.
It is reported to be continuing efforts to miniaturise nuclear warheads and fit them on missiles capable of reaching the US.
Washington has accused other UN Security Council members of not fully enforcing existing sanctions against the North, and has urged China in particular to use its trade links as influence.
But despite poor relations, North Korea recently said it would hold talks with the US "if the conditions were right".
The comments, by a senior North Korean diplomat, came after US President Donald Trump said he would be "honoured" to meet the North's leader, Kim Jong-un.
More about: #NorthKorea
