Keizo Okamoto, 60, a former senior member of a crime syndicate, and Hiroya Suemori, 67, a former investment adviser, had been sentenced to death in 2004 for a robbery-murder.
They stole about 100 million yen ($900,800) from the president of an investment advisory company in 1988 before killing him and another employee. They then packed the bodies into concrete.
"It is an extremely brutal case in which they robbed the precious life of the victims, indeed for selfish reasons," Justice Minister Takashi Yamashita said on Thursday.
Yamashita signed the order for their execution on Tuesday.
Executions are done in secret in Japan, with no advance warning given to the prisoner, their family or legal representatives, according to Amnesty International.
Prisoners often only learn of their impending execution a matter of hours before it takes place.
Capital punishment is usually reserved for those who have committed multiple murders. All executions are done by hanging.
Japan's Code of Criminal Procedure states the death penalty should be implemented within six months of the issuing of the sentence, but in fact that is almost never the case.
The controversial system made international headlines in July when 13 members of the Aum Shinrikyo cult, which carried out the deadly 1995 sarin gas attack on the Tokyo subway, were executed.
A total of 36 prisoners have now been executed since Prime Minister Shinzo Abe took office in December 2012, and places Japan at odds with the growing international trend towards abolition of the death penalty.
Japan and the US are the only two developed democracies to have capital punishment. About 170 states have either abolished or put a stay on executions since the United Nations General Assembly called for a universal moratorium on the death penalty in 2007.
More about: Japan