The court was hearing a case brought by opposition groups who wanted parliament to be compelled to begin impeachment.
It relates to Mr Zuma's use of state funds to upgrade his private home.
Handing down the Constitutional Court ruling, Judge Chris Jafta said: "We conclude that the assembly did not hold the president to account.
"The assembly must put in place a mechanism that could be used for the removal of the president from office."
But the court said it could not intervene on how parliament determined the mechanism and that it had no power to order an impeachment
The court awarded costs against Mr Zuma and parliament.
The court ruling was by majority. Dissenting Chief Justice Mogoeng Mogoeng said the ruling was a clear case of "judicial overreach".
Some $15m (£11.1m) in state money was spent upgrading Mr Zuma's home.
In March 2016, the court ruled he had ignored a watchdog's findings and said he must pay some of the money back. The president has reimbursed $631,000, deemed by the Treasury to be "reasonable".
Parliament then debated the matter and he survived a no-confidence vote, but Friday's court ruling said that the procedures it followed were insufficient.
Mr Zuma has been weakened by a number of corruption allegations and by his recent replacement by Cyril Ramaphosa as head of the ruling ANC.
Mr Zuma, 75, is scheduled to remain president until general elections in 2019.
He has faced a number of corruption allegations, all of which he denies.
Earlier this month Mr Zuma lost two court cases in one day, both linked to corruption allegations.
More about: #Zuma
