Details of the talks were not revealed but party leaders are to hold an emergency meeting on Monday.
Mr Zuma, who faces corruption allegations, was replaced as ANC leader by Cyril Ramaphosa in December.
Analysts say party chiefs want to avoid a power struggle that could split the ANC before elections next year.
They are expected to begin the process to remove President Zuma through a formal recall or by introducing a motion in parliament.
The six most senior figures of the governing ANC party arrived one by one on Sunday at President Zuma's residence in Pretoria.
They remained tight-lipped as the talks ended but have called a meeting of the party's national working committee for Monday.
Julius Malema, an opposition leader and former ANC member, said on Twitter that Mr Zuma had refused to stand down.
Mr Zuma is not due to leave office until the 2019 elections. However, the ANC has seen its popularity fall during his second term amid a weakening economy and claims of corruption.
The president faces numerous allegations of corruption relating to his relationship with businessman Shabir Shaik who was tried and found guilty in 2005 of soliciting bribes from a French arms company.
Mr Zuma and other government officials were accused of taking kickbacks from the purchase of fighter jets, patrol boats and other arms.
Charges were first brought against Mr Zuma in 2005 but dropped by prosecutors in 2009.
Last year, the High Court ruled in a case brought by the opposition Democratic Alliance that he should face the charges.
Mr Zuma lodged a challenge with the Supreme Court of Appeal but it ruled he should face the charges. He has consistently denied any wrongdoing.
Mr Ramaphosa, who as the ANC's new leader is in a strong position to win the 2019 elections, has called for party unity.
But correspondents say fears are growing that he and his allies in the ANC may move against Mr Zuma if he refuses to stand aside.
More about: JacobZuma