"The problem is that in many ways it was doomed to failure from the beginning, because there was never the preparatory work that has to be done prior to a summit meeting," Panetta said on ABC's "This Week" Sunday. "This was all about show, it was about shaking hands and exchanging words."

The underlying work on process and looking at nuclear weapons sites and determining inspection regimes, Panetta said, didn't happen as needed. Now, Panetta said he's "very worried" about the North Korea situation, "because frankly, I think we have a failed summit on our hands right now."

"There's a long list of failed efforts throughout history, this may be another one of those," said Panetta who served as defense secretary and CIA director under former President Barack Obama.

Mr. Trump has remained optimistic about the future of North Korea, after declaring North Korea no longer a nuclear threat following the summit. But even he has had to admit the progress has been insufficient. In July, the Washington Post reported U.S. intelligence agencies found that North Korea is building new missiles, based on satellite photos and other evidence.

"I have asked Secretary of State Mike Pompeo not to go to North Korea, at this time, because I feel we are not making sufficient progress with respect to the denuclearization of the Korean Peninsula," the president tweeted last month.

Trump grumbles that deal with North Korea isn't getting enough praise
It's unclear what the next steps are for North Korea, as the U.S. has not given any timeline for denuclearization. Pompeo said last month that North Korea would determine the timeline, which came as a surprise to some.

"The ultimate timeline for denuclearization will be set by Chairman Kim, at least in part," Pompeo said. "The decision is his. He made a commitment, and we're very hopeful that over the coming weeks and months we can make substantial progress towards that and put the North Korean people on a trajectory towards a brighter future very quickly."

Panetta also weighed in on the threat of impeachment, should Democrats win control of the House. He was White House chief of staff for President Bill Clinton during his impeachment proceedings, and on Sunday, he warned Democrats against pursuing impeachment, urging them to wait for special counsel Robert Mueller to complete his investigation into Russian election meddling and any ties to Trump associates. Mueller has not given a timeline for completing that investigation.

"I think Bob Mueller's report will ultimately determine whether or not there are going to be additional steps taken against the president," Panetta said, adding that Democrats "ought not to get ahead of that report because that will be the key to determining what happens in the future."

CBS news

More about: NorthKorea