From the outset, Harvard historian Calder Walton argues in a new book, Western leaders only belatedly recognized the vulnerabilities of their open societies and the magnitude of the threat posed by Soviet intelligence. Are they making the same mistake with China?

Studies of the role of intelligence operations inevitably have a Rashomon-like quality, with the same events yielding various, sometimes contradictory, interpretations. The world of intelligence, after all, is one of secrets, special-access compartments, covert action, clandestine relationships, and occasionally off-the-books escapades. This makes it very difficult to assess successes and failures, and to chronicle the role of intelligence in political leaders’ decisions.

Harvard historian Calder Walton confronts this challenge head on in a new book, Spies: The Epic Intelligence War Between East and West, which recounts the rise and role of modern intelligence capabilities through the history of the West’s competition with the Russian security services. It is an ambitious and entertaining story, but one that is also firmly grounded in academic research. In fact, Walton’s account sheds new light on seemingly well-studied events, from the Bolshevik Revolution and World War II to the deployment of moles in the ranks of American, British, and Russian intelligence services at the end of the century.

Walton draws on newly opened archives, formerly classified in-house histories, memoirs, and interviews with policymakers and spies. He thus illuminates how intelligence contributed to episodes like the 1962 Cuban Missile Crisis and the 1983 Able Archer incident, when a NATO drill triggered Soviet fears of a Western first strike, bringing the world to the brink of nuclear war.

But Walton does more than add previously secret details to old accounts. In an example of “applied history,” he uses his examination of the past to weigh in on current events, such as the Ukraine war, and raise important questions about the future, including whether Western intelligence can prevail against China in the new cold war.

Spies details the rise and role of Russia’s intelligence services, from Lenin’s Cheka to Stalin’s KGB to their post-Soviet incarnations. It offers new insights into their domestic and international missions, from the murderous role of the NKVD, the KGB’s predecessor, in Stalin’s Great Purge to the penetration of the Manhattan Project by Soviet spies gathering secrets vital to Moscow’s nuclear breakthrough. And it describes how the SVR and the FSB, the KGB’s successors, still use a Soviet-era playbook to guide their espionage, disinformation, and covert action abroad and repression at home.

Thanks to President Vladimir Putin – a former KGB man himself – KGB alumni dominate the Russian elite, including its corrupt economic oligarchy, and lead the powerful coercive institutions that are transforming Russia into an authoritarian security state. Whatever their labels, Russia’s security services have formed the backbone of its ruling regimes in the twentieth and twenty-first centuries.

But, as Walton demonstrates, Russia’s intelligence services have always had significant deficiencies, rooted not least in a totalitarian culture that has made it impossible to speak truth to power. From the deadly purges that reached deep into the NKVD in the 1930s to the top-down decision-making in Putin’s Kremlin, the message to the rank and file has remained simple: say what your leader wants to hear, because unvarnished reporting and analysis can have deadly – not just career-ending – consequences.

According to Walton, the crippling conformity that characterized Soviet intelligence assessments meant that they played at most a marginal role in the Kremlin’s Cold War decision-making. The disconnect between approved narratives and reality has also led to signature strategic failures, such as Stalin’s rejection of intelligence warning of Hitler’s impending invasion in 1941, and Putin’s disastrous decision to invade Ukraine in 2022.

Walton’s historical perspective also provides a valuable framework for assessing the performance of Russia’s Western counterparts. From the outset, he argues, Western leaders only belatedly recognized the vulnerabilities of their open societies and the magnitude of the threat posed by Soviet intelligence.

In fact, US intelligence got off to a rocky start. When the Central Intelligence Agency was created in 1947, Chairman of the Joint Chiefs of Staff Omar Bradley was angry that he was not put in charge. So, he teamed up with FBI Director J. Edgar Hoover, himself no fan of the intelligence start-up, to keep the CIA’s director, Roscoe Hillenkoetter, in the dark. Hillenkoetter was thus not informed of the top-secret Venona project or the revelations of far-reaching Soviet spying, before, during, and after WWII, that the program’s cryptologists produced. This US intelligence “mess,” as Walton aptly calls it, lasted until 1952.

Of course, bureaucratic rivalries and political infighting continued, even as the US intelligence community matured. But popular ambivalence has also affected the trajectory of America’s intelligence services, owing to disagreements about their role in a democracy, together with decades of controversy, from failed covert-action programs and proxy wars abroad to abuses of power at home.

As a changing world transforms the security landscape, national leaders will need public cooperation. But deep polarization and growing popular distrust in government, exemplified by (but not confined to) those who believe former President Donald Trump’s claims about the mythical “Deep State,” pose significant hurdles and threaten America’s democracy and international role.

While Walton acknowledges this danger, he is far more concerned about the threat China poses in an increasingly technology-driven world. Here, Walton doesn’t mince words. US leaders, he contends, have ignored China’s massive, multi-front intelligence offensive, underfunding collection and counterintelligence, and underplaying the economic, technological, and strategic challenge the country poses. “If I were to situate where we…are (regarding China) today,” he writes, “I would place us at approximately the year 1947.”

That said, Walton’s case for action on China lacks the depth of analysis that buttresses his judgments about Russia. And his recommendations for future intelligence efforts – based on “lessons” from the last 100 years – are thought-provoking, though less than persuasive. “The age of the secret service is over,” he writes in his conclusion. “In a world that will be increasingly dominated by open-source information, the “future of intelligence lies in the private sector, not with governments.” Maybe so. But Walton’s faith in technology as the intelligence world’s silver bullet, and in the power of persuasion to enlist corporations that demonstrate daily their reluctance to leave the Chinese market, merit skepticism.

As Walton notes, history may not repeat itself, but it does, as Mark Twain quipped, sometimes rhyme. Whether high-tech tools and open-source collection or classic espionage will deliver what countries need remains to be seen. But one thing is certain: in any democratic society, popular and political support for intelligence is vital. The success of any intelligence service begins at home.

Kent Harrington, a former senior CIA analyst, served as national intelligence officer for East Asia, chief of station in Asia, and the CIA’s director of public affairs.

More about: