Computer errors from outer space - iWONDER

  13 October 2022    Read: 1538
 Computer errors from outer space -  iWONDER

The Earth is subjected to a hail of subatomic particles from the Sun and beyond our solar system which could be the cause of glitches that afflict our phones and computers. And the risk is growing as microchip technology shrinks.

Zap. A muscle in her chest twitched. Zap. And again. Marie Moe could feel it. She could even see it. She looked down and the muscle, just to the left of her breastbone, was visibly pulsating. Convulsing with the rhythm of a vigorous heartbeat.

The cyber-security researcher was on a plane, about 20 minutes from its destination, Amsterdam, when it started. Fear gripped her. She knew immediately that something was wrong with her pacemaker, the small medical device implanted in her chest that used electrical impulses to steady her heartbeat.

Could one of the wires that connected the pacemaker to her heart have got damaged? Or come loose? Moe alerted the cabin crew, who at once arranged for an ambulance to be ready and waiting for her at the airport. Had the plane been any further from Amsterdam, the pilot would have made an emergency landing at another airport, she was told.

When Moe arrived at a nearby hospital, doctors pored over her. A pacemaker technician soon found the problem. It was the gadget's tiny computer. Data stored inside the pacemaker's computer, so crucial to its functioning, had somehow got corrupted.

And for Moe, the prime suspect that she says most likely sparked this unsettling episode was a cosmic ray from outer space: a chain of subatomic particles slamming into one another in the Earth's atmosphere, like balls colliding on a snooker table, with one eventually careering into her pacemaker's built-in computer mid-flight.

The theory is that, upon impact, it caused an electrical imbalance that altered the computer's memory – and ultimately changed her understanding of the life-saving technology inside her forever.

When computers go wrong, we tend to assume it's just some software hiccup, a bit of bad programming. But ionising radiation, including rays of protons blasted towards us by the sun, can also be the cause. These incidents, called single-event upsets, are rare and it can be impossible to be sure that cosmic rays were involved in a specific malfunction because they leave no trace behind them.

And yet they have been singled out as the possible culprits behind numerous extraordinary cases of computer failure. From a vote-counting machine that added thousands of non-existent votes to a candidate's tally, to a commercial airliner that suddenly dropped hundreds of feet mid-flight, injuring dozens of passengers.

As human society only becomes more dependent on digital technology, it's worth asking how big a risk cosmic rays pose to our way of life. Not least because, with the continuing miniaturisation of microchip technology, the charge required to corrupt data is getting smaller all the time, meaning it is actually getting easier for cosmic rays to have this effect.

Plus, since giant ejections from the sun can sometimes send huge waves of particles towards Earth, what's called space weather, an unnerving prospect looms: we could see much more disruption to computers than we're used to during a massive geomagnetic storm in the future.

Moe's frightening experience with her pacemaker happened in 2016. Once she was discharged from hospital, she received a detailed report from her pacemaker's manufacturer about what had happened. "That's where I learned about the bit flips," recalls Moe, who is now a senior consultant at cyber-security firm Mandiant.

Inside the pacemaker's computer memory, data is stored in the form of bits – often referred to as "ones and zeroes". But the report explained that some of these bits had reversed, or flipped, altering the data and causing a software error. Think of it like pressing the wrong end of the rocker in a long row of light switches. A part of the room will stay dark.

In this case, the error prompted the pacemaker to go into "backup programme mode", says Moe, and it began pacing her heart at a default 70 beats per minute with a heightened impulse. "That's what caused the very uncomfortable twitching," she explains.

In order to fix it, the pacemaker technician had to reset the device to factory settings in the hospital and these were later reconfigured appropriately to suit Moe's heart. But the report offered no definitive conclusions as to why those pivotal bits had flipped in the first place. One possibility mentioned, however, was cosmic radiation. "It's hard to be 100% sure," says Moe. "I don't have any other explanation to offer you."

That such a thing can happen has been understood since at least the 1970s, when researchers showed that radiation from outer space could affect the computers on satellites. This radiation can take various forms and originate from a number of different sources, both inside and outside our Solar System. But here's what one scenario might look like:  protons blasted towards Earth by the Sun smash into atoms in our atmosphere, releasing neutrons from the nuclei of those atoms. These high energy neutrons don't have a charge but they can go on to smash into other particles, triggering secondary radiation that does have a charge. Because bits in computer memory devices are sometimes stored as a tiny electrical charge, that secondary radiation now flying around can upend the bits, flipping them from one state to another, which changes the data.

Cosmic radiation increases in prevalence with altitude, mainly because our atmosphere helps to shield us from the majority of it. Air travellers, for example, are more exposed to this radiation than people on the ground, which is why air crews have limitations on the amount of time they can spend flying each month. But if this subatomic hurly-burly was behind Moe's pacemaker glitch, it must be an extraordinarily rare occurrence, she stresses.

"The benefit of having a pacemaker very much outweighs this risk," she adds. "I actually feel more confident trusting my device because I know it has this backup in case something goes wrong with the code."

But the impact of cosmic rays on other computers could, in theory, be disastrous. In one much-discussed incident, a 2008 Qantas Airways flight over Western Australia fell hundreds of feet twice within 10 minutes, injuring dozens of passengers on board – many of whom were not in their seats or buckled up at the time. Several suffered bruises to their limbs while others knocked their heads against the interior of the cabin, for example. One child who was wearing a seatbelt was jolted so badly that they suffered injuries to their abdomen.

An investigation by the Australian Transport Safety Bureau found that, prior to the erratic behaviour of the plane, erroneous computer data in the on-board systems had misrepresented the angle at which the aircraft was flying. This prompted the two automated nose-dives. As for what actually set off this chain of events, the report noted: "there was insufficient evidence available to determine whether [an ionising particle altering computer data] could have triggered the failure mode" – meaning that it remains a possibility. In contrast, all the other possible triggers considered by investigators were judged as "very unlikely" and one other as "unlikely".

There's also the case of the voting machine in Belgium in 2003 that gave a political candidate in an election 4,096 additional votes. Some have theorised that this, too, was the result of ionising radiation messing with a computer.

And what about the speedrunner – someone who tries to complete video games in record time – who experienced a weird glitch in Super Mario 64 back in 2013? To the gamer's surprise, Mario suddenly teleported upwards in the game, a behaviour later traced back to a flipped bit in the code that determines the position, in 3D, of the moustachioed character at any given time. Analysis uncovered little in the way of an explanation for this behaviour, dubbed an upwarp, and so the possibility of cosmic particles interfering with the game cartridge arose in discussions about the incident.

More recently in April 2022, Travis Long, a software engineer at Mozilla, posted a blog in which he explained that the huge swathes of telemetry data that the company routinely gathers from users of its Firefox web browser sometimes contain unexplained errors, on the order of individually flipped bits. Long noted that a recent bug associated with these tiny errors coincided with a geomagnetic storm.

"I started to really wonder if we could possibly have detected a cosmic event through these single-event upsets in our telemetry data," he wrote.

Whether ionising radiation is behind them or not, we can encounter flipped bits as we browse the internet. In 2010, a cyber-security researcher called Artem Dinaburg, who now works for the appropriately named firm Trail of Bits, realised this. He registered a handful of domain names that were similar to popular domains but with one incorrect character in the URL.

A bit flip isn't something that is itself visible to a computer user, though they might notice the consequences. A bit flip happens within the computer's memory and, in the processing of a URL, it could occur at various stages, such as when your computer requests a web page on the internet or when the web server to which you connect responds to that request.

Once Dinaburg had some bit-altered URLs registered, he just sat back and waited. "To my huge surprise, I started getting things connecting," he recalls. "In a lot of the world's computers, there are single bit errors, or sometimes multiple bit errors that happen, and if they happen at just the right place at just the right time, they can affect what domain your software is looking up."

The problem with all of the above examples is that there is no way to prove that a cosmic particle was behind any of them. And though some may lean towards that explanation, it can easily be challenged by more mundane theories. Dinaburg says computer memory bugs could be behind a lot of the connections he recorded in his experiment, for example.

And last year, the speedrunner who experienced the weird Super Mario glitch posted a video to YouTube of his game frozen mid-play.

The title of the video, "Was it really an ionising particle, though?" appeared to jokingly suggest that the speedrunning incident might have just been a random game glitch. A fellow speedrunner who uses the pseudonym pannenkoek2012 and who offered $1,000 (£900) to anyone who could explain why Mario teleported suddenly in the 2013 incident tells BBC Future, "I lean towards hardware malfunction" – rather than cosmic rays as the culprit.

In certain scenarios, there's enough data to indicate that radiation was behind multiple bit flips. To return to satellites, one group of researchers recently investigated more than 2,000 bit errors logged by a satellite over roughly two years in orbit. The team published the results of this work in 2020. The data errors were automatically corrected during the satellite's flight but, had they stayed in place, they would have misrepresented the vehicle's position.

By analysing the satellite's memory records, the researchers were able to plot when and where the errors occurred during its orbit. A huge number of the errors were clustered in an area called the South Atlantic Anomaly (SAA), where there is heightened cosmic radiation above the Earth's surface. It is well-known that this plays havoc with computer systems on satellites and spacecraft. According to Nasa, astronauts on the space shuttle used to notice that their laptops sometimes crashed when the space shuttle, now no longer in service, passed through the SAA.

But for single errors that occur more or less randomly on or nearer the ground, proving the involvement of cosmic rays is not easy. The slipperiness of the subatomic particles zooming all around us is not news to Paolo Rech at Trento University in Italy. "It's impossible to be conclusive. That is the fun part," he says, referring to incidents such as the Super Mario upwarp. And yet the possibility that such particles can cause tiny yet impactful data errors in computer systems is not in dispute, as Rech explains.

In lab experiments, he has some equipment that can accelerate neutrons artificially in order to point them at electronics and track the bit errors that the flow of particles induces. It's designed to emulate the neutron flux at ground level on Earth – but multiplied 100 million times.

"Rather than waiting months or years to discover an error, you can have errors in seconds or minutes," he says.

It's a way of studying the effects that single-event upsets could have out in the wild, just sped up for convenience. Rech and his colleagues have a specific goal in mind, though. With the rise of self-driving car technology, it's possible that computer systems on these vehicles could malfunction due to cosmic rays. What if, during an automated trip, imagery from a camera mounted at the front of the car became corrupted and the on-board computer failed to spot a person walking out in front of the vehicle?

By generating imagery with distortions that could conceivably be caused by cosmic rays, and using this to train artificial neural networks, Rech says he and colleagues have reduced the chances of such an error 10-fold. However, the research is yet to be published and he says he's not allowed to reveal what the starting level of accuracy was during the experiments.

Such interventions could make self-driving cars of the future safer but they wouldn't eliminate the possibility of a cosmic ray causing other problems. And this raises an interesting conundrum for insurers.

"In a world of fully autonomous vehicles, how can you prove the accident happened because of cosmic rays?" says Rech. "That is very challenging. I mean, it's impossible, by definition." In ambiguous cases, disputes over whether a human or technology manufacturer – or space weather – was at fault might be difficult to resolve.

One other point. Rech says it would, in principle, be possible for someone to try and induce bit errors in a computer system intentionally (and perhaps maliciously) by building a particle accelerator and aiming it at a computer's memory modules. It would be very difficult to actually do this effectively, however, he adds.

Natural sources of radiation remain the most important. And when it comes to cosmic rays, or space weather, it's important to make clear that it is just like Earth's weather – it varies. Sometimes big storms arise.

In early September 1859, the most intense geomagnetic storm ever recorded raged in the planet's atmosphere. The Carrington Event, named after British astronomer Richard Carrington, was caused by solar flares that flung huge quantities of subatomic particles towards Earth. The geomagnetic activity caused incredible displays of aurora borealis and induced charges in electrical wires. Some telegraph operators reported seeing sparks bursting out of their equipment.

If such an event were to occur in the future, it could theoretically damage power lines and internet cables across many regions, says Sangeetha Abdu Jyothi at the University of California, Irvine. "There is also this risk of charged particles causing data corruption," she adds. "Right now, the actual extent of damage, it's very difficult to predict."

Daniel Whiteson, also at the University of California, Irvine, agrees, adding that such an incident could potentially be "catastrophic" and that our understanding of the physics inside the Sun is not well-developed enough to allow us to be able to predict major solar ejections well in advance.

He and colleagues have proposed a method for gathering data from millions of smartphone cameras – which are sensitive to some subatomic particles – in order to detect instances of electromagnetic interference. That could help us better understand the prevalence and nature of cosmic rays that reach us here on Earth.

Separately, Michael Aspinall at Lancaster University in the UK and colleagues recently highlighted plans at the Royal Society's Summer Exhibition to build a neutron monitoring device in Great Britain. It would help to plug a gap in our ability to track the neutrons whizzing around us, he argues: "There's less than 50 of these ground-level neutron monitors still operational, none of them are in the UK."

The monitor would be built in either Scotland or Cornwall and if it detected a dangerous spike in neutron activity in the future, such information could be passed on to the UK Met Office, which might then advise aviation authorities to ground planes or take other precautionary action.

It's important to put all of this in context. Crucially, it is highly unlikely that cosmic rays are causing significant errors in computer systems on a regular basis. Data centre manager Tony Grayson, of Compass Datacenters in the US, says he has never felt the need to discuss the threat posed by radiation with colleagues in the industry. That's largely because small bit-level errors in data are often inconsequential, or corrected by automated error-checking software.

Going to great lengths to shield a data centre from cosmic rays, say by lining it with lead, would be eye-wateringly expensive. It's much easier and cheaper to just keep geographically distributed backups of data. If the worst happens, customers can be shifted over to the backup server, says Grayson.

But for some applications, cosmic rays are taken very seriously. Consider the pile of electronics in a modern plane that connects the pilot's controls to the rudder, for example. Tim Morin, technical fellow at semiconductor firm Microchip, says major aerospace and defence manufacturers use components that are resistant to certain cosmic ray effects. His company is among those that supply these components.

"It's just immune to single-event upsets caused by neutrons," he says. "We are not affected by that."

Morin declines to elaborate on exactly the approach his firm took to manufacture computer chips that are untroubled by neutron interference, except to say that it is to do with materials and circuit design.

Clearly, not every application requires such high-level protection. And it's also not possible to achieve this with every kind of computer memory, Morin adds. But for organisations that put planes and satellites above our heads, it is obviously an important consideration.

The technology upon which practically all of us now depend has varying levels of risk associated with it. But it's important to note that, as the transistors in computer chips get smaller in newer, more advanced semiconductors, they get more susceptible to electromagnetic interference, too.

"The charge needed to reverse a state is smaller," explains Rech. If only a very tiny charge is required, the chances of a subatomic particle inducing such a charge go up, in principle. Plus, there are growing numbers of computer chips out there, in devices from phones to washing machines. "The overall area that can be corrupted is actually significantly increasing," says Rech. The subatomic rain falling down on our devices has ever more targets to strike.

The consequences of that could conceivably be dire but, so far, it's hard to known to what extent this could harm us or the systems that power the modern world. For Marie Moe, the strange behaviour of her pacemaker on that flight to Amsterdam six years ago led to a heightened knowledge of the device that is so important for the healthy functioning of her heart. It even aided her research into the cyber-security vulnerabilities of pacemakers.

If a stray neutron really was behind it all, that's quite a chain reaction. So at least there can be positive outcomes from bit flips, as well as scary ones.

"I'm really happy, actually," she says, "that this happened to me."



More about:

News Line