In space, no one can hear you kernel panic (2020)

When you’re millions of miles from home, it’s hard to install an operating system update—but not impossible. From the dawn of the Space Age through the present, NASA has relied on resilient software running on redundant hardware to make up for physical defects, wear and tear, sudden failures, or even the effects of cosmic rays on equipment.

The software architecture of space missions must be robust without being rigid to “deal with the kinds of uncertainties that arise in the context of space,” says David Garlan, a founder of the field of software architecture and a distinguished visiting scientist at the NASA Jet Propulsion Lab. Garlan, who is also an associate dean and professor at Carnegie Mellon’s School of Computer Science, says spacecraft systems, in particular, need a fault-protection layer that allows them to switch to emergency protocols without immediate intervention from Earth. But he also believes spacecraft should be designed with more autonomy in normal operations so they can achieve broader scientific goals without a constant guiding hand on the controls by scientists.

This makes for an architecture that might dismay terrestrial developers, since the software must avoid becoming bogged down in tasks. A data center server’s slow performance can be fixed by throwing more servers at it—virtual or real. The computational power on a spacecraft remains static for the mission’s duration, and systems must be designed to dump any given task without warning. A database server won’t melt down and bring adjoining racks with it if it can’t insert a row in real time, but a craft hurtling toward Mars might miss the planet completely if its cycles aren’t perfectly managed.

A craft’s software is also made more resilient by doubling—or quadrupling—down on replication and physical backups. For NASA’s Space Shuttle Program, which ran from 1972 to 2011, three or four computers weren’t enough: Shuttles had five flight computers, and planners considered a sixth. “Once you get humans on board, you’re in a whole different game,” Garlan explains: The tolerance for risk is minuscule.

Though running identical software on multiple computer systems is the name of the software-architecture game across crewed and uncrewed missions, satellites, probes, landers, and rovers, they take different approaches to dealing with errors, updates, and detection. NASA’s obsessive focus on software testing to find and remove bugs, plus a strategy to allow software to recover in the worst of circumstances, is one approach that has paid off repeatedly. First implemented midway through the Apollo program, which ran from 1961 to 1972, the strategy was designed explicitly for when things go wrong. Without it, mission after mission would have had to be abandoned or would have only achieved a fraction of its goals.

During its 1977 launch, for example, the NASA space probe Voyager 2 couldn’t interpret all the shaking it recorded during launch; mission scientists hadn’t anticipated how its sensors would read that activity. But the probe correctly knocked itself into a recovery mode and restored itself. This provided insight that allowed scientists to update the code for its sibling, Voyager 1, in time for its reverse-order launch 16 days later. Apollo 11 would have had to abort the first moon landing had its software not been designed for instant, continuous recovery. The Mars rover Opportunity would have seen a premature end to its much-extended life if a cable short hadn’t been bypassed by rewriting which measurements were gathered for movement. “(Launched in 2003, Opportunity was initially supposed to stay active for 90 Martian days; it remained operational until mid-2018, for a total of 5,111 days.) The Mars rover Curiosity might have failed during its first and fifth years on the planet after experiencing glitches in its two main computers. (Launched in 2011, Curiosity was initially supposed to stay active for 687 Martian days; it remains active today at over 2,700 days and counting as of this writing.)

Sometimes redundancies also add opportunities. We would have far fewer pictures from Voyager 2’s pass by Uranus, in 1986, and Neptune, in 1989, without a duplicate set of computers standing by, coupled with mission control’s ability to upload new software to take advantage of them. The inertial measurement units of the Mars Odyssey, launched in 2001, and the Mars Reconnaissance Orbiter, launched in 2005—which allow the two satellites to determine their absolute position in the universe while also noting changes in rotation, orbital speed, and other parameters—are nearing the end of their utility in the coming months (or years). Currently, they help the orbiters remain in the right spot and elevation above Mars, and keep their antennae pointed back at Earth. When these sensors fail, it would mean an end to the orbiters’ ability to collect and transmit data back to Earth—were it not for flexibility built into the software. Scientists are testing new software that will eventually allow both satellites to analyze their position from a star-tracking camera on board each craft, affording them additional years of usable life.

Expect the unexpected

Anyone who has managed physical or virtual servers under load understands the trade-offs between keeping a handful of critical machines running with a few minutes of downtime a year and having failover solutions that allow any link in a chain or any parallel task to have its slack picked up by another system. It usually boils down to cost and criticality: Can you afford to have that extra capacity? What’s the worst thing that happens if it fails briefly? A company’s website typically doesn’t freeze forever or crash into a planet if it’s down for a few minutes.

Still, earthbound computing hardware has evolved from monolithic business mainframes to redundant arrays of powerful servers that allow for the failure of one or more of them without breaking the business. NASA presaged this move decades before the rest of humanity out of necessity: In space, having more computers running duplicate functions, capable of weathering catastrophe, worked far better than having a monolithic system for which failure wasn’t a possibility. This has also proved, over time, to be the right course. Perfect software, perfect hardware will still break in space—a backup (or two or three or five) defeats Murphy’s Law and cosmic rays.

But it wasn’t always that way. In the 1988 book Computers in Spaceflight, commissioned by NASA, author James E. Tomayko notes that during the Apollo program, the agency focused on ensuring every component and system was tested until it was determined to be perfect. This, however, resulted in a process that was both expensive and brittle.

Later missions introduced a variety of architectures, which were still subjected to relentless testing to eliminate bugs but which allowed for a recovery or standby mode when failures occurred for any number of reasons, such as a hardware module breaking in flight or radiation damage.

A pair of famous examples illustrates the problem with relying on perfection. Software engineer Margaret Hamilton, the director of the MIT group that developed Apollo’s software in the late 1960s, had a hand in both. Hamilton frequently brought her daughter Lauren, then a toddler, to the office on late nights and weekends. Before the Apollo 8 mission in late 1968, which would mark the first time astronauts circled the moon, Lauren was playing with the command module simulator via a DSKY, a keyboard and display combination. She managed to crash a flight simulation by unexpectedly triggering a prelaunch sequence.

Hamilton tried to get NASA to let her introduce error checking to prevent an astronaut from making the same mistake during the mission, however unlikely. NASA overruled her, insisting astronauts would perform the task perfectly. Hamilton was reduced to putting a note in the manual about the possibility o