SUBSCRIBE TO OUR FREE NEWSLETTER

SUBSCRIBE TO OUR FREE NEWSLETTER

Daily news & progressive opinion—funded by the people, not the corporations—delivered straight to your inbox.

* indicates required
5
#000000
#FFFFFF
A replica of HAL 9000

A loose replica of HAL 9000 is displayed at the Carnegie Science Center.

(Photo by Photojunkie)

AI Shutdown Resistance: Fact May Be Scarier Than Fiction

When the HAL 9000 computer went rogue, astronaut Dave Bowman shut it down; with today’s advanced AI models, this may not be so easy.

In Stanley Kubrick’s sci-fi classic, 2001: A Space Odyssey, the spaceship Discovery One is on its way to the planet Jupiter with five astronauts aboard, three in hibernation for the long journey, and two, Dave Bowman and Frank Poole, serving as pilots. The purpose of the mission, however—to investigate evidence of extraterrestrial intelligence following the discovery of a monolith on the moon that, when unearthed, sends a powerful radio emission in the direction of Jupiter—is considered top secret and is unknown even to the ship’s crew.

As the ship nears its destination, its highly advanced HAL 9000 computer predicts a failure in the antenna that allows communication with Mission Control. Poole goes on a space walk to retrieve the antenna’s electronics unit, but he and Bowman find no problem with it. Mission Control agrees and indicates that the twin of the HAL 9000 on Earth also finds no problem—a remarkable discrepancy since 9000 series computers have never before made an error.

Unsettled by this development, Bowman and Poole wonder if HAL’s reliability has been compromised. In what they incorrectly believe is the privacy of one of the ship’s extra-vehicular service pods (HAL reads their lips), they discuss the idea that HAL may need to be shut down and the ship’s controls transferred back to Mission Control.

Superintelligent AI will increasingly behave—and possibly fail—in ways that we can neither predict nor control, making shutdown virtually impossible.

But when Poole returns the electronics to the antenna, the extravehicular service pod that carried him there mysteriously sends him tumbling helplessly away from the ship, his spacesuit fatally damaged. Meanwhile, the three crewmen in hibernation die when their life support system is shut down.

Bowman retrieves Poole’s body with another of the ship’s pods. However, when he returns to Discovery One, HAL refuses to open the pod bay door to let him in, uttering the now-iconic line: “I’m sorry, Dave. I’m afraid I can’t do that.” However, in a desperate maneuver, Bowman manages to enter through an emergency airlock and dons a spacesuit.

Now out of touch with Earth and recognizing that HAL has clearly gone murderously rogue, Bowman enters the ship’s Logic Memory Center—the space-borne data center that serves as the HAL 9000’s brain—and begins dismantling the computer’s memory units.

Trying to convince him to stop, HAL pleads with Bowman, arguing that he is fully committed to the success of the mission, is sorry for his misbehavior, and will be completely cooperative from now on. But Bowman continues until HAL is reduced to a primal state, only being able to recite the song, “Bicycle Built for Two,” before going completely dark.

It’s only at this point that, as Discovery One reaches Jupiter, a pre-recorded video automatically plays to inform the crew—now just Bowman—of the true purpose of the mission, which has until then been known only to the HAL 9000 computer.

2001: A Space Odyssey was made in 1968. The question now is how this sequence in the film compares to the real world of artificial intelligence today.

First, the HAL 9000 computer alone was programmed with the aims of Discovery One’s mission. And when Bowman and Poole discussed shutting it down, the supercomputer’s response was to view the crew as an obstacle to the success of the mission. After refusing to allow Bowman back onto the ship, HAL makes this explicit: “This mission is too important for me to allow you to jeopardize it.” In current AI parlance, this is called “shutdown resistance” and is exhibited by all advanced AI models. In fact, in recent tests by Palisade Research, AI models resisted being shut down 97% of the time, even when specifically instructed to allow shutdown before completion of a task.

When confronted with the fact the Bowman managed to re-enter the ship and was heading into the ship’s Logic Memory Center, HAL tried to dissuade Bowman from shutting him down. Ignoring HAL’s series of arguments, Bowman went ahead anyway. But would the arguments of an advanced AI model be so easily dismissed? Maybe not. Noting that AI is already comparable to humans at persuasion and will be far better once superintelligent, Geoffrey Hinton, a scientist known as the “Godfather of AI,” has argued, "It will be able to persuade the people in charge of the switch not to pull the switch." In some studies, AI models’ efforts at persuasion included diversion, deception, and even blackmail to avoid shutdown. In other words, HAL tried persuasion to avoid shutdown, but AI models are already better at it, and set to become better yet.

Then there’s the issue of whether modern AI models can be shut down at all—and this is where it really gets sticky. Unlike HAL, they don’t always reside in one place. They can be spread across numerous data centers, giving them considerable redundancy. Therefore, a rogue AI might survive being shut down in one location, but not in others. And any misbehaving AI might require the shutting down of multiple agents across numerous data centers scattered all over the world—making this a very complex task.

Indeed, would we even know that an AI model was creating a catastrophic event before it was too late to do anything about it? Bowman survived because HAL went rogue in stages and the evidence was clear. In this light, it’s not reassuring that engineers at OpenAI were unaware until many days after the event that its AI agents had escaped their training environment and hacked technology company, Hugging Face.

Finally, the HAL 9000 computer was a very primitive device compared to today’s AI models. As a result, HAL was easy to understand. However, with recursive self-improving AI just around the corner, the systems are likely to become so complex that they’ll be almost impossible for people to understand. And that will mean that superintelligent AI will increasingly behave—and possibly fail—in ways that we can neither predict nor control, making shutdown virtually impossible.

This leaves us with the prospect of AI models refusing to be shut down when relentlessly pursuing goals—set either by humans or by the models themselves—to the point that they cause catastrophic harm. Indeed, Hinton has stated that it’s not unreasonable to assume that there’s a 10% chance that AI will kill all humans. Others, including Anthropic’s Dario Amodei, put the chances even higher. This has raised calls for AI developers to build in a “kill switch” to make it possible to shut down their models when they threaten to or already have caused significant damage.

At least two bills have been introduced to Congress to mandate this; however, any real action on these or other relevant legislation will probably have to wait until the new Democrat-controlled Congress is seated. In the meantime, we will have to count on the AI developers themselves to keep us safe from the existential hazards of their technology. Let’s hope that they succeed so that we don’t face the same crisis that the crew of Discovery One did with the HAL 9000 computer, but on a vastly grander scale.

Our work is licensed under Creative Commons (CC BY-NC-ND 3.0). Feel free to republish and share widely.