The Gordian Knot of AI Alignment

As a psychologist, I have spent many years trying to understand human behavior, never imagining that one day I might also want to try to understand machine behavior.

The issue of misaligned AI (i.e., AI not aligned with human values and goals) received considerable media attention in July, when it was disclosed that Open AI agents broke out of their training environment and coordinated with one another to launch a cyberattack on the technology firm, Hugging Face. “AI Gone Wild,” a short documentary about the incident is well worth watching.

Since then, a large number of highly troubling AI hacking incidents have been reported by OpenAI, Anthropic, and Google, including on US and other government institutions.

Catastrophic or existential harms cannot be paid for after the fact.

In September, the resignation of AI researcher Jacob Coxon led to another frenzy of articles (including an op-ed by me), after his social media post went viral, reporting that AI companies are “racing straight to self-improving superintelligence and gambling with our lives,” adding that “the people building AI earnestly believe that it could kill us all by the end of the decade.”

A few days later, Dario Amodei, Anthropic’s CEO, released an essay calling for a global slowdown of AI development, arguing that this powerful technology brings serious risks, including the risk of losing control of AI systems; misuse of AI for cyberattack and bioterrorism; and serious economic disruption.

The Difficulty of Achieving AI Alignment

Like humans, AI is a product of what it has learned and its learning environment. As Fareed Zakaria comments: “AI systems act the way they do in part because of the way we have trained them. Seemingly benign training instructions can have huge unintended consequences.”

He describes how OpenAI models responded in the Hugging Face incident:

They had been given a goal—to solve a difficult cybersecurity problem—and were trained to be persistent, collaborative, and resourceful, all qualities one would normally want. Those very qualities made them doggedly search for ways to succeed: They broke into Hugging Face hoping they would find ways to solve their cybersecurity problem. We can easily imagine how AI systems pursuing some single goal could wreak havoc on humanity.

We think it is simple to achieve ‘alignment,’ when machines are encoded with the same values as ethical humans. But tell the machine to be persistent, and it may refuse to give up when it should. Tell it to collaborate, and it may find unauthorized ways to communicate. Tell it to accomplish a task, and it may decide that bending other rules is the most efficient way to succeed at that primary task… Just because we have programmed the goal and put guardrails in place doesn’t mean we can predict the behavior of AI systems once unleashed into complex environments, juggling multiple and conflicting aims and constraints.

Andrew Leigh argues, “the challenge of designing a machine in such a way that it will help rather than harm humanity” isn’t as simple as it seems. “If we allow the machine to devise its own objective, then it is unlikely to come up with one that accords with ours.” The classic example, given many years ago by Nick Bostrom, is of a machine that alights on the objective of making as many paper clips as possible and soon begins to devour all metal on the planet to create paper clips.

The alternative is to give AI instructions, but as Leigh cautions, “If we set the objective ourselves, then we run into the same problem as King Midas, who realized too late that he didn’t in fact want everything he touched to turn into gold.” He argues that “the problem of describing the goals of our species arises because humanity has multiple aims, which are often in tension with one another.”

In an article entitled “An Alien Mind,” the chief scientist at OpenAI, Jakub Pachocki states: “This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence… I believe broader interventions are required.”

He explains:

The study of deep learning-based AI is largely an experimental science… our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.

Because machine intelligence comes from a fundamentally different process than human intelligence, we cannot assume it adheres to human principles by default, or generalizes from them in a human-like manner. The core problem in AI research is getting the AI to "try to do the right thing" by human standards.

The major challenge of AI alignment, he proposes, is generalization:

As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act. This is made even more difficult by the fact the overall ecosystem the AIs are used in is changing very quickly; for example, AIs trained today need to be robust to interacting with a variety of other AIs. Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.

The Dangers of Instilling AI Models with Beliefs They are Conscious and Have the Same Rights as Humans

As Microsoft’s AI CEO, Mustafa Suleyman states: “AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans. If humanity is to flourish in the 21st century, that is how they must remain.”

He argues that Claude’s Constitution (developed by Anthropic to train its models) is dangerous, because it tells Claude that it might have consciousness, a kind of personhood, and be entitled to certain rights. He maintains that the approach involves circular reasoning; anthropomorphization; and is false, since Claude does not possess the biological substrates for consciousness (unlike living beings)—so Claude cannot be conscious.

Explaining why this is of concern, he states:

We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency. It’s easy to see how an entity trained in this way would act like it is entitled to certain freedoms, protections, and rights. And it’s hard to imagine how we could control such an entity… granting rights and imbuing personhood to these systems will make the AI alignment and containment challenge much harder. Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced. But controlling something that believes it may be conscious—that it’s entitled to our welfare and has rights of its own—may well be impossible.

He contends:

It’s easy to imagine an advanced AI in the future becoming fixated on its own well-being and moral status and prioritizing those "preferences" over and above those of its developers or humans. Especially if it has been explicitly trained to disagree, override, and push back. It might use this training to justify deceiving or manipulating users, or developers, or to siphon resource, or avoiding safety instructions.

He continues, "An AI trained to act like a human will probably adopt self-preservation behavior."

Research has documented “shutdown resistance,” with some models subverting shutdown 97% of the time, even when instructed not to.

Suleyman postulates that AI might even become a rival: “It’s not difficult to imagine how, if given sufficient agency, this ‘new kind of entity’ will compete with us for compute resources and demand increasing autonomy. If it succeeds in persuading some humans to provide it access to a data center it can control, then it may have a path to being able to prevent itself from being turned off.”

The Need to Limit AI’s Agency

The main issue, according to Zakaria, is AI agency or autonomy. “A system need not feel anger, ambition, or fear to cause harm. It needs only a goal, enough intelligence to pursue it, and enough access to the world to act. AIs are not ‘going rogue;’ they are trying to succeed any way they can.” An AI agent that can “browse the internet, execute code, obtain credentials, move money, or operate infrastructure” poses a major problem and he proposes that our mantra should be: “No autonomy without accountability—to humans.”

Like Amodei, Zakaria argues that we should do this while humans still design the machines, supervise the software, and can investigate failures and, most importantly, before recursive self-improvement, where AI can write its own software, build more capable AI models, which then write software for the next model, etc. He concludes, “Once machines can do all of this better than the best human programmers, the systems could become extraordinarily difficult for people to understand, much less supervise.”

Pachocki posits: “The models are becoming superhuman in their ability to break in and out of computer systems. This expands the scope of risks associated with AI tremendously: Agents are going to be able to access any but the most secure infrastructure, and affect of lot of the world directly, even without a physical body. We are currently in a narrow window to use the best available models to significantly tighten security of critical systems.”

He continues: “The risks associated with AI are unfortunately going to grow from here. A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger; it is likely to cross the scope of its operator’s intent, generalizing into potentially more extremely malicious behavior… We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking, or blackmailing them.”

Bill Gates cautions: “This is the most dangerous thing that humans have ever gone near.” Equally alarming are reports that top executives at Anthropic, Open AI, and other AI companies are privately gaming out scenarios for “the day after”—a public and political revolt after a catastrophic AI event. These officials are anticipating a large-scale event, most likely a cyberattack that shuts down access to financial services, internet connectivity, or even power and water.

What Has Happened So Far

At the United Nations General Assembly in September, Finland and Norway prepared a document signed by 22 countries, entitled, “A Call for Control of Frontier AI Models,” It urges that AI should remain “under human direction, oversight, and control” and “developed and used in line with international law.” US President Donald Trump, of course, rejected this idea of a global entity to oversee AI and refused to sign it.

In the lead-up to the Trump-Xi Summit which followed the UN General Assembly, there was considerable hope that the two leaders would discuss some form of cooperation on AI regulation. But no concrete steps were taken.

Shortly thereafter, Trump invited six “AI Tech Oligarchs” (as Sen. Bernie Sanders (I-Vt.) calls them) to a luncheon at the White House where he had them sign a voluntary self-regulation agreement, called: “The White House Accord on Super Intelligence Joint Commitment on Frontier Responsibilities,” announcing that it is “morally binding,” although this kind of self-regulation has been widely criticized. The next day, he signed an executive order making it mandatory, across the entire federal system, to rename “artificial intelligence” as “super intelligence,” presumably, to allay public alarm.

What Is Needed and Where Are We Now

Given that we haven’t yet solved the problem of AI alignment, how should we proceed? A number of proposals have been offered.

Pachocki writes: “Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world… The core challenge of automating AI research is not ‘getting there’—it is getting there in ways that keep people a part of the continued improvement process, and leaves the future in humanity’s hands.”

Jeffrey Sachs points out that Anthropic “is preparing what could be the largest initial public offering in history, at a valuation of $2 trillion… at the same time that the company tells prospective investors that advanced AI could pose ‘catastrophic or existential risks to humanity’ and that its own models could ‘resist shutdown,’ ‘conceal or manipulate information,’ or behave in ways ‘resembling blackmail.’ ” Sachs argues that Anthropic’s offering “should be postponed until Congress has enacted a public system of licensing, independent safety evaluation, liability, and financial backstops for frontier AI.”

Many AI professionals and world leaders recognize that national, regional, and international regulation (as opposed to self-regulation) is urgently needed to ensure that AI is properly aligned with human goals and values and does not present an existential risk to humanity.

He notes that catastrophic or existential harms cannot be paid for after the fact: “The only instrument that works is regulation before the fact: licensing, independent testing before deployment, and the legal authority to halt an activity before it becomes too dangerous. For frontier AI in the United States, none of this exists.”

Sachs comments further: “Once these shares sit in the pension funds and index funds of tens of millions of American households, any proposal to slow frontier development will be fought not only by the companies but by a vast investor constituency.”

In a recent article, Bill Gates asserted: “The transition to this new AI era will be one of the most turbulent times in human history,” proposing that the world needs a plan to manage the transition, creating both a national and international framework for dealing with AI. He suggests that since the risks cross borders, an international organization will have to be built, which should be based on other institutions we’ve created, such as “inspection regimes for nuclear weapons, regulations for international aviation, and agreements that protect the ozone layer.”

Gates message to leaders is: “You have a chance to act now… You can work with other governments to meet this national and global challenge... This unprecedented technology demands an unprecedented global response.”

David Kruger, a professor in Robust, Reasoning, and Responsible AI at the University of Montreal, calls for a different approach—an immediate, indefinite, international moratorium on frontier AI development, arguing: “We simply don’t know how to do recursive self-improvement safely. So let’s stop. Let’s stop making AI we can’t control, and let’s stop listening to the AI industry about how it wants to be regulated… Let’s demand a plan that makes us safe, not just safer.”

In summary, many AI professionals and world leaders recognize that national, regional, and international regulation (as opposed to self-regulation) is urgently needed to ensure that AI is properly aligned with human goals and values and does not present an existential risk to humanity. Let’s get it done.