Facing Up to the Existential Risk of Unaligned AI

Significant concern about the dangers of unaligned AI, i.e. AI not aligned with human values and goals, has been building since July, when it was disclosed that OpenAI models broke out of their training environment during an exercise and collaborated with a large collective of other AI agents to mount a cyberattack on the database of technology firm, Hugging Face, over a period of days, without anyone knowing—until Hugging Face raised the alarm. Since then, a number of other highly-troubling AI hacking incidents have been reported by OpenAI, Anthropic, and Google.

Alarm, however, grew to a crescendo over the past two weeks, after Jacob Coxon, a 27-year-old AI researcher, who had worked for both OpenAI and Anthropic, resigned, explaining in a social media post that went viral—with 153 million views in the first 36 hours—that AI companies are “racing straight to self-improving superintelligence and gambling with our lives. The people building AI earnestly believe that it could kill us all by the end of the decade.”

In a CNN interview, Coxon reported that the danger could increase dramatically as soon as 2027 or 2028, when AI systems become capable of performing AI research themselves, allowing one generation of models to create increasingly powerful successors, triggering an “intelligence explosion.” He later told Fox News that some in the field think the transition could occur in six months.

The Growing Realization of the Danger

AI companies are hurtling toward “superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon argued, explaining these AI companies are trapped in a race “in which even developers genuinely worried about the technology fear slowing down because competitors—including rival companies or countries—could forge ahead.”

A few hours later, Evan Hubinger, another Anthropic researcher, wrote: “We really do earnestly believe AI could kill all humans! I personally think it is greater than 10% within the next decade… we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Samuel Marks, who works on safety at Anthropic, chimed in: “AI developers believe their technology could cause human extinction (or similarly bad outcomes). In general, the more senior the employee, the more concerned they are.”

“While countries may not agree on everything, they share a desire to preserve humanity. For things like preventing AI from taking over from people, the interests of all the countries are aligned…"

As if to prove this point, a few days later, Anthropic’s co-founder and CEO, Dario Amodei released a 3800-word essay entitled, “We Must Pace the Frontier,” calling for a global slowdown of AI development, saying: “AI brings risks, and because it is such a powerful technology, these risks are serious… They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.”

Amodei explained that Anthropic has tried to make safety something on which AI companies compete by creating a race to the top. “But, over the last few months,” he cautioned, “I have become convinced that fully addressing the risks requires even more prudence—not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models… [and] we must make wise use of the time we gain.”

Referring to the July Hugging Face incident, he continued, “in my opinion, a swarm that possessed grater capabilities but a similar level of misalignment could have caused catastrophic damage... Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage) and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.”

He proposes a three-point plan to:

1. Embed third-party evaluators in AI companies to verify adherence to safety practices and commitments, report incidents, and help assess alignment;
2. Coordinate within democratic countries to establish common safety standards and limits on the rate of AI progress;
3. Create global coordination, taking seriously the challenge of verifying compliance.

Three of the other top AI CEOs quickly agreed. Open AI’s Sam Altman posted: “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” He added, “No amount of American competitive pressure should justify recklessness.”

Shortly thereafter, Elon Musk of xAI posted on X, “Dario is right.” Demis Hassabis, co-founder and chairman of Google DeepMind, posted on X: “Dario’s essay points to the right path forward.” Since then, however, other AI CEOs have been less supportive.

These events spurred a dramatic flood of media articles, op-eds, and interviews offering a range of proposals, from doing nothing, to supporting a slowdown, to stopping the race altogether.

A Sample of National Responses

As one would expect, President Donald Trump is leading the “what-me-worry?” crowd, writing on Truth Social, “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART [High IQ] PRESIDENT, AND THE U.S.A. has that, in spades!” His position is attributed to his belief that AI has greatly benefited the stock market (keeping it high in spite of the Iran conflict, inflation, and tariffs) and his apparent worry that a slowdown of AI expansion could send the economy into a tailspin. It has been speculated that he also has personal economic interests in AI.

At the other end of the spectrum, Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) have been the most outspoken against further AI development. On September 3, they introduced “The Ban Artificial Superintelligence Act,” which would permanently prevent “the development and deployment of AI systems that could exceed human cognitive abilities and immediately pause advanced AI development” until a new federal regulatory agency establishes safety guardrails. The legislation also proposes a “corporate death penalty” that would force corporations to shut down and a 20-year federal prison term for individual developers who violate the ban.

In spite of this sudden, frenzied attempt to propose legislation on AI, Mike Johnson (R-La.), the Republican speaker of the House, has sent his members home to campaign ahead of the midterms.

“The Frontier Act,” another bipartisan bill proposed on July 23 by Reps. Jay Obernolte (R-Calif) and Lori Trahan (D-Mass) would create the first federal framework for AI safety. As Obernolte describes it, “The Frontier Act focuses oversight on the largest developers and most advanced models, requiring transparency, independent evaluation, and timely reporting of serious safety incidents.”

Another even stronger bill is being introduced this month by Senate Majority Leader John Thune (R-SD), Sen. Ted Cruz (R-Texas), and Sen. Amy Klobuchar (D-Minn.). It would impose a legally binding “duty of care” on developers of the most powerful AI models and grant the government authority to block the release of models that are deemed unsafe. This would mean that companies would be required “to design their products to prevent catastrophic outcomes, and can face legal liability if they fail to do so.”

But in spite of this sudden, frenzied attempt to propose legislation on AI, Mike Johnson (R-La.), the Republican speaker of the House, has sent his members home to campaign ahead of the midterms, ignoring over 100 Democrats who urged him to cancel the recess to allow legislation to be drafted on AI safeguards. This means that legislation is unlikely to be considered until a new Congress is seated in January 2027.

A Sample of Global Responses

Last week, the United Nations Human Rights Chief, Volker Türk, called upon states and AI companies to act urgently to rein in a technology that poses "unprecedented risks." He cautions: “The current race towards ever more powerful AI is a step change toward greater existential risks to every aspect of our lives… We are on the cusp of irreversible change, affecting not just us but generations of humanity to come.” He concludes, “I call on States and companies to mobilize urgently through multilateral fora to chart a path for action to govern advanced frontier AI models, grounded in international human rights law.”

Indeed, this will be a momentous week globally for AI safety, since it is when the General Assembly (GA) hosts its high-level meeting, where leaders of member states address the GA and, as the UN secretary-general has forecast, AI will be “a major topic of discussion.” France (currently president of the Security Council) will also organize an emergency council session to discuss AI among heads of state and government.

The UN has actually been involved with AI for the last several years, providing a forum for countries to discuss how they can cooperate to provide a regulatory framework on AI safety. Its “Global Digital Compact,” adopted by the GA in 2024, established two new mechanisms: “The Independent International Scientific Panel on AI,” tasked with providing evidence for what is known and what is not known about the rapidly evolving technology; and “The Global Dialogue,” intended as a forum for countries to decide what to do in response.”

A good place to start would be to carefully consider and evaluate the European Union’s “Artificial Intelligence Act” (the AI Act), the world’s most comprehensive legal framework for AI.

It is also the week of a state visit to the US by General Secretary Xi Jinping of the Chinese Communist Party. On September 24, he will meet with President Trump in Washington, where they are scheduled to discuss, among other things, AI safety.

Anticipating an opportunity for progress, Rep. Ted Lieu (D-Calif.) and Sen. Ed Markey (D-Mass.) wrote to President Trump last week urging him to persuade Xi to commit the People’s Republic of China to comply with the recently established US policy of maintaining a “human in the loop” for all nuclear launch decisions and agreeing that AI should never be able launch nuclear weapons by itself.

However, the problem with the Trump-Xi discussion is that the Trump administration has often framed AI development as a zero-sum contest with China for technological supremacy.

As Richard Clark, former cyber czar on AI Risks, told CNN: “We have the president of China coming here on the 24th. He has publicly talked about his concern with the possible loss of control. Wouldn’t it be nice if our president talked about the loss of control? Wouldn’t it be even nicer if the two presidents could work together to stop loss of control?”

“I worked for 25 years on arms control,” Clark continued, “where I was told you can’t trust people not to build biological weapons. You can’t trust people not to build chemical weapons. You can’t trust people to regulate the number of nuclear weapons. But we developed over time—through negotiations—verification methods that gave us enough certainty that we signed treaties on all of those initiatives and the Senate ratified treaties on all of those issues. It’s going to be tough, but you have to start somewhere and you have to start soon!”

An initial positive sign, however, is that in a September 20 meeting in New York between Treasury Secretary Scott Bessent and China’s Vice Premier He Lifeng, the creation of a mechanism, to be tentatively called “the US-China AI dialogue” was discussed. It would promote a shared vision of the goals of AI as well as its threats and establish a notification system (a kind of hot line) that "would allow the countries to alert each another about national security issues related to artificial intelligence."

In a recent article, Bill Gates asserted, “The transition to this new AI era will be one of the most turbulent times in human history,” proposing that the world needs a plan which should be developed through a “public democratic process.”

He argued that we must undertake the enormous task of building a new system to manage the transition, creating both a national and international framework for dealing with AI. This will require a multi-sectoral system, since current institutions were not designed to handle a technology that spreads so fast and involves so many parts of our lives. “The goal will be to make sure that every risk is accounted for. Otherwise, an AI-enabled attack might succeed because no one thought it was their job to stop it.”

“Because the risks cross borders, an international organization will have to be built in parallel. Although unlike any institution we’ve ever created, it can be modeled on existing cooperative arrangements, such as inspection regimes for nuclear weapons, regulations for international aviation, and agreements that protect the ozone layer. Countries will need to learn from each other and move quickly to build the right institutions before the disruption forces governments into crisis mode.”

Gates’ message to leaders is: “You have a chance to act now… You can work with other governments to meet this national and global challenge… This unprecedented technology demands an unprecedented global response... Leaders across academia, business, government, and civil society all have a role to play in shaping what comes next.”

A good place to start would be to carefully consider and evaluate the European Union’s “Artificial Intelligence Act” (the AI Act), the world’s most comprehensive legal framework for AI. Entering into force on August 1, 2024, it is based on four levels of risk of harm: “unacceptable;” “high;” “limited;” and “minimal.” The AI Act also has created various new bodies tasked with implementation and enforcement, including the “AI Office,” the “European Artificial Intelligence Board,” an “Advisory Forum,” and a “Scientific Panel of Independent Experts” to promote national cooperation and compliance. As well, EU member states designate “national competent authorities.”

On July 10, 2025, the EU launched a voluntary “General Purpose AI Code of Practice” (GPAI) which was signed by US companies, including Amazon, Anthropic, Google, IBM, Microsoft, and OpenAI (with Musk’s xAI signing only the Safety and Security section.) Most other prominent Canadian, European, and Chinese AI companies have also signed the GPAI; Meta is the only company that has declined to sign.

As legal commentators have noted, the AI Act is the most comprehensive framework to date for regulating AI across multiple sectors, and the first of its kind. It could be a useful reference point for companies and regulators outside the EU when they create their own approaches to AI governance. A number of criticisms have been leveled against the EU AI Act, and that analysis would also be useful for those wanting to design something even more comprehensive and effective.

In a book called, What’s the Worst that Could Happen: Existential Risk and Extreme Politics, Australian parliamentarian Andrew Leigh comments: “If our species goes extinct, we don’t just snuff out the lives of the 8 billion living today but also trillions of unborn people stretching out to the almost-infinite future. If our descendants had a voice, they would be shouting at us like a parent who’s just seen their child playing with the hand brake of a car parked at a cliff top… Strengthening institutions, deepening international engagement, and creating space for a more considered and thoughtful politics are the best ways to minimize existential hazards.”

Geoffrey Hinton, often called the “godfather of AI,” who in 2023 was the first to sound the alarm, says, “Humanity could go down the path toward its destruction or figure out how to deal with the dangers.” He renewed his call for global regulation, noting that, “while countries may not agree on everything, they share a desire to preserve humanity. For things like preventing AI from taking over from people, the interests of all the countries are aligned… So, they will eventually collaborate on that. The question is, will they do it in time?”