When AI Takes Control of Weapons Systems
Precisely because machines accelerate military processes, policymakers must remain wise and persistent enough to decide what machines must not be allowed to decide.
Artificial intelligence is rapidly making its way into national defense. Reconnaissance, early warning, target detection, mission planning—everywhere, machine-learning systems promise faster analysis, faster responses, and faster decision-making.
In a military context, this sounds like progress. In reality, AI exacerbates a security problem that cannot be solved technically. Anyone who attempts to do so anyway risks achieving the opposite of what they are striving for.
Current Developments
US President Donald Trump sees no problem in the unchecked advancement of AI. He argues that a president’s high IQ is the only “guardrail” AI needs, so Americans have nothing to worry about.
He maintains this even in the face of concerns raised by leading US AI developers regarding further unchecked AI development. According to Trump, Chinese President Xi Jinping also sees no problem that could hinder the rapid advancement of AI. At the same time, more and more incidents are coming to light in which AI agents act independently and attack other systems. This has occurred not only in the US but also in China. Furthermore, the autonomous attack by an AI agent from the US company OpenAI on an Australian government portal came to light, as well as other incidents in the US in which swarms of AI agents attempted to gain unauthorized access to secure areas.
Have we already crossed the point at which humans still retain control over the AI they have developed? How far are we from the development of a superintelligence that optimizes itself autonomously, develops superhuman capabilities across various performance spectra, and—without being instructed to do so—turns against humanity?
Anthropic CEO Dario Amodei is now warning the United Nations Security Council about the risks of continued unregulated AI development: “If mismanaged, I even believe that AI could pose a risk to all of humanity.”
Anyone who uses a spam filter tacitly accepts a margin of error. Anyone who deploys a military AI system does the same—only the consequences of a misjudgment are different.
In light of the current dangers, other AI developers and CEOs of leading AI companies called on the Security Council for international cooperation and political oversight of AI development. These included OpenAI CEO Sam Altman, Hugging Face co-founder Clement Delangue, and Canadian AI developer Yoshua Bengio.
Trump, on the other hand, had previously dismissed criticism of unchecked AI development as a “hoax” and a “sick conspiracy”—as Trump stated on Truth Social: “The idea that AI will take over the world, destroy humanity, and do all those other terrible things is a hoax.”
Although there doesn’t seem to be any problem requiring regulation, Trump and Xi agreed during Xi’s recent state visit to the US to establish a communication channel and a dialogue regarding AI development. Here, Xi’s somewhat greater level of prudence on this issue appears to be evident.
At a subsequent meeting at the White House attended by, among others, Alex Karp (Palantir), Elon Musk (SpaceX), Mark Zuckerberg (Meta), Dario Amodei (Anthropic), and Jeff Bezos (Amazon), Trump stated that he remained opposed to government or international regulation, as self-regulation by companies was sufficient. Consequently, a voluntary commitment was signed by the AI companies; however, it was non-binding and based on voluntary participation, without, for example, the formation of a supervisory body. Trump said after the meeting, “This will all lead to good things, and we will ensure that everything runs smoothly and safely.”
It was precisely the opposite situation that had prompted the tech executives to address their warning to the US government and the United Nations.
However, it must be noted that the incidents to date have involved AI infiltrating civilian structures. But what if AI agents infiltrate weapons systems, or if AI implemented in weapons systems takes on a life of its own?
What Military AI Actually Does
Traditional software follows rules that a human has written beforehand: If A, then B. Anyone who understands such a program can trace, step by step, how it arrives at its result. Modern AI works differently. It isn’t programmed, but rather trained. It is fed vast amounts of examples—images, texts, sensor data—and the system derives patterns from them on its own. In the end, no one knows exactly how it weighs these patterns in detail, not even the developers.
We’ve long encountered such systems in our everyday civilian lives. The spam filter decides, based on statistical similarity, whether an email lands in the inbox or the junk folder. Facial recognition on a smartphone compares pixel patterns with stored patterns. The traffic forecast on a navigation device draws conclusions about the next 20 minutes based on millions of trips. Such applications are useful because a misjudgment usually has no consequences. An email that was mistakenly filtered out can be searched for and found. If a face isn’t recognized by a cell phone, the user can simply enter the PIN code.
In national defense, the stakes are much higher. Here, systems aren’t meant to distinguish between advertisements and invoices, but rather to assess—based on satellite images, radio signals, and drone footage—whether movement on the ground poses a threat—and they do so under enormous time pressure. Anyone who uses a spam filter tacitly accepts a margin of error. Anyone who deploys a military AI system does the same—only the consequences of a misjudgment are different. It is this shift in scale that turns what appears to be a technical question into a political one.
When Probability Looks Like Certainty
The logic of the time advantage is an old one. The so-called decision cycle—observe, assess, decide, act—is supposed to become shorter. But acceleration has an unpleasant characteristic: It does not automatically improve the quality of a decision. Sometimes it simply speeds up error. AI does not operate in a clean-cut world. It makes decisions under conditions of incompleteness, vagueness, and uncertainty. Sensors provide unclear signals, data is missing, and adversaries camouflage and deceive. Electronic warfare further alters the information landscape. What appears precise remains fallible.
A simple example illustrates this clearly. Anyone seeking to distinguish combatants from civilians looks for distinguishing features. Wearing a helmet is considered one of them. But the uncertainty doubles: Even determining whether someone is actually wearing a helmet is a statistical matter and not definitive. A certain hairstyle or a shadow can lead to the detection of a helmet even though none is present; conversely, an unfavorable reflection of light can prevent the detection of a helmet. And the inference from a helmet to a combatant is, of course, merely probable, since a soldier might have briefly taken off his helmet, and a civilian—such as a construction worker—might be wearing one. No certain conclusions can be drawn from uncertain information, no matter how large the model becomes. Even the best system merely shifts uncertainty into more elegant weightings—and ultimately delivers probabilities that look like certainties.
Where technology becomes opaque, politics must become more transparent.
Those who do not deal with statistics on a daily basis easily underestimate what is happening here. Even a system with a high hit rate produces errors on a scale that inevitably grows with the amount of data analyzed. When intelligence agencies process hundreds of thousands of images per day, even a small percentage of incorrect assessments results in a considerable absolute number of misjudgments. Added to this is an effect well known in statistics: The rarer the case being sought occurs in the population, the more severely the proportion of false alarms affects innocent bystanders. Especially in conflict zones, where civilians make up the majority, this is not a theoretical problem but one of great moral and political significance. In military applications, however, such a system is often touted as progress because it operates faster than any human analysis. Speed, however, does not produce truth; it merely multiplies the number of decisions, whether right or wrong.
Added to this is the lack of verifiability. Modern systems rely on countless features simultaneously. Their internal reasoning is rarely traceable in a form that truly enables human oversight in an emergency. With generative AI, the problem is particularly evident: it provides answers but no verifiable rationale. In security-related contexts, this is not a minor issue but a red line. Anyone who wants to deploy systems where decisions about life and death are made should be able to explain very precisely why technical sophistication should be considered a basis for judgment in this context.
The Invisible Armament
But the problems extend beyond individual decisions. AI is changing the very architecture of security policy. In traditional arms control, the balance of power was roughly measurable: tanks, aircraft, soldiers, delivery systems, ranges. Much of it was visible, countable, and at least partially verifiable through agreements. The Open Skies Treaty, signed in 1992, allowed for mutual observation flights between NATO countries and former Soviet republics—until the US withdrew in 2020. The 1987 INF Treaty banned land-based medium-range missiles for more than three decades. Such treaties were laborious. But they worked.
With AI and cyberweapons, this principle no longer applies. Capabilities can be developed in secret. Software can be copied, adapted, and distributed in countless variations. Once something has been developed, it never disappears. Even if one wanted to conclude agreements, there would be no quantifiable units to limit. One’s own capabilities, too, can only be tested to a limited extent. In reality, cyberweapons can hardly be tested without risking collateral damage. The joint US-Israeli Stuxnet attack on an Iranian uranium enrichment facility in 2010 accidentally infected more than 50,000 computers in India, Indonesia, Pakistan, and other countries. If neither side knows what it is capable of—let alone what the other is capable of—stability becomes a matter of chance.
Humans Are Too Slow for Their Own Weapons
This exacerbates a risk that has long been recognized within the military itself. The requirement that a human must remain “in the loop” sounds reassuring. But what does this “loop” actually mean? It means that while a machine may generate a proposal, a human must review and approve it before any action is taken. In practice, the available time windows vary greatly—ranging from hours in strategic situation assessment to a few seconds in tactical air defense. The shorter the window, the less room there is for an independent evaluation of the data on which the system based its recommendation. The notion that a truly independent review always takes place does not consistently hold up in reality.
Added to this is a phenomenon described in human-machine research as “automation bias”: Those who have repeatedly observed a technical system to function reliably are more inclined to confirm rather than contradict it in an emergency. Furthermore, a tendency toward passive involvement over time leads to a loss of competence, which further narrows the scope for independent judgment. In the announcement for a conference hosted by the Joint Air Power Competence Center this year in Essen, this question was framed as a key point of discussion: How can service members “maintain meaningful control while the pace of operations approaches the speed of machines”? Anyone who poses this question in this way already senses how difficult it is to answer.
What Can No Longer Be Postponed
This is precisely where the real political challenge lies. Where technology becomes opaque, politics must become more transparent. Where armaments grow behind closed doors, the channels of communication between capitals must become more frequent. Where learning systems shorten response times, politics must remain slow enough to set limits. This is not an old-fashioned longing for détente. It is the sober response to a technological reality that cannot be reversed.
First: We need robust communication channels between the major military powers—including, and especially, with rivals. Direct links between defense ministries and military leadership, crisis hotlines, and regular consultations on false alarms and near-misses. Such mechanisms existed during the Cold War, and they prevented escalation. In the age of automated decision-making, they are once again indispensable. Those who do not speak with their adversaries let AI do the talking—and that is a poor trade-off.
Second: We need arms control that addresses this new reality. Traditional treaties counted unit numbers. New frameworks must account for software behavior, training data, rules of engagement, and autonomous functions—as well as the thresholds beyond which a system acts on its own. International voluntary commitments not to delegate the use of lethal force to machines are a first step. Binding bans on fully autonomous weapon systems—those permitted to kill without human authorization—would be the second. Such rules are difficult to verify. But they are better than agreeing to nothing.
Anyone who believes that intelligent weapons can replace smart diplomacy has failed to understand that, in the age of AI, security is first and foremost a political task before it becomes a technical one.
Third: Europe has a responsibility here that it cannot delegate. The European AI Act regulates civilian applications. In terms of security policy, the work is only just beginning. The EU should not outsource the proposal for an international framework for military AI to Washington or Beijing. It can take the initiative itself—together with countries such as Switzerland, Brazil, Mexico, or South Korea, which, as impartial mediators, are more credible than the major powers that are direct rivals. A European initiative for military AI arms control would also serve as a counterargument to those who see Brussels as nothing more than a regulatory machine and fail to recognize European sovereignty as something in its own right.
Fourth, and most importantly: If you want effective rules, there’s no getting around negotiations. If you want to mitigate global risks, you must also talk to rivals. That sounds old-fashioned, but it isn’t. It’s simply realistic. Security does not arise from strength alone. It arises from understanding, transparency, and robust channels of communication between states and societies. Trust is not a “soft” word. It is the only resource that can still foster stability in a world of invisible capabilities.
The real challenge of AI in national defense is therefore not technical, but political in nature. The systems are getting faster. This does not make the world any clearer—quite the opposite. Precisely because machines accelerate military processes, policymakers must remain wise and persistent enough to decide what machines must not be allowed to decide. Anyone who believes that intelligent weapons can replace smart diplomacy has failed to understand that, in the age of AI, security is first and foremost a political task before it becomes a technical one.
It is to be hoped that Xi will prevail here over Trump, whose family—according to media reports—is said to have recently directed its investment activities toward shares in AI companies and data centers.
Perhaps this also presents an opportunity for cooperation between the EU and the People’s Republic of China to avert threatening developments, particularly with regard to the possible emergence of a superintelligence.
