Anthropic Report Says Company Blocked Possible Use of AI Model to Build Biological Weapons

As industry insiders issue louder warnings about the catastrophic danger of unchecked artificial intelligence, Anthropic has revealed in a report out Thursday that it had to "disrupt" several attempts by scientists to use its AI models for research that could contribute to the creation of biological weapons.

In the report, titled “Detecting and Countering Misuse of AI,” Anthropic said that over the past eight months, it banned several accounts that had used its chatbot Claude for research into highly contagious diseases.

As NBC News reported:

In each of the five cases Anthropic cited, the company suspected that the researchers behind them were affiliated in some way with a government or military in a country it had banned or took deliberate steps to hide their location and identity, or both. The company declined to say where any of the researchers were based, but its terms of service allow all use from all but a few regions, including China, Iran, and Russia.

In one example, Anthropic said researchers used a third-party platform that attempted to use Claude to complete a grant application for work that would look into how to repeatedly mutate chikungunya, a mosquito-borne virus that causes severe pain and fever in humans. Anthropic said it blocked all attempts by the researchers. The research, which would be conducted at a military institute, would seek to repeatedly mutate the virus in live animals to become increasingly more infectious, Anthropic said.

Another had used Anthropic’s Claude to assist in research into how to make bird flu, which historically has a global fatality rate of around 50% in confirmed human cases, more virulent and damaging to mammals. Anthropic said the researcher only received assistance from Claude’s weakest class of models, which provided clerical help.

Anthropic clarified that it did not find any concrete instance of a scientist attempting to use its technology for nefarious purposes, and that these use cases may well have been part of beneficial scientific inquiry, like research into vaccines or other cures for disease. Nevertheless, it said it erred on the side of caution and shut down the accounts.

"Biological misuse is one of the most serious risks of frontier AI models," the report explains. "It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences."

The report acknowledges that even as safeguards are implemented, those with the intent to do harm are getting more savvy about how to evade them and using the appearance of legitimate science to do it.

"Sophisticated threat actors are aware that we (and other AI providers) are attempting to detect dangerous uses of our models, and they use the dual-use nature of biology to maintain a kind of 'plausible deniability' about their research," the report says. "More sophisticated actors can hide their intent, extracting assistance from an AI model in interactions that look plausibly beneficial, but when put in context and analyzed holistically, can provide clear warning signs of misuse."

The report details other ways in which actors have used its technology for malicious purposes. It found that governments like China, Iran, and Mali had used Claude to collect information about journalists and dissidents. Others used it to conduct hacking campaigns, automate identity theft operations, and create malware.

Anthropic said it hoped that by publishing the report, it would "help other developers recognize similar patterns on their own platforms, give governments and civil society a clearer view of how emerging threats take shape, and strengthen collective defenses."

The report comes amid a torrent of scrutiny directed at Anthropic and its competitor OpenAI following reports in recent months of severe security breaches involving their technology, which has been documented acting autonomously to hack into other websites and evade human detection.

Earlier this week, Anthropic researcher Jacob Coxon, who previously worked at OpenAI, resigned from his post and sounded the alarm about fears within the industry that the technology “could kill us all by the end of the decade" and that the companies are prioritizing the race toward superintelligence without regard for the potential risks.

In the wake of Coxon's resignation, several other researchers from Anthropic, OpenAI, and Google have shared similar warnings that the technology could result in an extinction-level disaster or otherwise cause severe harm to human society.

A poll published on Wednesday by Data for Progress showed that 68% of likely voters in the US support a proposal announced last week by Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) for legislation that would permanently ban the development of "superintelligent" AI that can surpass human cognition and would put a pause on "advanced" AI development until a federal body is put in place to regulate the industry.

Casar said the fact that Anthropic was being relied upon to stop its own users from using its tech to potentially create bioweapons illustrated the urgent need for regulation.

"Right now, we're counting on AI corporations to voluntarily block weapons of mass destruction. Voluntarily," Casar emphasized in a post to social media on Thursday. "Congress must do the obvious thing: BAN catastrophic AI uses like creating biological or nuclear weapons."