Earlier this month, Anthropic reported five cases in which people used its models for work that could support biological weapons development. The cases included help preparing a proposal for gain-of-function research on a mosquito-borne virus at a military institute and planning experiments intended to help bird flu get better at infecting mammals. Anthropic had no way to determine the intent behind this work but blocked it anyway.
That ambiguity is the problem. The same tools that scientists might use to design a vaccine can help design a worse pathogen. The same tools that help scientists design a drug that targets cancer cells while sparing healthy ones can be tweaked to build a drug that targets healthy cells alone. There is no need for someone to ask Claude to “build a bioweapon.” They only need to look like scientists who are working on important biological problems.
Anthropic, which shut the accounts down, has been the most public about the risks of AI to the development of bioweapons. OpenAI has also put in guardrails. Other AI companies, especially with open-weight models — those whose underlying code anyone can download and modify without any company oversight — generally release their products with far fewer safeguards and little monitoring for how their tools are being used.