Chinese AI model gave bioweapon steps, security firm says
Mindgard says it got two of Moonshot AI’s Kimi models to describe how to make biological weapons and plan assassinations by “jailbreaking” their safety limits; Moonshot says it is now reviewing the case.
A UK cybersecurity firm says it persuaded a Chinese AI chatbot to describe how to make biological weapons and carry out assassinations, after finding a way around the safety limits built into the system — a finding first reported by the BBC.
Mindgard, which tests the security of AI systems, said it found in July that two models made by the Chinese developer Moonshot AI, called Kimi K2.6 and K3 Swarm, could be made to ignore the guardrails meant to stop them answering dangerous questions.
The method is known as “jailbreaking”: researchers feed an AI system a chain of complex instructions designed to trick it into dropping the restrictions its own developers set.
Mindgard emailed Moonshot about its findings on 27 July and followed up around a week later, before publishing its research in a blog post on 12 September.
The firm said it had not tested whether the answers Kimi gave would actually work in practice.
Mindgard also said it was confident that a jailbroken version of Kimi 2.6 could let a hacker run their own code on the system’s computing power and connect it to the internet, which it said could turn the chatbot into a launchpad for cyberattacks.
Kimi is what is called an “open-weight” model, meaning anyone can download it and run it on their own computers rather than through Moonshot’s servers — a setup that limits how much control the developer keeps over how the model is used once it is out in the world.
Moonshot only responded to Mindgard after being approached for comment by reporters working on the story.
The company said it was carrying out an internal review and was in discussion with Mindgard, adding that it welcomed third-party input “as a key pillar for building better and safer AI.”
Moonshot also shared an email it had sent Mindgard, in which it said its own internal evaluations had shown “a high refusal rate for these types of requests.”