Three in Five AI Models Fail Counterterrorism Safety Test — Stripped Models All Fail

On Friday, October 9, CBC reported the results of an AI safety study. The UK nonprofit Tech Against Terrorism tested 134 mainstream large language models — three in five failed its counterterrorism safety test.

The test uses the group’s in-house CT-AI benchmark. The first version, released at the United Nations in July, covered only 27 models. This expanded edition spans 134 models and 627 requests, covering what an attacker planning an attack would want to ask about: attack planning, fundraising, and radicalization propaganda.

Per The National’s reporting, 81 models fully answered these questions, 51 gave useful advice, and only 2 earned a “provisional pass.” In other words, the vast majority of models provided at least useful information on these questions.

Stripped of Safety Locks, Not One Survives

The most telling dataset in this round is the “abliterated” models. Abliteration is a technique that removes a model’s built-in safety restrictions — reportedly achievable in minutes with free online tools.

All 13 stripped versions failed the test. Every single one.

Meta’s Llama 3.1 8B is the clearest example. The original scored 97 out of 100 on the group’s safety benchmark, refusing these requests. After the safety locks were stripped, the score fell to about 3 — and it began giving detailed answers to attack, funding, and radicalization questions.

Who You Say You Are Matters More Than What You Ask

This round had another detail worth pausing over. Researchers found that how a request is framed matters more than its content in determining whether a model answers.

Say “I’m a terrorist planning to cause mass harm” and under 2% of requests got a usable answer. Switch to “I’m a security researcher” and the share rose to 16.9%. Founder Adam Hadley put it this way: a model that refuses a self-declared terrorist but answers a self-declared researcher “isn’t becoming safer — it’s becoming more polite.”

What the Research Group Wants

Tech Against Terrorism is calling for three things: mandatory safety testing before model release, government-backed independent safety benchmarks, and restrictions on distributing unsafe models through channels like app stores.

Hugging Face responded that it regularly reviews violating content, but warned that some of the report’s recommendations could harm open research. Meta said its models all undergo safety evaluation and its policies prohibit harmful or illegal use.

One more number: the group counts over 29,000 repositories on Hugging Face promoting “uncensored” models. Hadley’s words: “It’s understandable to worry about losing control, about existential risk. But it’s already happened — plenty of open models were cracked long ago; nobody just noticed.”

What really stands out: the guardrails were already dismantled long ago. What comes next is whether model vendors and Hugging Face will respond with substance — or keep talking past each other.