01What red teaming is — and why AI needs its own kind

Red teaming borrows a military and cybersecurity idea: a dedicated team plays the adversary, attacking your own system so you find the weaknesses before a real attacker does. AI red teaming applies that to AI and machine-learning systems — especially large language models and generative AI — through structured, adversarial testing designed to discover, measure, and help reduce harmful, unsafe, or insecure behavior, both before and after deployment.

Here's the key difference from classic security work. Traditional red teaming targets infrastructure — networks, applications, identities, the things attackers have always gone after. AI red teaming targets model-specific failure modes that simply didn't exist before: adversarial inputs, prompt injection, jailbreaks that talk a model past its guardrails, attempts to extract training data, model backdooring, and data poisoning. Doing that well takes traditional security skill plus AI subject-matter expertise — you need to understand how the model behaves, not just how the server does (Google AI Red Team report; MITRE ATLAS).

• Goal: find harmful or unsafe behavior on purpose, so it can be measured and reduced before users hit it.
• What's new: the attack surface is the model's behavior — jailbreaks, prompt injection, data extraction — not just the surrounding infrastructure.
• Who does it: people who combine security skills with AI expertise, working ideally independent of the team that built the model.

02Two ways to attack: human and automated

Red teams work along a spectrum from hands-on human probing to fully programmatic attack generation — and serious programs use both.

Manual (human) red teaming puts expert testers in front of the model to craft attacks and probe for harms at scale. Anthropic's team did exactly this and released a dataset of 38,961 red-team attacks, documenting harm types and lessons learned (Ganguli et al., 2022). A striking finding from that work: RLHF-trained models became harder to red team as they scaled up, while other model types didn't show the same trend — bigger, better-aligned models pushed back more.

Automated red teaming generates large volumes of test cases by program or by model. Perez et al. (2022) showed a "red" language model can automatically write test cases against a target model — methods ranging from zero-shot prompting to reinforcement learning — surfacing tens of thousands of offensive replies from a 280-billion-parameter chatbot. Others, like the GCG attack (Zou et al., 2023), use combined greedy and gradient-based search to find adversarial "suffixes" that maximize the chance of a non-refusing answer; unsettlingly, those suffixes often transfer to black-box, publicly released models. Automation scales coverage; humans bring creativity, judgment, and novel-risk discovery. You want both.

• Manual: expert testers, creative and judgment-heavy — best for novel and context-specific harms.
• Automated: model- or program-generated attacks (red-LM, GCG) — best for volume and reproducible coverage.
• Scaling insight: RLHF models can get harder to break as they scale, but automated transferable attacks remain a live threat.

03The frameworks that give it structure

Red teaming isn't ad-hoc poking. A set of public frameworks gives teams a shared vocabulary, a catalog of risks to cover, and a place for the work in the wider governance picture. Switch between the four most-cited references to see what each one contributes.

04Run a coverage board for yourself

Here's the core loop of a red-team exercise, simplified into something you can drive. You're testing a simulated model against six categories of probe — five you can defend, plus one, novel / unknown attacks, that cannot be pre-tested because it doesn't exist yet. Hit Run probes to see which categories the model currently fails. Then turn on mitigations — the defenses a team would actually deploy — and run again to watch the coverage scorecard climb. The point isn't to learn any attack; it's to feel how defense raises coverage, and to see for yourself that even fully defended, coverage tops out below 100% — the residual category structurally caps it.

• Each mitigation closes specific failure categories — no single defense covers everything.
• Stacking defenses raises coverage, but a residual gap always remains: red teaming reduces risk, it doesn't eliminate it.
• This is a teaching simulation — real coverage depends on the behavior set, the judge, and the threat model, and novel attacks appear after testing.

05From findings to fixes — and how it's measured

Finding a problem is only half the job; red teaming earns its keep when findings feed back into the model. Documented harms become training signal for alignment methods like RLHF and Constitutional AI — the latter has the model critique and revise its own outputs against a written list of principles, reducing harms with fewer human labels (Bai et al., 2022). Other findings harden the system around the model: tighter system prompts, input and output filters, and stricter deployment controls that escalate as capability grows (for example, Anthropic's AI Safety Levels under its Responsible Scaling Policy).

To know whether any of this worked, teams measure with reproducible benchmarks. HarmBench standardizes automated red-teaming evaluation across 18 attack methods and 33 target models and defenses, scoring by attack success rate (ASR). JailbreakBench provides a 200-behavior dataset and a public leaderboard so attacks and defenses can be compared on equal footing. One honest caveat: ASR and benchmark numbers depend on the judge model, the behavior set, and the threat model, so scores are not directly comparable across papers unless they use the same protocol.

• Findings feed mitigation: RLHF and Constitutional AI turn documented harms into training signal.
• System hardening: system prompts, input/output filters, and capability-tiered deployment controls.
• Measured with: HarmBench and JailbreakBench (attack success rate) — reproducible, but protocol-dependent.

06Check your understanding

07Take it with you & go deeper