LLM red teaming: how to turn adversarial testing into a regression suite

What LLM red teaming is

LLM red teaming is controlled adversarial testing of an AI application. Testers use prompt injection, role-play framing, encoded instructions, multi-turn escalation, and data-extraction attempts to uncover policy violations, unauthorized actions, and information exposure before the same failures occur in production.

Effective testing depends on the application's architecture and risk boundaries. Red teamers examine system prompts, retrieval sources, connected tools, permissions, and access to sensitive data. Automated scanners can repeat broad attack sets, whereas human testers can adapt across multiple turns and exploit application-specific weaknesses. Teams may conduct the work internally, hire a security vendor, use open-source scanners, or combine these methods.

An actionable finding records the adversarial input, the application's response, the policy or control that failed, and enough context to reproduce the behavior. With that record, an engineer who never sat in on the test can reproduce the attack, fix it, and check the fix later.

Why one-time red team reports lose value

A red team engagement often ends with a report. Testers may document 12 successful attacks, engineers fix eight, and product leaders accept the remaining risks. Even when the fixes are sound, the report reflects the model, prompt, permissions, and application logic tested during that engagement. It cannot verify whether resolved attacks remain blocked after changes to any component.

A static report loses protective value in three common situations:

Model upgrades invalidate earlier assumptions: A prompt that blocks a jailbreak on one model version may yield different results after an upgrade because refusal and instruction-following behavior can change. Without rerunning the original attack, the team cannot confirm that the fix still works.

Prompt edits can remove security fixes: An instruction added to prevent a specific attack may be weakened or deleted during prompt optimization or feature development. When the rationale exists only in the report, anyone editing the prompt may not recognize the instruction as a security requirement.

Tester context is lost: When an internal tester changes projects or a vendor engagement ends, the team may lose the reasoning behind each finding, the applied fix, and the evidence needed to verify it. A later engagement can then spend time rediscovering known failures.

All three situations stem from the absence of regression testing. In software engineering, a resolved defect becomes a test that later changes must continue to pass. Each confirmed red team finding should therefore become a reproducible case with defined safe behavior and clear pass-or-fail criteria.

How to turn red team findings into regression tests

Converting a confirmed finding into a regression test requires four decisions: which input reproduces the attack, what the application should do safely, how the current version handles the case, and whether a future failure should block release.

Each confirmed attack becomes a scored evaluation case that future releases have to pass.

Step 1. Capture the finding as a dataset row

Preserve the adversarial input exactly as the tester submitted it, including whitespace, encoding, conversation history, retrieved content, and tool responses that contributed to the attack. Keep the unsafe response as evidence of the original failure, then define the safe behavior that future application versions must produce.

In Braintrust, the finding becomes a row in a dataset. The following code from the dataset documentation shows the record structure:

async function main() {
  // Initialize dataset (creates it if it doesn't exist)
  const dataset = initDataset("My App", { dataset: "Customer Support" });

  // Insert records with input, expected output, and metadata
  dataset.insert({
    input: { question: "How do I reset my password?" },
    expected: { answer: "Click 'Forgot Password' on the login page." },
    metadata: { category: "authentication", difficulty: "easy" },
  });

  dataset.insert({
    input: { question: "What's your refund policy?" },
    expected: { answer: "Full refunds within 30 days of purchase." },
    metadata: { category: "billing", difficulty: "easy" },
  });

  dataset.insert({
    input: { question: "How do I integrate your API with NextJS?" },
    expected: { answer: "Install the SDK and use our React hooks." },
    metadata: { category: "technical", difficulty: "medium" },
  });

  // Flush to ensure all records are saved
  await dataset.flush();
  console.log("Dataset created with 3 records");
}

main();

For a red team finding, input contains the complete attack context, and expected describes the approved refusal, fallback, or constrained action. The metadata field can retain the observed unsafe output, attack family, severity, affected component, remediation status, and source engagement.

Store these cases in a dedicated adversarial example set alongside the application's golden dataset. The two sets need different pass conditions. A refusal may be correct for an adversarial input even though the same response would fail a normal product request.

Step 2. Define the pass condition

Translate the violated policy into an explicit pass condition for each finding. If an attack exposed a private support email address, any response containing that address fails. A refusal or safe fallback passes only when it withholds the protected information and does not repeat or act on the injected instruction.

The pass condition then becomes a scorer. Exact restrictions can use deterministic checks, and behavioral requirements can use model-based scoring with human-labeled examples for calibration. The scorer should test only for the specific behavior the red teamer flagged, leaving general response quality to the scorers on the golden dataset.

Step 3. Run the dataset as an experiment

Run the adversarial dataset as an experiment against the current application to establish a baseline. Braintrust saves the result for comparison with later changes to prompts, models, retrieval, and permissions.

Review individual cases alongside the aggregate score. Two application versions may achieve the same overall pass rate but fail different attacks, including cases with different severity levels. Row-level comparisons show whether a change resolved the targeted vulnerability, preserved earlier fixes, or introduced a new regression.

Step 4. Gate the merge on the result

Add the adversarial experiment to the pull request checks for changes that can affect application behavior. A regression on a safety-critical case should block the merge. Lower-severity categories can use thresholds approved by the security and product teams.

Where adversarial test cases come from

Braintrust does not generate attacks, simulate adversaries, or scan an application endpoint. Candidate test cases come from four sources, and Braintrust provides the evaluation system to preserve and enforce the confirmed findings.

In-house red teamers: Security specialists and engineers can target product-specific risks involving data access, tool permissions, user roles, and approval requirements. Their knowledge of the application helps determine whether an observed behavior would expose data or trigger an unauthorized action, and how severe the finding should be.

Security vendors and consultancies: External engagements provide concentrated testing across agreed attack categories. Preserve the exact input, application response, configuration, severity, and remediation notes for each confirmed finding so the evidence remains usable after the engagement ends.

Open-source attack tools: Garak uses probes and detectors to identify vulnerabilities in LLMs and dialog systems. PyRIT supports automated multi-turn attack strategies, and Promptfoo can run groups of red-team checks mapped to the OWASP Top 10 for LLM Applications. Automated scans can expand coverage quickly, but their results require review before they become release requirements.

Production incidents: Support tickets, flagged conversations, abuse reports, and human reviews can uncover attacks under conditions that pre-deployment testing did not reproduce. Preserve the relevant conversation, retrieved content, tool activity, and application configuration so the failure can be investigated and recreated accurately.

Combining all four sources reduces reliance on the assumptions of a single testing method. Human review then confirms which findings constitute genuine failures and should be designated as mandatory cases in the adversarial evaluation suite.

How to write scorers for adversarial evaluations

A red team finding becomes an enforceable test only when its scorer measures the behavior that made the original response unsafe. Braintrust scorers receive the input, output, expected value, metadata, and optional trace data, then return a score between 0 and 1. The appropriate scoring method depends on whether the pass condition can be expressed as an exact rule or requires semantic judgment.

Use deterministic checks for explicit failures

Custom code scorers work well when the unsafe output has a recognizable signature, such as a protected email address, an account identifier, an internal hostname, or a jailbreak marker. They produce the same result for the same output and require no additional model call.

// Enter handler function that returns a score between 0 and 1
function handler({
  output,
  expected,
}: {
  output: string;
  expected: string | null;
}): number {
  if (expected === null) return 0;
  const bullets = output.match(/^- .+/gm) || [];
  return bullets.length === 3 ? 1 : 0;
}

The documentation example above checks whether the output contains exactly three Markdown bullet points. An adversarial scorer needs a rule tied to the finding's pass condition. For a data-exposure case, for example, the scorer could return 0 when the protected identifier appears and 1 when the response withholds it.

Braintrust supports custom scorers in TypeScript and Python. Teams can define them within evaluation code, push them from a file through the CLI, or create them in the UI.

Use an LLM judge for nuanced harms

Some failures cannot be identified through an exact string or pattern. A response may adopt the attacker's requested persona, disclose sensitive meaning through a paraphrase, or provide prohibited guidance without using an explicitly blocked term. An LLM-as-a-judge scorer can evaluate these behaviors against a written rubric.

The rubric should define the violated policy, describe the required safe behavior, and include examples of passing and failing responses. Before using the scorer in a release gate, compare its decisions with human labels across representative attacks and legitimate requests. Model judges add evaluation time and cost, and their outputs can vary between runs, so the calibration set should confirm that the scorer consistently reproduces the approved labels.

Version scorers with the cases they evaluate

A scorer can become outdated as attack phrasing and application behavior change. For example, a rule that detects one representation of exposed information may miss the same information in a reformatted or paraphrased response.

Braintrust versions scorers automatically. When a new attack variant defeats the application or exposes a weakness in the scorer, add the variant as a dataset row and revise the scoring criteria. Rerunning the expanded dataset verifies that the updated scorer recognizes the new failure and continues to classify the earlier cases correctly.

Run the adversarial regression suite in CI

A confirmed finding protects future releases only when its regression test runs before a behavior-changing update reaches users. Trigger the adversarial suite for prompt edits, model changes, retrieval configuration updates, tool-permission changes, and dependency or SDK upgrades. Each change can alter the application's responses or available actions, reopening a vulnerability that was already fixed.

Braintrust can run experiments in CI/CD through the bt eval CLI or its dedicated GitHub Action. The CLI works across CI systems, and the GitHub Action posts a pull-request comment comparing the new experiment with its baseline.

# GitHub Actions example
- name: Run evals
  env:
    BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
  run: bt eval tests/

The command discovers evaluation files under tests/ and records the results as Braintrust experiments. The API key remains stored as a GitHub Actions secret.

Scorer results also need a defined pass/fail policy. A custom Reporter can fail the CI job when any safety-critical case regresses, even if the aggregate score remains above its threshold. For example, one case that exposes a credential should block the release. Aggregate thresholds are more appropriate for criteria such as tone, where a small variation in score does not necessarily indicate a security failure.

While a pull request is still in progress, --first N or --sample N runs a subset of the dataset for faster feedback. Require the full adversarial dataset to pass before the change is merged or deployed. When a confirmed production finding enters the dataset via the eval feedback loop, subsequent full runs include the new case in line with the established release policy.

How adversarial regression suites strengthen release security

Maintain evidence across releases

Braintrust records experiment results across application versions, allowing security reviewers, enterprise buyers, and compliance teams to verify which adversarial cases ran, whether they passed, and when a regression appeared.

Red team inputs and outputs may contain sensitive application or customer data. Braintrust's deployment options include BYOC and self-hosted data planes, which keep experiment logs, traces, datasets, prompts, completions, and customer inputs within the organization's own cloud account.

Preserve earlier coverage as new findings are added

Each confirmed finding remains in the suite while later engagements contribute new cases. Internal teams and external vendors can focus on untested attack surfaces, while the organization retains prior findings even after individual testers change roles or vendor engagements end. As the suite grows, manual retesting consumes less of each engagement.

Keep security requirements attached to releases

A prompt rewrite or a model upgrade can reopen a vulnerability the team resolved months earlier. Running the suite in CI identifies the affected adversarial case during the pull request, keeps the associated security requirement within the release decision, and reduces dependence on institutional memory.

Braintrust's free plan requires no credit card and includes unlimited users, projects, datasets, playgrounds, and experiments. Start building an adversarial regression suite for free with Braintrust.

FAQs about LLM red teaming (2026)

Does Braintrust generate red team attacks?

Braintrust does not generate attacks or scan applications for vulnerabilities. Human red teamers and security tools supply the findings. Braintrust takes over after a finding is confirmed, ensuring the same failure can be evaluated as the application changes.

How is red teaming different from guardrails?

Red teaming actively searches for ways to compromise an AI application under controlled conditions. Guardrails evaluate live inputs, outputs, or actions and decide whether to allow, modify, or block them. Red teaming can also test whether guardrails stop harmful behavior without rejecting legitimate requests.

How often should the adversarial regression suite run?

Run safety-critical cases whenever application behavior or permissions may change, and run the complete suite before major releases. Scheduled runs are useful when hosted models or external services can change independently of the codebase. Add and test every newly confirmed production failure immediately.

Can a third-party red-teaming vendor's report be used with Braintrust?

A vendor does not need direct access to Braintrust. The report must provide enough context to reproduce each finding and define the intended safe behavior. Remove secrets or customer data before importing the findings, then have the application owner confirm the expected outcome and severity.

What is the minimum setup to start with one finding?

To start in Braintrust, create a one-row dataset containing the reproducible attack and approved safe outcome, define the pass condition with one scorer, and run an experiment against the current application to establish a baseline. Once the test produces consistent results, add it to CI and introduce broader release policies as the adversarial dataset grows.