Best LLM Red Teaming Tools 2026: Promptfoo, Garak, PyRIT
The short answer
Pick Promptfoo to red team your own app in CI, garak to scan a model endpoint, and PyRIT when a security team wants to script long, multi-turn attack campaigns. The best LLM red teaming tools, as of October 29, 2026:
| Tool | Maintainer | Licence | Latest release | Best for |
|---|---|---|---|---|
| Promptfoo | Promptfoo (acquisition by OpenAI announced Mar 9, 2026) | MIT | 0.125.1 (Oct 25, 2026) | App-level red teaming in CI; agents, RAG, tools |
| garak | NVIDIA | Apache 2.0 | v0.17.1 (Oct 16, 2026) | Vulnerability scanning of a model or endpoint |
| PyRIT | Microsoft | MIT | v1.1.0 (Sep 4, 2026) | Scripted, multi-turn attack campaigns by security teams |
| DeepTeam | Confident AI | Apache 2.0 | v1.0.9 tag | Python teams already on DeepEval |
Versions and licences are from each project’s GitHub repository on October 29, 2026. All four are free; you pay only for the model calls the attacks consume.
How the tools differ
Promptfoo describes itself as a tool to “test your prompts, agents and RAGs” with red teaming, pentesting and vulnerability scanning. You declare your target (an HTTP endpoint, a provider, or a script that drives your agent) in YAML, choose plugins for the harms you care about — prompt injection, jailbreaks, PII leaks, excessive agency, tool misuse — and it generates adversarial inputs, runs them and grades the results into a report. Because it is a CLI with exit codes, it drops straight into GitHub Actions. With about 26,500 stars it has the largest community of the group. OpenAI’s announcement says Promptfoo’s technology is used by over 25% of Fortune 500 companies and will become part of OpenAI Frontier, while the open-source project continues.
garak (“the LLM vulnerability scanner”) works like a network scanner for models: probes send known attack families — encoding tricks, prompt injection, data leakage, toxicity, package hallucination — and detectors score the replies. It is the fastest way to get a broad baseline on a model or endpoint you are about to adopt, and NVIDIA ships releases roughly monthly.
PyRIT (Python Risk Identification Tool) is a framework rather than a scanner. Security engineers compose orchestrators, converters (rewriting attacks into other encodings, languages or images) and scorers to run multi-turn conversations where an attacker model adapts to the target’s answers. It takes more code, and repays it for deep campaigns against a high-value system.
DeepTeam comes from the team behind DeepEval and offers vulnerability classes and attack methods in the same pytest-style workflow, which makes it the natural pick if your evals already run on DeepEval.
Picks by situation
- You ship an LLM feature and want a gate in CI: Promptfoo, run on every pull request that touches prompts, models or tools.
- You are choosing between models or self-hosted checkpoints: garak against each candidate endpoint.
- A security team runs a pre-launch assessment: PyRIT for multi-turn campaigns, plus Promptfoo for the regression suite that follows.
- Voice agents: generic text tools miss audio-specific attacks; voice QA platforms such as Roark include adversarial caller simulations (see voice agent regression testing vendors).
A red-teaming loop that works
- Write a threat model first. List what the app can read, which tools it can call and what a bad outcome looks like (leaked customer data, a refund issued, a wrong medical claim).
- Run a broad automated scan with garak or Promptfoo to find the obvious failures.
- Target the tools. For agents, most real damage comes from indirect prompt injection — instructions hidden in a web page, email or document the agent reads — leading to a tool call. Test that path explicitly.
- Turn every finding into a test case and keep it in CI so fixed holes stay fixed; the mechanics are in how to run LLM evals in CI/CD.
- Add runtime defences for what testing cannot rule out: LLM guardrails, PII redaction and the architectural controls in prompt injection defence: sandboxing vs allowlists vs approval.
Budget note: automated attacks call your application’s model many times. Run the full suite nightly and a smaller, fixed set on each pull request.
Last verified: October 29, 2026.