How do I run LLM evals in CI/CD?

Keep a versioned golden dataset of 30 to 200 real inputs with expected properties, write assertions for each (exact checks first, model-graded checks second), and run them on every pull request that touches prompts, model settings or retrieval code. As of October 2026 the two common open-source runners are promptfoo (MIT; its GitHub Action posts a before-vs-after comparison on the pull request and needs Node.js 22.22.0 or later, Node 24 recommended) and DeepEval (Apache-2.0; plugs into pytest through assert_test and the deepeval test run command). Fail the job when the pass rate drops below a threshold you set.

Should I use promptfoo or DeepEval for CI?

Use promptfoo if your team works in JavaScript or TypeScript, wants declarative YAML test cases and wants a before-vs-after diff of edited prompts posted on each pull request. Use DeepEval if your app is Python and you want evals to look like unit tests in pytest, with metrics for RAG, agents and multi-turn conversations. Both are free and open source; you pay only for the model calls the evals make.

How do I keep LLM eval costs down in CI?

Run evals only when relevant paths change (prompts, model config, retrieval code), cache model responses between runs (promptfoo keeps a cache directory you can restore with actions/cache), use a small model as the grader for routine checks, run the full suite nightly instead of on every commit, and keep the per-PR suite to the cases that caught real regressions.

What should fail an LLM eval build?

Fail on hard rules every time (invalid JSON, a forbidden phrase, a missing citation, a tool call to the wrong function, a leaked secret) and on a drop in the model-graded pass rate beyond a set margin, for example more than 5 percentage points below the main branch. Report everything else as a warning so flaky model-graded scores do not block every merge.

Quick Answer

How to Run LLM Evals in CI/CD: A 2026 Step-by-Step Guide

Published: October 25, 2026

The short answer

Treat prompts, model settings and retrieval code like application code: every change runs a fixed test set, and a regression blocks the merge. The recipe, as of October 25, 2026:

Build a golden dataset of real inputs.
Write assertions: deterministic first, model-graded second.
Run them with promptfoo (YAML, Node) or DeepEval (pytest, Python) in GitHub Actions on pull requests that touch the relevant paths.
Fail on thresholds, cache responses, and run the long suite nightly.

Step 1 — Build the golden dataset

Collect 30–200 inputs from production logs or support tickets, not invented ones. For each, record what a good answer must do rather than one exact answer: “mentions the refund window”, “returns valid JSON matching the schema”, “cites at least one retrieved document”, “calls lookup_order before answering”. Include the failures you have already seen — every bug fixed becomes a test case. Store the file in the repository next to the prompts so a change to one is reviewed with the other.

Done when: the dataset is versioned in git and every case has at least one checkable expectation.

Step 2 — Write assertions in two tiers

Deterministic checks (cheap, never flaky): JSON schema validity, regex or substring, forbidden words, length limits, the right tool called with the right arguments, latency and cost ceilings.
Model-graded checks (costly, noisy): relevance, faithfulness to retrieved context, tone, rubric scores. Use these where deterministic checks cannot reach, and grade with a fixed model and a written rubric.

Done when: each test case has at least one deterministic assertion wherever one is possible.

Step 3a — Run promptfoo in GitHub Actions

promptfoo (MIT; release 0.125.1 published October 25, 2026) describes test cases in a promptfooconfig.yaml. Its official action, promptfoo/promptfoo-action@v1, runs on pull requests that modify the watched paths, evaluates the before and after versions of the edited prompts and posts a link to the comparison on the pull request. Per promptfoo’s docs it needs Node.js 22.22.0 or later (Node 24 LTS recommended) on the runner, a pull-requests: write permission to comment, your provider API key as a repository secret, and optionally an actions/cache step for ~/.cache/promptfoo so unchanged cases are not re-billed.

on:
  pull_request:
    paths: ['prompts/**']
jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/setup-node@v6
        with: { node-version: '24' }
      - uses: actions/cache@v4
        with: { path: ~/.cache/promptfoo, key: "${{ runner.os }}-promptfoo-v1" }
      - uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          prompts: 'prompts/**/*.json'
          config: 'prompts/promptfooconfig.yaml'
          cache-path: ~/.cache/promptfoo

Step 3b — Or run DeepEval with pytest

DeepEval (Apache-2.0; python-v4.2.4 released September 22, 2026) plugs into pytest: you load an EvaluationDataset (in code, CSV, JSON or from Confident AI), build test cases, call assert_test() with your metrics, and run deepeval test run test_app.py — the same command locally and in a workflow .yml. It covers single-turn and multi-turn, end-to-end and component-level evaluation, and its docs include a guide to comparing runs against a baseline.

Done when: the eval job runs on a pull request and its result is visible on that pull request.

Step 4 — Set thresholds that block a merge

Hard rules fail the build every time.
Model-graded pass rate fails the build only when it falls more than a set margin (for example 5 points) below the main branch’s last run; otherwise post it as a warning.
Pin the grader model and temperature so a score change means your app changed, not the judge.

Step 5 — Control cost and time

Trigger on relevant paths only, cache responses, use a small model as the grader for routine checks (current API prices), and move the slow, expensive suite to a nightly schedule. Re-sample production traffic into the golden set monthly so the tests track what users actually ask.

Which tool for which team

	promptfoo	DeepEval
Language	Node/TypeScript, YAML config	Python, pytest
Licence	MIT	Apache-2.0
CI integration	Official GitHub Action, before/after PR comment	`deepeval test run` in any CI
Strengths	Prompt diffs, red-teaming, many providers	RAG, agent and conversational metrics

For tracing and online evaluation after deployment, see how to monitor and evaluate AI agents in production and LangSmith vs Weights & Biases vs DeepEval.

Last verified: October 25, 2026. Requirements from promptfoo’s GitHub Action docs and DeepEval’s CI/CD docs; versions from GitHub releases.