What is the best alternative to OpenAI Evals?

Promptfoo is the default, and the one OpenAI itself recommends: it is open source, free in its Community edition, and runs evals from a config file locally or in CI. As of November 2, 2026, choose Braintrust instead for a hosted eval and tracing platform with a free Starter tier and Pro at $249 a month; Langfuse for open-source, self-hostable tracing plus evals (cloud from free to $199 a month on Pro); and LangSmith if you already build on LangChain or LangGraph (Plus $39 per seat a month).

When does OpenAI Evals shut down?

OpenAI's deprecations page lists three dates: deprecation announced June 3, 2026; existing evals became read-only on October 31, 2026; and the Evals dashboard and API are scheduled to shut down on November 30, 2026. Agent Builder and the v1/prompts API shut down the same day.

Can I export my OpenAI evals to Promptfoo?

Not automatically. OpenAI's migration cookbook covers manually recreating each evaluation in a promptfooconfig.yaml file: prompts, provider, test cases and assertions. It does not rely on an Evals export feature. Graders become Promptfoo assertions, and OpenAI advises validating any recreated grader, especially an LLM-as-a-judge grader, before using it for regression decisions, because similarity scores may differ between systems.

Is there a free OpenAI Evals replacement?

Yes. Promptfoo Community is free forever and includes all evaluation features, all model providers and red teaming up to 10,000 probes a month. Langfuse is open source and can be self-hosted for free, and its cloud Hobby plan includes 50,000 units a month. Braintrust's Starter plan is $0 with 1 GB of processed data and 10,000 scores a month.

Quick Answer

OpenAI Evals Alternatives: Promptfoo vs Braintrust vs Langfuse

Published: November 2, 2026

The short answer

OpenAI’s hosted Evals became read-only on October 31, 2026 and is scheduled to shut down on November 30, 2026. Copy your datasets and grader definitions out now. Then:

Promptfoo — the default and OpenAI’s recommended path: open source, free, config-file evals in CI.
Braintrust — hosted evals, experiments and tracing with a dashboard like the one you are losing.
Langfuse — open-source tracing plus evals you can self-host.
LangSmith — if your agents already run on LangChain or LangGraph.

The comparison

Tool	Type	Free tier	Paid (Nov 2, 2026)	Self-host	Best for
Promptfoo	Open-source CLI + library	Community: all eval features, red teaming 10K probes/mo	Enterprise / On-prem: custom	✅	CI regression tests, red teaming
Braintrust	Hosted platform	Starter $0: 1 GB data, 10K scores, 14-day retention	Pro $249/mo: 5 GB, 50K scores, 30-day retention	Enterprise	Dashboard experiments, team review
Langfuse	Open-source platform	Hobby: 50K units/mo, 2 users	Core $29, Pro $199, Enterprise $2,499/mo	✅ (free OSS)	Tracing + evals + prompts in one place
LangSmith	Hosted platform	Developer: 5K traces, 1 seat	Plus $39/seat/mo, 10K traces	Enterprise	LangChain / LangGraph teams

Prices from each vendor’s pricing page, read November 2, 2026.

The dates that matter

June 3, 2026 — OpenAI announced the Evals deprecation.
October 31, 2026 — existing evals became read-only: you can view and copy, not change.
November 30, 2026 — the Evals dashboard and API shut down, together with Agent Builder and the v1/prompts API.

If you also used Agent Builder, see OpenAI Agent Builder alternatives.

1. Promptfoo — the recommended migration

OpenAI’s own cookbook is titled Moving from OpenAI Evals to Promptfoo. The building blocks map one to one — test data, prompts, providers and scoring criteria — but the workflow changes: instead of a dashboard, you keep a promptfooconfig.yaml beside your code and run:

promptfoo eval -c promptfooconfig.yaml --no-cache
promptfoo view

Graders become assertions and metrics. The Community edition is free forever with all evaluation features, every model provider and red teaming up to 10,000 probes a month; Enterprise adds team sharing, SSO and continuous monitoring, and On-Premise adds full data isolation, both on custom pricing.

Pick it if evals should run on every pull request and live in Git.

2. Braintrust — the closest hosted experience

Braintrust gives you datasets, experiments, playgrounds, human review and tracing in a web app, with unlimited users, projects and experiments on every plan. Starter is $0 with 1 GB of processed data and 10,000 scores a month (then $4/GB and $2.50 per 1,000 scores) and 14-day retention; Pro is $249/month with 5 GB, 50,000 scores and 30-day retention. Its Loop agent can generate test cases and iterate on prompts.

Pick it if product managers and engineers review eval results together in a UI.

3. Langfuse — open source, self-hostable

Langfuse combines tracing, evaluation datasets, LLM-as-a-judge scoring and prompt management. Cloud plans: Hobby free (50,000 units a month, 30-day access), Core $29/month, Pro $199/month (3-year access, SOC 2 and ISO 27001 reports, HIPAA BAA available), Enterprise $2,499/month; extra usage is $8 per 100,000 units. The open-source version can be self-hosted at no licence cost.

Pick it if traces and eval data must stay in your own infrastructure.

4. LangSmith — for LangChain teams

LangSmith’s Developer plan is free for one seat and 5,000 base traces; Plus is $39 per seat per month with 10,000 base traces and access to Deployment. Enterprise adds self-hosted and hybrid deployment, custom SSO and RBAC.

Pick it if your agents are LangGraph graphs and you want traces and evals in the same place.

How to migrate in an afternoon

Copy out every eval’s dataset rows, prompt template and grader settings while the read-only view exists.
Recreate one eval in the new tool and run it against the current production model.
Validate graders: compare scores on 20–50 examples you have labelled by hand; LLM-judge scores rarely match across systems.
Wire into CI so every prompt or model change runs the suite — see how to run LLM evals in CI/CD.
Move prompt objects (pmpt_…) into code or a prompt management tool before November 30.

Last verified: November 2, 2026.