TL;DR

Gauge CLI (gauge-sh/gauge-cli) is an open-source command-line tool that runs real coding agents (Claude Code, Codex, Pi and OpenCode) against your product, records where they get stuck, and lets you retest a fix to your docs, Skill, CLI or SDK before you ship it. Gauge calls this “Agent Experience”: can an agent actually finish a task with your product? It calls the companion question “Agent Preference”: does the agent pick your product at all?

Last verified: November 12, 2026. Facts as of that date:

  • 146 GitHub stars, 11 forks, MIT license, written in TypeScript. The public repository opened on October 20, 2026.
  • Latest release: CLI 0.29.0, published November 9, 2026 (also 0.29.0 on npm as @withgauge/cli). The changelog lists 27 versions.
  • 1 open issue (GitHub’s combined counter shows 5 because it also counts four open pull requests).
  • Requires Node.js 22.13+, Docker running Linux containers, and credentials for the agent you want to test.

Not run by us: our hands-on lab (a throwaway GitHub Actions container) could not start on November 12, 2026 because GitHub refused the job over a billing limit on the account. Even with the lab running, a full Gauge loop needs a paid Claude or OpenAI login, which the lab does not have. This review is built from the README, the CLI reference, the changelog, the official docs and pricing pages, the issue tracker, the Show HN thread and a recorded interview with Gauge’s CEO.

Disclosure: Gauge’s CEO said he would consider linking to this review if it is useful to their users. He did not see it before publication, and nothing was promised about what it would say.

Our verdict: Gauge CLI turns something careful devtool teams already do by hand (watch an agent fail at your quick-start, fix the docs, try again) into a repeatable, recorded experiment. The local mode is free, needs no Gauge account and keeps results in a SQLite file in your repo. The useful idea is the paired experiment: same task, same agent, same model, only your docs or Skill changed. The costs are real, though. Every run burns your own agent tokens, the analysis step is done by your coding agent rather than by Gauge, the project is three weeks old in public with essentially two contributors, and the CLI has broken its own commands between versions. Worth trying if you ship a CLI, SDK, MCP server or Skill that agents are supposed to use. Overkill if your product is not something agents install.

Quick Reference

FieldValue
What it isCLI that runs coding agents in Docker against your product and records friction
Agents (local)Claude Code, Codex, Pi, OpenCode
Agents (hosted, per docs)Also Antigravity
Inputs it can test locallySkills, npm CLIs and SDKs, docs and files, web fixtures
Installnpm install -g @withgauge/cli
RequirementsNode.js 22.13+, Docker, an agent login or API key
Storage.gauge/gauge.sqlite plus raw transcripts under .gauge/runs/
LicenseMIT
PriceLocal: free (you pay your model provider). Hosted: free tier, Pro $499/month
TelemetryOn by default, opt out with gauge telemetry disable
Latest release0.29.0, November 9, 2026

What problem Gauge is solving

A growing share of developer-tool adoption now happens inside a coding agent. Someone types “add auth to this app” and Claude Code or Codex picks a library, reads some docs, runs some commands and either gets it working or gives up. If your CLI has an interactive prompt, the agent hangs. If your docs show an outdated API, the agent writes outdated code. If your Skill’s description is vague, the agent never loads it.

The Gauge README lists the findings it sees most often:

  • an interactive command blocks a coding agent
  • an agent reaches for deprecated commands or outdated API patterns
  • a Skill goes unused even when it would help
  • an agent confidently claims a limitation that does not exist
  • an agent cannot work out the intended workflow across several commands

None of these show up in unit tests. Gauge’s answer is to run the agent, in a clean sandbox, many times, and read the transcripts. As CEO Caelean Barnes put it in a Mux Dev Talks interview (November 3, 2026): “the agent is being held constant. It’s Claude Code or it’s Codex, and we’re tweaking the environment.” He also said technical documentation, not marketing pages, was the main thing agents read when choosing a product in their runs.

Gauge is a Y Combinator company that, per the Show HN post, started out building open-source devtools (including Tach), pivoted to AI search optimization, and moved into coding agents when devtool customers asked how to show up in Claude Code and Codex. Its paid products are Gauge Chat (AI search visibility) and Gauge Agents (hosted agent evaluation); the CLI is the open-source front end to the second, with a local backend that needs no account.

How the loop works

The README calls the workflow “hillclimbing”: make a change, measure whether it helped, repeat. In practice it has five parts:

  1. Define a task. A saved eval (a prompt plus success criteria) or a preference prompt (an open-ended task where the agent picks its own tools). Saved measurements live in SQLite; shareable ones can be Markdown files in gauge-cases/.
  2. Run your current product. Gauge starts a fresh Docker session per sample, injects the working-tree version of your inputs, and lets the agent work.
  3. Find friction. Gauge normalizes the event streams of the four supported agents into one transcript format. In local mode, your coding agent reads that evidence and saves structured findings with gauge run analysis save. In hosted mode, Gauge analyzes each session automatically.
  4. Change and retest. Edit the Skill, docs or CLI, rebuild the package if needed, rerun the same eval ID. The new batch snapshots the changed files; the baseline batch stays intact.
  5. Compare. gauge run compare summarizes completion, turns, duration and criterion judgments between two batches.

The architecture, as the Show HN post describes it, is two pieces: sandboxes and a network gateway. Locally the sandboxes are Docker containers, and the gateway is a mitmproxy sidecar container per session. The gateway is where Gauge terminates TLS and swaps in a variant of a web page transparently, so the agent fetches docs.yourproduct.com and gets your unpublished draft at the real hostname. It also handles credential replacement so agents see placeholders rather than real keys. Inference still goes to Anthropic or OpenAI; only the sandbox is local.

Getting started

The install, from the README:

npm install -g @withgauge/cli
gauge local runtime pull      # downloads the runtime image pinned to this CLI release
gauge init --local            # private .gauge/ workspace, changes no tracked files
gauge install-skill           # teaches your coding agent where the CLI's instructions are

The runtime image (ghcr.io/gauge-sh/gauge-runtime) is pinned by digest per release, with native linux/amd64 and linux/arm64 builds. Windows needs WSL. Adding --cases to init also writes gauge.json, a gauge-cases/ folder and guidance in AGENTS.md; without it, nothing tracked changes (the default flipped in 0.27.0).

The README’s example: testing a release-notes Skill

The repository walks through a deliberately small case. First a Skill:

---
name: release-notes
description: Write release notes for a new version in the team's format.
---

Write RELEASE-NOTES.md with the version as its title, an Added section for new
features, and a Fixed section for bug fixes. Use one bullet per change.

Then declare it as an input in gauge.json:

{
  "version": 2,
  "inputs": {
    "release-notes": { "type": "skill", "path": "skills/release-notes" }
  }
}

Save an eval with a task and a criterion, preview it, then launch:

gauge eval create --local --name "Release notes" \
  --agent codex --samples 1 \
  --text "Write the release notes for version 1.2.0. We added dark mode and fixed a login crash." \
  --criterion "Team format: RELEASE-NOTES.md has a version 1.2.0 title, dark mode under Added, and the login crash fix under Fixed."

gauge run --eval <eval-id> --local --input release-notes --dry-run
gauge run --eval <eval-id> --local --input release-notes
gauge instructions session-analysis --local

The detail that makes this a real test rather than a demo: the task never tells the agent to use the Skill. So the session also shows whether the agent discovers the guidance on its own, which is the failure most Skill authors never see.

After editing the Skill, rerun the same eval and compare:

gauge run --eval <eval-id> --local --input release-notes
gauge run compare --local --baseline batch=<before-id> --candidate batch=<after-id>

For an npm CLI or SDK you pack a tarball first; Gauge installs the pre-release artifact in the sandbox, so you can check whether a reworded error message changes agent behavior before publishing.

Gauge is meant to be driven by your coding agent, not typed by hand: gauge instructions prints version-matched guidance, read commands accept -o json, and gauge watch starts a local read-only viewer on 127.0.0.1 (from port 4040).

Local vs hosted

Local (free)Hosted Gauge
ExecutionDocker on your machine, your agent credentialsManaged compute; runs continue with your laptop closed
AnalysisYour coding agent writes findings into SQLiteAutomatic judging after each session
InputsSkills, npm CLIs/SDKs, docs and files, web fixturesAlso native binaries, website previews, authenticated MCPs and APIs
AutomationScripts and local batchesSchedules, GitHub App PR checks, more concurrency
Newer features—Run forking (0.29.0), CLI rules (0.28.0), setup/teardown hooks (0.27.0)
AccountNoneGauge workspace and credits

Most release notes since late October are hosted-side: 0.29.0’s gauge run fork <run-id> --before <seq> replays a recorded hosted run up to a point, then continues live with a changed message or mocked page. If you only use local mode, the free CLI is narrower than the changelog suggests.

What it costs

Local: Gauge charges nothing. Its pricing page says the open-source CLI runs Agent Experience and Agent Preference evaluations “with no Gauge account, subscription, or credits required.” You pay your model provider for every agent session, and a hillclimbing loop is by design the same task run many times.

Hosted Gauge Agents (as of November 12, 2026):

  • Free: 500 starting credits, refilling to 100 credits a week, open models on OpenCode, one onboarding run on Claude Code and Codex, two active runs, two seats.
  • Pro: $499/month for 5,000 credits; Claude Code, Codex, Antigravity, scheduling, five seats.
  • Enterprise: custom volume.

Gauge says each run costs 5 infrastructure credits plus token credits, and that runs average about 20 credits for Agent Experience and about 12 for Agent Preference. On those averages, Pro buys roughly 250 Agent Experience runs a month. Gauge Chat, the AI-search visibility product, is billed separately from $599/month.

Results Gauge reports

All of these are Gauge’s own claims; we have not reproduced any of them.

  • Clerk: per the Show HN post, Gauge helped Clerk move coding agents from a legacy “keyless” mode to the new clerk init command. Clerk validated a pre-release change to its SDK’s error messages with Gauge and then “saw a 3x increase in coding agents using ‘clerk init’ post-deploy.”
  • Gauge itself: the team says agents were reporting that Gauge only supports Claude Code and Codex. They traced that to the web pages the agents fetched and fixed it. Gauge’s own pages still differ slightly today: the CLI README lists four agents, the hosted docs list five.
  • Mux: in the interview, Mux’s Joshua Alphonse says Mux uses Gauge for its prompt library, to steer agents toward the Mux integrations users want.

Community reactions

The Show HN thread (November 4, 2026, 43 points) was small but on point:

  • The token-cost objection. One commenter already does this by hand (try a task, note the friction, document the workaround) and was “extremely leery” of an automated loop “burning tokens on the same task over and over.” The Gauge author replied that the loop doesn’t have to run unattended and mostly automates the manual process, with the agent earning autonomy as you add guardrails.
  • RL environment or test suite? Another asked whether agent experience should look like a hillclimbing environment or like the eval suites Supabase and Next.js publish. Gauge’s answer: active work should be exploratory hillclimbing tied to launches, with eval-style regression tests as a complement for critical features. The follow-up question, where to start in such a large search space, went unanswered in the thread.
  • Measurement. A third asked how Gauge knows a document variant fixed the problem, and what happens when agents disagree. The README’s answer is the paired comparison: fixed task, criteria and model pin, only the input changes.

Honest limitations

  • Not free to run. “No Gauge account required” is true; “free” is not. Every sample is a real Claude Code or Codex session on your bill or subscription quota, and agent behavior is noisy, so one sample per arm proves little. Budget for several samples per variant.
  • Your agent is the analyst in local mode. Gauge collects evidence; the judgment of what went wrong and whether a criterion passed is written by your own coding agent against a schema. That is flexible, but the quality of the conclusion depends on that agent and on you checking its work.
  • Young and moving fast. Public since October 20, 27 versions in the changelog, two contributors on GitHub. 0.26.0 shipped a breaking rewrite of gauge optimizations and removed gauge actions outright; 0.27.0 changed what plain gauge init writes. Pin your CLI version in CI.
  • Rough edges are visible. The one open issue, #23, notes that gauge run --verify rejects --files, so manual-only cases can be previewed and launched but not statically verified.
  • Telemetry is opt-out. The CLI sends the command name, a timestamp, the CLI version and a random installation ID to Gauge’s PostHog project by default. Arguments, prompts and file contents are excluded, per the README. Set GAUGE_TELEMETRY_DISABLED=1 in CI.
  • The best features are hosted. Run forking, CLI rules, authenticated MCP/API testing, PR checks and scheduled measurement are hosted-only. Local mode is a complete loop, but a narrower one.
  • Docker must be local, and hosted runs mean trusting Gauge’s gateway, which terminates TLS and brokers credentials.

Gauge CLI vs the alternatives

OptionWhat it measuresCostBest for
Gauge CLI (local)Agent completion and friction on your product, with transcripts and paired comparisonsFree tool + your tokensDevtool teams iterating on docs, Skills, CLIs
Gauge Agents (hosted)Same, plus automatic judging, PR checks, schedules, forksFree tier; $499/mo ProTeams that want it in CI without running Docker
Published eval suites (e.g. Supabase, Next.js)Pass/fail on fixed tasksYour build timeRegression testing of critical flows
Doing it by handWhatever you noticeYour time + tokensSmall projects, first pass

The closest DIY version is a script that runs claude -p in a container and greps the result. Gauge adds normalized transcripts across four agents, page swapping at the real hostname, and bookkeeping that keeps baseline and candidate batches comparable.

FAQ

Do I need a Gauge account to use the CLI?

No. gauge init --local and local runs work without an account. You need Node.js 22.13+, Docker, and a Claude Code, Codex, Pi or OpenCode login or API key. A Gauge account is only needed for hosted runs.

Which coding agents does Gauge support?

Locally: Claude Code, Codex, Pi and OpenCode. The hosted docs also list Antigravity. You can use native Claude/Codex logins or API keys referenced by environment-variable name through gauge local profiles.

How much does a Gauge run cost?

Locally, Gauge charges nothing; you pay your model provider per session. Hosted, Gauge says an Agent Experience run averages about 20 credits and an Agent Preference run about 12, with Pro at $499/month for 5,000 credits.

Can Gauge test unpublished docs?

Yes. Web fixtures replace pages or route a site to a preview, in both local and hosted runs, so an agent fetching your docs URL sees the draft. For npm CLIs and SDKs, pack a tarball and Gauge installs the pre-release build in the sandbox.

How is this different from an AI search visibility tool?

Visibility tools check whether ChatGPT or Perplexity mentions your product. Gauge checks whether a coding agent can actually install and use it to finish a task, and why it fails when it doesn’t.

Bottom line

Gauge CLI is a well-documented, MIT-licensed way to stop guessing how coding agents experience your product. If you maintain a CLI, SDK, MCP server or Agent Skill, the release-notes example is a small, cheap way to learn whether agents even find your guidance. Go in knowing that every run spends real tokens, that the local analysis is only as good as your own agent’s reading, and that the tool is weeks old and still renaming commands. Start local, keep the samples few and the tasks concrete, and pin the version.

Sources