billion-context Review: Context Compression for AI Agents

TL;DR

billion-context (ranxianglei/billion-context) is a local proxy that sits between your coding agent and the model API. It gives the model four extra tools (compress, decompress, search_context, acp_status) and lets the model itself fold finished parts of the conversation into short summaries. Old work is not thrown away: each summary block can be restored on demand. The goal is a single agent session that runs for days or weeks inside a 100K–200K window instead of hitting the limit and getting a lossy /compact.

Facts as of October 18, 2026:

1,812 GitHub stars, 147 forks, 30 contributors. GitHub Trending lists it with 888 stars this week.
Created August 6, 2026. Latest release: v0.1.200, published today. The project shipped three releases in the last two days.
158 open issues (1,611 filed in total).
License: MIT plus an extra attribution term (more on that below). GitHub shows it as “NOASSERTION” because of the addition.
Installs from npm as billion-context; needs Node.js 20 or newer; the published package has a single runtime dependency (zod).

Not run by us: our hands-on lab (a throwaway GitHub Actions container) could not start today because GitHub refused to start the job over a billing limit on the account. Everything below comes from the README, the project’s own research paper, the license file and the public issue tracker. We did not install billion-context or route any agent through it.

Our verdict: the idea is the best-argued answer to context bloat we have read this year. Compressing only finished work, keeping the cache prefix intact and letting the model restore what it folded beats one-shot summarization on paper, and the author’s own paper is unusually frank about where the savings come from. But “5x fewer tokens” is a best case from 1M-window marathon sessions, the evidence comes from one heavy user, the code changes several times a day, and the tool rewrites your model traffic and auto-updates itself by default. Use it for long agent sessions on a machine you control, pin a version, and turn auto-update off.

What problem it solves

Long coding sessions fill the context window with tool output: test logs, file reads, grep results, diffs. Agents deal with this in one of two ways. Some drop old messages (a sliding window). Most, including Claude Code and Codex, wait until the window is about 80% full and then summarize the whole history into one block. That summary is lossy, and it is repeated every time the window fills again, so early decisions get summarized, then summarized again, and slowly disappear. The paper cites outside work on this “compaction cliff”: in one study, safety rules survived one round of /compact-style summarization 53% of the time and five rounds only 10% of the time.

billion-context changes three things:

The model decides what to fold, and when. When the conversation has grown by about 50K tokens since the last fold, the proxy adds a short instruction with a list of candidate message ranges and their real token sizes. The model calls compress on ranges it has finished with. It may refuse mid-debugging and do it later.
Only the new increment is folded. Each piece of history is summarized once and then stays as it is. Older summaries can be folded again into a second or third tier, but in the author’s data that touched only 60 of 1,711 blocks.
Nothing is deleted. Folded ranges stay in the proxy’s store. decompress brings a range back verbatim, and search_context searches summaries and visible messages by keyword.

Because folds only replace ranges after the stable start of the conversation, the provider’s prompt cache keeps hitting. The README treats a 95–97% cache-hit rate as a healthy session and says compression itself costs at most 2%.

How it works

Agent (Claude Code / Codex / OpenCode / pi / Aider ...)
   │  base URL points at the proxy
   ▼
billion-context proxy
   1. parse the request (Anthropic, OpenAI or Google shape)
   2. run acp-kernel compression on the conversation
   3. inject the compress tool + compression instructions
   4. forward to the real model API
   5. rewrite the streaming response
   ▼
real model API

The compression engine is a separate package, acp-kernel. The proxy keys its state on the session ID the client already sends (x-claude-code-session-id for Claude Code, session_id for Codex, x-opencode-session-id for OpenCode). Clients that send none, such as plain pi, are matched by their message history. State lives in ~/.local/share/billion-context/, logs in ~/.local/state/billion-context/bili.log.

Install and first commands

The README’s install puts the bili command in ~/.local/bin without sudo:

npm install -g billion-context --prefix=~/.local
export PATH="$HOME/.local/bin:$PATH"   # if ~/.local/bin is not on PATH yet

Then pick one of three ways to attach it:

# 1. Launcher: starts a fresh proxy and the client together, edits no config
bili claude
bili codex
bili opencode

# 2. Native plugin: bili becomes part of the client; start the client as usual
bili plugin install opencode   # also turns off OpenCode's own auto-compaction
bili plugin install claude     # managed settings block + SessionStart hook
bili plugin remove opencode    # undo; original config was saved to .bili-bak

# 3. Plain URL change: run the proxy and prefix any client's base URL
bili
# before: https://api.openai.com/v1
# after:  http://localhost:8787/bili/https://api.openai.com/v1

Claude Code users can also install it from inside the client, because the repository doubles as a plugin marketplace:

/plugin marketplace add ranxianglei/billion-context
/plugin install billion-context@billion-context
/billion-context:bili-setup

To check it is alive, the README gives two endpoints:

curl -s http://localhost:8787/__bili/health
# → {"ok":true,"upstream":"https://api.anthropic.com"}
curl -s http://localhost:8787/__bili/stats

A web dashboard at http://localhost:8787 shows sessions, logs, a config editor and a cache-hit chart. Inside clients with the native plugin, /acp shows what is folded and /acp-cache prints a cache report that blames each miss on a cause: provider cache expiry, a model switch, a bili bug, or unknown.

Supported agents

The README lists more than 20 clients, but the depth of support varies a lot:

Client	Mode	What to know
OpenCode, pi, omp, dsh, Kimi, Hermes, ZCode	Native plugin	Best integration; `/acp` panel in-client
Claude Code	Launcher or native	Native mode sets `DISABLE_AUTO_COMPACT=1`. The Claude Desktop “Code” tab overrides the proxy URL, so desktop sessions get neither bili nor built-in compaction (issue #2290)
Codex	Launcher (full) or MCP tools only	A plugin install cannot route Codex’s own traffic; use `bili codex`
Copilot CLI, Amp, Crush, Zed, Aider, Qwen Code	Launcher only	Traffic is intercepted with a local certificate (TLS man-in-the-middle)
Gemini CLI, iFlow, Antigravity	Launcher only	Base-URL rewrite, no in-loop tool seam
Goose	Launcher only	Cannot be MITMed; rides base-URL overrides

“cert-MITM” deserves a plain explanation. For closed binaries that cannot change their API URL, the launcher starts a proxy with its own certificate authority and tells only the child process to trust it. The proxy then decrypts traffic to the whitelisted model hosts and passes everything else through as a blind tunnel. That is a reasonable design, but it means a third-party Node process sees every prompt, every file the agent reads, and your API key in transit.

The numbers, read carefully

The repository ships a long research paper, Model-Driven Incremental Hierarchical Compression (v0.2, MIT-licensed, in paper/). It is more careful than the tagline, and the tagline needs that care. All figures below are the author’s.

Production use. Over 4.5 months on three machines, the author reports 174,327 model calls and 18.76B cumulative input tokens on the main host (about 24.7B across all three), with no request ever exceeding a 204,800-token window. The longest sessions ran 8,584 and 12,049 calls on 1M-token models at roughly 120K average context. On the pi host, measured context per call was a mean of 88K and a median of 72K. Heavy sessions with around 900K tokens of history ended up holding about 13K tokens of summaries.

“Billions of tokens” means cumulative input, re-counted on every call, about 90% of it cache reads (17.01B of 18.76B) — not a billion distinct tokens.

Where “5x” comes from. The paper compares the leading 1M-window session (126K average context) with what a normal 80%-threshold compaction would have held (about 670K average) and gets a modeled ratio of about 5.3x. On 200K windows the same model predicts 1.2–1.8x. In the author’s words, the advantage there is “zero violations and unbounded session length, not raw savings.” The 80% trigger is an assumption, not something measured at 1M scale.

Controlled pilot. Against a sliding-window baseline, billion-context configurations cut billed tokens by 31–54% with recall probes scoring the same, and calls to re-read already-seen content dropped 63%. The paper then admits that a control arm that never compressed anything, just added the instructions and message tags, got almost the same re-read drop (−62%). Two more honest findings: a dumb fixed-cadence policy was 37% cheaper on repetitive work (the default won on phase-structured work), and at a 16K window plain one-shot compaction was the cheapest arm, because bili’s ~2K-token instruction block costs about 25% overhead per call until windows reach 64K or more.

Cache. The paper reports a 94.2% cache-read share when calls are under two minutes apart, 90.7% blended.

Limits the paper lists itself: a single user, observational data with no randomized deployment, savings partly modeled, controlled arms run on one model (Qwen3.8-27B) at temperature 0, probe suites near saturation, English-centric instructions, and summary quality not yet measured. The 28.6B-token “single-session ceiling” is a simulation extrapolated about 19x beyond the largest observed session.

Community reaction

Attention is real but narrow. We found no Hacker News discussion. Uptake shows up instead in agent ecosystems: the package is listed on pi.dev and in the dsh plugin registry, several people maintain public forks, and the project runs two QQ chat groups (one full). The issue tracker is the loudest signal: 1,611 issues in about ten weeks, many bilingual English and Chinese, many filed with detailed measurements.

Today’s open issues show what running it is like. #3691: emergency truncation at 95% usage flips between full and clipped history and drops the cache-hit rate to 15%. #3661: a malformed compress call is echoed back, the upstream returns HTTP 400, and the nudge re-fires every round. #3671: compressed source text is never freed and made up 78% of one session’s archive. These are the bugs of a sophisticated system under heavy daily use, in the layer that touches every request your agent makes.

Honest limitations

It moves very fast. v0.1.200 in ten weeks. Pin a version and read the release notes before upgrading.
Auto-update is on by default. The proxy checks npm at startup and every three minutes and installs newer versions in place (now spread over a random 15-minute to 48-hour window per machine). Even with auto-update off, it polls an advisories package that can recommend a rollback. Disable it with --no-auto-update, ACP_AUTO_UPDATE=0 or "autoUpdate": false.
It sees everything. Prompts, code, tool output and API keys pass through it. Keep it on 127.0.0.1 (the default). --host 0.0.0.0 has no authentication.
Savings depend on window size and session length. On a 200K window the author’s own model predicts 1.2–1.8x, and short sessions see almost nothing: 62% of all compression happened in the 1.3% of sessions with more than a thousand messages.
Small windows pay overhead. Below about 64K, the instruction block costs more than it saves.
Restoring is not automatic. The paper notes that models “do not yet reach for” decompress unprompted. Folded detail is restorable, but someone has to ask.
Mixed signals on maturity. One section of the README still says “Early… Real-model integration testing is the next milestone”, while the paper reports months of production use.
The license is not plain MIT. Any product or service that users can see and that uses billion-context or acp-kernel must credit it with a link on its home page, docs or About page. Fine for personal use; check with legal before shipping it inside a product.

FAQ

What is billion-context?

An open-source context-compression proxy for AI coding agents. It sits between the agent and the model API, adds compress, decompress, search_context and acp_status tools, and lets the model fold finished conversation ranges into restorable summaries so one session can keep going inside a 100K–200K window.

Does it really cut tokens by 5x?

Only in the best case the author modeled: very long sessions on 1M-token windows, compared with an assumed 80%-threshold compaction. The controlled pilot measured 31–54% fewer billed tokens, and the paper predicts 1.2–1.8x on 200K windows. Short sessions see little benefit.

How is it different from Claude Code’s /compact?

/compact summarizes the whole history into one block when the window fills, and repeats that every time, so early details decay with each round and the prompt cache is rebuilt. billion-context folds small finished ranges, each only once, keeps the cache prefix intact, and can restore any folded range verbatim.

Is it safe to run?

It is a local process that reads all your agent’s traffic, including API keys, and updates itself from npm by default. Run it on localhost only, pin a version, turn off auto-update, and use bili --passthrough to rule it out when debugging. For closed clients it uses a local certificate authority trusted only by the launched process.

How does it compare to Headroom or Context Mode?

Those tools mostly shrink tool output before it enters the context. billion-context manages the conversation history itself after the fact. They work on different layers; if you combine them, make sure only one thing compresses history, or you get double compression, which the project warns about for its own plugins.

Verdict

billion-context attacks the right problem with the right shape: compress only what the task has finished with, never rewrite the cached prefix, and keep everything restorable. Its paper is the rare project document that reports its own control arm reproducing most of its headline effect. That honesty is the reason to trust the parts it does claim: bounded context over months of real use and a cache that keeps hitting.

The reasons to hold back are practical: big savings only for marathon sessions on large windows, evidence from one expert user, and daily changes in the most sensitive spot in your stack. If long OpenCode, pi or Claude Code terminal sessions keep hitting compaction, try bili <client>, pinned and with auto-update off. If your sessions are short, you will mostly pay its instruction overhead.