bitnet.cpp Review: Microsoft's 1-Bit LLMs on a Plain CPU

TL;DR

bitnet.cpp (microsoft/BitNet) is Microsoft’s official inference engine for “1-bit” language models. These are models whose weights take only three values: -1, 0 and +1. It is a fork of llama.cpp with its own CPU kernels. It runs Microsoft’s BitNet b1.58 2B4T model, two new 1-bit embedding models, and a few community ternary models on an ordinary laptop or server CPU. You do not need a GPU.

Facts as of October 13, 2026:

41,069 GitHub stars, 3,821 forks, MIT license. The repo is on GitHub’s daily trending list today.
No tagged releases. The README badge says version 1.0. The last commit on main (0b341e5, July 27, 2026) added a news item. The last code change landed on July 16.
215 open issues and 158 open pull requests. Five of those PRs were opened today.
Official models: BitNet-b1.58-2B-4T (text generation), BitNet-embedding-0.6B and BitNet-embedding-270M (released July 20, 2026).

Not run by us: our hands-on lab could not start today because the GitHub Actions job behind it was blocked by a billing limit on the account. Everything below comes from the repository, the model card, the papers and public bug reports. We did not build bitnet.cpp or generate text with it.

Our verdict: the idea is real and the published numbers are strong. A 2-billion-parameter model whose non-embedding weights fit in 0.4 GB, and which scores close to Qwen2.5-1.5B, is a genuine result. But the repository has drifted since July. Several independent reports say the current main gives the flagship model the wrong activation function, and that this quietly damages output quality. ARM and macOS builds also have open breakages. Treat bitnet.cpp today as a research artifact that you pin and patch. It is not yet a drop-in replacement for llama.cpp or Ollama.

What “1-bit” actually means here

The name is a little loose. BitNet b1.58 stores each weight as one of three values {-1, 0, +1}. Three states need log2(3) ≈ 1.58 bits, which is where “b1.58” comes from. Activations are 8-bit integers. The model card calls this W1.58A8.

The important detail is in the model card: the model was trained from scratch with this quantization, not compressed after training. You cannot squash Llama or Qwen to ternary weights and expect the same quality. The card lists BitLinear layers, squared ReLU (ReLU²) in the feed-forward layers (this matters later), the LLaMA 3 tokenizer, a 4,096-token context and 4 trillion pre-training tokens followed by SFT and DPO.

Ternary weights make inference cheaper because a matrix multiply against {-1, 0, +1} turns mostly into additions, subtractions and skips. bitnet.cpp builds on the lookup-table methods from Microsoft’s earlier T-MAC project. It ships three kernel types: I2_S (2-bit packing, x86 and ARM), TL1 (lookup table, ARM) and TL2 (lookup table, x86).

What Microsoft claims

The README and the October 2024 technical report report these figures. We did not reproduce any of them:

Claim (project-reported)	ARM CPUs	x86 CPUs
Speedup vs full precision	1.37x – 5.07x	2.37x – 6.17x
Energy reduction	55.4% – 70.0%	71.9% – 82.2%

The project also says that a January 2026 kernel update (“parallel kernel implementations with configurable tiling”) adds another 1.15x to 2.1x on top of that.

For the 2B4T model, the model card compares it with instruction-tuned models of similar size:

	LLaMA 3.2 1B	Gemma-3 1B	Qwen2.5 1.5B	SmolLM2 1.7B	BitNet b1.58 2B
Memory (non-embedding)	2 GB	1.4 GB	2.6 GB	3.2 GB	0.4 GB
CPU decode latency	48 ms	41 ms	65 ms	67 ms	29 ms
Energy (estimated)	0.258 J	0.186 J	0.347 J	0.425 J	0.028 J
MMLU	45.58	39.91	60.25	49.24	53.17
GSM8K	38.21	31.16	56.79	45.11	58.38
Average (18 benchmarks)	44.90	43.74	55.23	48.70	54.19

So the claim is not “best small model”. It is “about as good as the best 1.5–2B model, at a fraction of the memory and energy”. Qwen2.5-1.5B still wins on average, MMLU and coding (HumanEval+ 50.6 vs 38.4).

The 100B headline

The README says bitnet.cpp “can run a 100B BitNet b1.58 model on a single CPU” at 5–7 tokens per second. This line causes the most confusion. No trained 100B BitNet model exists for you to download. In a review of the paper published yesterday, Wavect points out that Appendix A of the technical report describes the 100B result as a dummy configuration. The best figure is 6.58 tokens per second on a Mac Studio with an M2 Ultra and 64 GB RAM. With two threads it drops to 1.27 tokens per second. On a Surface Laptop Studio 2 (Core i7-13700H) the best figure is 1.70 tokens per second.

That is a kernel throughput benchmark of what the engine could do if someone trained a 100B ternary model. The README even ships utils/generate-dummy-bitnet-model.py for this kind of random-weight test. The largest real model you can run today is about 10B (Falcon3-10B-1.58bit).

What the 0.4 GB means

The 0.4 GB figure excludes embeddings. The official GGUF file, ggml-model-i2_s.gguf, is 1,187,801,280 bytes (≈1.19 GB) on Hugging Face. Plan for that download, plus the KV cache, plus runtime buffers. Wavect calculates about 300 MiB of KV cache for a full 4,096-token context with the default F16 cache. That is still very small next to a 4-bit 7B model, but it is not 400 MB.

Installation (from the README)

You need Python 3.10 or later, CMake 3.22 or later and Clang 18 or later. GCC alone is not enough. On Windows you need Visual Studio 2022 with the Clang toolset.

git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

conda create -n bitnet-cpp python=3.10
conda activate bitnet-cpp
pip install -r requirements.txt

# Download the official GGUF, then generate kernels + build + prepare the model
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

setup_env.py generates kernel code for your CPU, builds with CMake and clang, and converts the model if no GGUF is present. Build logs go to logs/, not to your terminal, so if something fails, read logs/compile.log.

Then you run inference:

# Chat mode: -p becomes the system prompt
python run_inference.py \
  -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "You are a helpful assistant" -cnv

# One-shot completion, 4 threads, 128 tokens
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "Explain ternary weights in two sentences." -n 128 -t 4 -temp 0

run_inference.py is a thin wrapper. It calls build/bin/llama-cli with -ngl 0, which forces CPU only. The default thread count is 2, so set -t to your core count if you want useful speed. For speed numbers on your own machine, use the benchmark script:

# 200 generated tokens, 256 prompt tokens, 4 threads
python utils/e2e_benchmark.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -n 200 -p 256 -t 4

If you are on Ubuntu 24.04, read issue #480 first. setup_env.py hard-codes the unversioned clang and clang++ names. The LLVM install script only creates clang-20-style binaries, so CMake fails until you symlink them. The const-correctness compile error from issue #407 is already fixed on main.

Do not use the Hugging Face transformers path if you want speed. The model card says this directly: transformers has no BitNet kernels, so speed and energy will be “comparable to, or potentially worse than” full-precision models. The gains only exist in bitnet.cpp.

The problem: current `main` has a quality regression

This is the most important section for anyone who wants to try bitnet.cpp today.

In July 2026, Microsoft moved the llama.cpp submodule to a newer fork (isHuangXin/llama.cpp, commit 390c3077). The goal was to support the new embedding models. According to issue #588, opened July 19, the newer llama.cpp code merged two BitNet graph builders into one file. The surviving builder uses SiLU in the feed-forward layer. BitNet b1.58 was trained with ReLU².

Nothing crashes. The model still produces plausible-looking text. But the reporter measured wikitext-2 perplexity of 99.8 on stock main, against 17.1 with a one-line fix. The official reference value is 17.109. A follow-up comment on the same issue measured 19.60 with an older BitNet commit (765741d, whose submodule still used ReLU²) on the same published GGUF file. Since then, people have confirmed the problem on Intel, AMD Zen 2 and Zen 4, Ivy Bridge and Apple M2 Pro.

The community fix in the issue is one line in 3rdparty/llama.cpp/src/models/bitnet.cpp:

-                LLM_FFN_SILU, LLM_FFN_PAR, il);
+                LLM_FFN_RELU_SQR, LLM_FFN_PAR, il);

Why is it not merged? A commenter quotes a llama.cpp code owner: Microsoft’s BitNetForCausalLM and 1bitLLM’s BitnetForCausalLM share one architecture name, and the 1bitLLM models really do use SiLU. A blanket change fixes one family and breaks the other. The thread agrees the right fix is per-model metadata; nobody from Microsoft has shipped it.

One more report in the same thread, from October 10, adds a second problem. The published GGUF has no tokenizer.ggml.pre field, so the loader falls back to a default pre-tokenizer. On that commenter’s own text sample, perplexity went from 381 (stock) to 276 (tokenizer override only), 90 (activation fix only) and 49 (both fixes). That suggests you need both fixes, not one.

Other open bugs you should know about:

ARM / NEON: issue #600 reports garbage output (all ? characters) from the official 2B4T GGUF on ARM64 CPUs without AVX2, including Ampere- and Graviton-class servers. The reporter blames a wrong I2_S unpacking scheme in the scalar fallback.
Apple Silicon: issues #611, #618 and #641 report linker errors and an undeclared variable during setup_env.py builds on M-series Macs. After workarounds, one reporter says Metal crashes unless you pass -ngl 0 and CPU output is still @@@@.
Conversion: issue #619 says the README’s “convert from .safetensors” flow fails on main. Issue #621 says Falcon3 conversion with --outtype i2_s silently writes full-size F16 tensors. Issue #622 says Falcon-E models no longer load because of an unknown pre-tokenizer.

The last merged pull request was on July 27. Community fixes keep arriving (five new PRs today alone), but nobody is merging them.

Community reaction

The March 2026 Hacker News thread (370 points) shows what developers think. The top comment noted that the “hundred billion parameter” headline does not match any official model, which are all under 10B. One commenter said, “Framework is ready. Now we need someone to actually train the model,” and asked why Microsoft had not trained a large one itself to prove the idea. Others defended the work as normal research practice: validate on small models first, then scale.

One commenter estimated that an N-parameter 5-bit model should be roughly comparable to a 3N-parameter ternary model, so a 100B ternary model “would be within the ballpark of a 30B.”

On r/LocalLLaMA, a February 2026 post reported 36.94 tokens/s and about 1,300 MB of RAM for BitNet b1.58 2B4T on CPU; treat it as one data point.

Where bitnet.cpp is going

The July releases point toward small, always-on workloads rather than chat. BitNet-embedding-0.6B and 270M have x86-only I2_S kernels; the project reports 1.42x–2.28x and 1.32x–1.74x prefill speedups over F16 on 8 threads. Issue #640 says their GGUFs declare MEAN pooling while the model card says last-token, so check that before you trust the vectors. VibeASR.cpp applies the same I2_S kernels to speech recognition on CPU. A separate CUDA kernel lives in gpu/.

How it compares

	bitnet.cpp	llama.cpp / Ollama (Q4)	MLX on Apple Silicon
Model choice	~15 ternary models, max ~10B	Thousands of GGUFs	Thousands
Memory per parameter	Lowest (~2 bits packed)	~4.5 bits	~4 bits typical
Quality at 2B	Close to best 1.5–2B models (when correctly configured)	Depends on model	Depends on model
Install	Build from source, Clang 18+	One binary or `ollama pull`	`pip install`
Maintenance (Oct 2026)	No merges since July 27	Very active	Very active

On a Mac, an MLX-backed runner is the practical choice today (see our Ollama MLX review). bitnet.cpp makes sense for ternary-inference research, edge-hardware sizing, or x86 servers where every watt counts. Even then, pin a commit and patch it.

Honest limitations

No 100B model and a thin model zoo: ternary models must be trained from scratch.
Silent quality regression on main (#588), plus ARM and Apple Silicon bugs.
Research posture. The model card says Microsoft does “not recommend using BitNet b1.58 in commercial or real-world applications without further testing and development.” It also notes limited non-English support.
Maintenance. 215 open issues, 158 open PRs, no releases and no code merges for nearly three months.

FAQ

What is bitnet.cpp?

It is Microsoft’s open-source (MIT) inference engine for 1.58-bit “ternary” language models such as BitNet b1.58 2B4T. It is built on llama.cpp and adds special CPU kernels (I2_S, TL1, TL2) that exploit weights limited to -1, 0 and +1.

Can bitnet.cpp really run a 100B model on a single CPU?

Only as a benchmark. The 100B result in the technical report uses a dummy model with random weights. The project reports 6.58 tokens/s on an M2 Ultra in that setup. No trained 100B BitNet model has been released. The largest real ternary models you can run are about 10B.

How much RAM does BitNet b1.58 2B4T need?

The official GGUF file is about 1.19 GB. The 0.4 GB figure in the model card counts only non-embedding weights. Add the KV cache (roughly 300 MiB at the full 4,096-token context with an F16 cache, per Wavect’s calculation) and runtime overhead. A machine with 4 GB of free RAM is comfortable.

Is BitNet b1.58 as good as normal small models?

According to Microsoft’s own benchmarks, it averages 54.19 across 18 benchmarks, against 55.23 for Qwen2.5-1.5B and 44.90 for LLaMA 3.2 1B. It uses much less memory and energy than either. Those numbers assume the correct ReLU² activation. Independent reports say current main uses SiLU, which raises perplexity sharply.

Why is my bitnet.cpp output poor or full of `@` or `?` characters?

There are three known causes. The ReLU²/SiLU regression in the July 2026 submodule (#588) degrades output everywhere. A missing pre-tokenizer field in the published GGUF also hurts quality. On ARM without AVX2 and on Apple Silicon, separate kernel bugs (#600, #641) can produce garbage. Pin an older commit or apply the community patches from those issues.

Verdict

bitnet.cpp proves something worth knowing: a model trained with ternary weights can come close to the best 2B-class models while using a fraction of the memory and energy. With 41,000 stars, it is the reference implementation for that idea. But “reference” now has an asterisk. The July submodule update appears to have broken the flagship model’s activation function, Apple and ARM builds are fragile, and fixes wait in a queue of 158 open PRs. Read it, benchmark it and learn from it. Pin a known-good commit before you trust its answers.