TL;DR

kimi-k3-in-c is a from-scratch inference engine, written in portable C99, that runs Moonshot AI’s Kimi K3 — a 2.78-trillion-parameter model — on a single CPU in as little as 8.24 GB of RAM, with no BLAS, no framework, and no GPU. Published by Fareed Khan on August 1, 2026, it hit the GitHub trending front page within days. Highlights:

  • 675× smaller footprint than the naive bfloat16 requirement (5.56 TB → 8.24 GB), with byte-identical output at every memory budget in between.
  • The whole engine is 176 KB of C across seven source files — libm and OpenMP are the only dependencies.
  • The trick is mixture-of-experts sparsity: only 16 of 896 experts fire per layer, so 93% of the checkpoint (1.447 TB of routed experts) never has to be resident. It’s read from NVMe on demand.
  • Memory becomes a dial, not a floor: the same model runs in 8 GB, 128 GB, or 224 GB. More RAM only changes the clock, never the answer.
  • The honest catch: at the 8 GB laptop preset you get roughly 0.03 tokens/second (~32 s/token). It’s a proof of engineering, not a daily driver.
  • Apache-2.0 licensed, Linux x86-64 only (uses O_DIRECT, posix_memalign, getrusage), and requires ~1.7 TB of free disk for the checkpoint.

If you’ve ever wanted to understand how a frontier MoE model actually fits — or doesn’t — on real hardware, this is one of the clearest teaching artifacts published all year.

What kimi-k3-in-c actually is

Kimi K3 is a 2.78-trillion-parameter mixture-of-experts model. As shipped from HuggingFace it’s 1.56 terabytes across 96 safetensors shards. No consumer machine can hold that, and — this is the key insight the project hammers home — waiting for a faster machine doesn’t help, because the wall isn’t compute, it’s capacity. You can’t run what you can’t fit.

kimi-k3-in-c is a clean-room C99 engine that runs this exact checkpoint, unmodified, on one CPU. It does not quantize the model down, distill it, or drop layers. Instead it exploits the one structural property of an MoE model that makes the impossible tractable: most of the model is asleep for any given token.

The headline run from the README says it all:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
           --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

--- generated text ---
 Paris.",
+            "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS for the whole run: 8.24 GB

Correct answer, 8.24 GB of RAM, from a checkpoint 190× larger than the memory it ran in. (It’s a base model with no chat template, so what follows “Paris.” is a raw continuation, not a chat reply.)

The four reductions: how 5.56 TB becomes 8.24 GB

The project’s real value is that it shows its work. The whole thing is one ledger of four steps, and it’s worth understanding because it generalizes to every large MoE model:

StepMemoryWhy
Naive5,560 GBEvery parameter resident at bf16
As shipped1,560 GBThe routed experts already arrive at ~half a byte per weight (4-bit + a shared scale)
Routing113.49 GBOnly the always-active trunk needs to be resident; experts never load
Streaming8.24 GBThe dense trunk is streamed from disk instead of held in RAM

That’s a 675× reduction from the bfloat16 model and 189× from the shipped checkpoint — and nothing is approximated. The output at the bottom of the ladder is byte-for-byte identical to the output at the top.

Reduction one: the experts already ship small

Kimi K3 has 93 layers. Layer 0 is dense; the other 92 route, each picking the top 16 experts out of 896. A shard census in the repo counts the actual bytes:

routed experts    : 82,432   (896 routed x 92 MoE layers)
bytes per expert  : 17,547,264  exactly
routed expert set : 82,432 x 17,547,264 = 1.447 TB

Those 82,432 routed experts are 1.447 TB — 93% of the entire checkpoint. They already arrive at roughly half a byte per weight (4-bit nibbles plus a shared E8M0 scale), and the engine multiplies them straight out of that packed form without unpacking to float first.

Reduction two: routing means most of it never loads

For any single token, only about 104 billion of the 2.78 trillion parameters are active — 3.7%. The other 96.3% still has to exist somewhere reachable, but it does not have to be in RAM. Strip the experts out of the resident set and you’re left with the “always-on” trunk: 113.49 GB (108.81 GB of dense per-layer weights + 4.70 GB of embeddings and the output head).

Reduction three & four: streaming turns a floor into a dial

The final move is the clever one. The 93 dense layers get repacked (once, ~4 minutes) into a single 109 GB trunk.bin file where layer L lives at a known offset and can be read in one call. From there the engine pins as many layers as your memory budget allows and streams the rest from NVMe, keeping a small ring buffer plus an LRU cache for the routed experts.

The consequence is remarkable: the same model runs in 8 GB and in 224 GB and produces byte-identical output at every budget in between. Memory stops being a hard floor and becomes a performance dial. Give it more RAM and the answer doesn’t change — only the wall-clock time does:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
           --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental

28 tokens in 299.3 s, 10.69 s/token average
PEAK RSS for the whole run: 127.92 GB

Same engine, more memory pinned, ~3× faster per token.

Trying it yourself

You can verify the entire engine in about a minute with no checkpoint, no network, and no Python — this is genuinely the best part of the developer experience:

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c

make -j            # seconds. Seven C files, a compiler and OpenMP
make test          # under a minute

That runs a full end-to-end oracle against a 13-layer model built with the same tensor graph as the real one, checked against a committed PyTorch reference. It ends with:

GATE 1  teacher forcing : 32/32 positions match tf_pred
GATE 2  greedy decode   : 20/20 generated tokens match full_ids
GATE 3  incremental     : 20/20 generated tokens match full_ids

VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY
ALL WEIGHTLESS TESTS PASSED

Actually running the real model is a bigger commitment. The gate is storage: the checkpoint is 1.56 TB, plus ~109 GB for the packed trunk, so budget ~1.7 TB of free (ideally NVMe) disk. The full path:

./scripts/k3-doctor.sh                       # checks toolchain, sizes RAM to a preset, measures disk
make -j                                       # build
make test                                     # verify BEFORE the 1.56 TB download
export HF_TOKEN=hf_your_token_here
./scripts/download-model.sh ~/k3model         # 1.56 TB, resumable, hours not minutes
./scripts/pack-trunk.sh ~/k3model ~/k3trunk   # ~4 minutes, once → the 109 GB trunk.bin
./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
         --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

Two flags matter more than the rest:

  • --trunk is what enables streaming. Without it the engine loads all 113.5 GB of trunk resident regardless of the preset you asked for — so --preset laptop without --trunk does nothing useful.
  • --incremental carries the KV cache and recurrent state between tokens. Without it, every step re-runs the entire prefix (O(T²)); with it, you pay for the prompt once. Both paths are gated to produce identical tokens, so it’s a pure speed choice — always pass it.

The download script is paranoid in the right way: it verifies all 96 shards individually against published byte sizes, because “a partial download does not fail loudly; it produces wrong tokens.” Per-shard checking turns “re-download 1.56 TB” into “re-download this one 17 GB file.”

Requirements, in plain terms

OSLinux, x86-64 only (O_DIRECT, posix_memalign, getrusage)
CPUAVX2 + FMA. AVX-512 is not required
RAM8 GB and up — every preset works, more is faster not different
Storage~1.7 TB free, ideally fast local NVMe
ToolchainGCC ≥ 9 or Clang ≥ 10, GNU make or CMake
Python3.9+ only for the download/pack/analysis tools, not for make test

Presets (laptop · desktop · workstation · server · max) are just shorthand for two numbers: the trunk budget and the expert-cache budget. All measurements in the docs come from one two-socket EPYC 7763 (124 cores, 228 GB RAM, 3.2 TB NVMe) — notably with four L40 GPUs that sat completely idle, because there is no GPU path at all.

Community reactions

The reception has been a mix of genuine awe at the engineering and clear-eyed skepticism about practicality:

  • The dominant Hacker News reaction was practical skepticism — and a fair one. At roughly half a token per second (or slower), a medium-length answer takes hours, and several commenters noted that paying for the Kimi K3 API costs less than the electricity alone for a long local run.
  • On r/LocalLLaMA, the announcement thread drew the opposite energy: local-inference enthusiasts treated the 8 GB figure as a milestone in itself, independent of throughput.
  • A recurring, more nuanced take: the value here isn’t the 8 GB laptop run, it’s that storage — not RAM or compute — is now the real bottleneck for frontier MoE inference. The engine makes that visible and measurable.
  • The write-up itself (Part II builds every component from scratch) is being shared as a teaching resource on how MoE routing, MLA attention, and 4-bit expert packing actually work at the byte level.

Honest limitations

Let’s be blunt about who this is and isn’t for:

  • It is slow. ~0.03 tok/s at the 8 GB preset is not usable for interactive work. Even at the 128 GB server preset you’re at ~0.09 tok/s. This is a feasibility demonstration, not a chatbot backend.
  • The economics are upside down for casual use. For most people, the hosted Kimi K3 API is faster and cheaper than running it locally on a slow CPU path — the electricity for a multi-hour generation often exceeds the API cost.
  • The storage requirement is the real gate. ~1.7 TB of fast local disk is a bigger ask than 8 GB of RAM. Spinning rust will make the streaming path miserable.
  • Linux x86-64 only. No macOS, no ARM, no Windows. The O_DIRECT and AVX2 assumptions are baked in. Apple Silicon users are out.
  • It’s a base model. No chat template ships with it, so out of the box you get raw continuations, not assistant-style replies.
  • Version 0.1.0. It’s days old. Treat it as a reference implementation and a learning tool, not production infrastructure.

Where it genuinely shines: understanding. If you want to know precisely how 2.78 trillion parameters map onto real memory — and where every byte lives — there is no clearer artifact right now.

FAQ

Can I really run a 2.78T model on my laptop? If your laptop is Linux x86-64 with AVX2, 8 GB of RAM, and — critically — ~1.7 TB of free fast disk, yes. But expect roughly 32 seconds per token. It works; it just isn’t practical for real use.

How is the output identical across memory budgets? Because nothing is approximated. Streaming only changes where weights live (RAM vs. NVMe), not their values. The engine gates every path (teacher forcing, greedy decode, incremental) to produce byte-identical token IDs, so 8 GB and 224 GB emit the same text — the larger budget is just faster.

Why is it so much smaller than the 1.56 TB checkpoint? Two reasons stack. First, 93% of the checkpoint is routed experts that ship at ~4 bits/weight and are streamed from disk, never resident. Second, the always-active 113 GB trunk is also streamed rather than held. Only a working set stays in RAM.

Is this faster than llama.cpp or Ollama? No — and that’s not its goal. Those run smaller models fast on hardware that fits them. kimi-k3-in-c runs a model that doesn’t fit, trading speed for the ability to run at all. Different problem entirely.

Should I use this in production? No. It’s a v0.1.0 reference/teaching implementation. Use the hosted API for real workloads. Use this to understand MoE inference, verify claims yourself, or as a foundation to build on.

Sources

kimi-k3-in-c is Apache-2.0 licensed. All figures cited come from the project’s measured data in docs/data/. Verified against the repository on August 4, 2026.