TL;DR

oMLX is an open-source LLM inference server for Apple Silicon that solves a specific, badly under-served problem: local coding agents grind to a halt because every MLX server throws away its KV cache the moment a prompt prefix shifts. oMLX persists KV blocks across a hot RAM tier and a cold SSD tier, so returning context is restored from disk instead of recomputed — even after a server restart.

Key facts (verified 2026-08-20):

  • 19,977 GitHub stars, 1,701 forks, ~1,100 stars gained this week
  • Apache 2.0, Python, created 2026-02-13 — roughly six months old
  • Latest release: v0.6.3rc1 (2026-08-19) — very active, ~984 open issues
  • Tiered KV cache: hot in-memory + cold SSD blocks in safetensors format, survives restarts
  • Continuous batching via mlx-lm’s BatchGenerator (default 8 concurrent requests)
  • Both OpenAI and Anthropic APIs — drop-in backend for Claude Code, OpenCode, Codex, Cursor
  • Multi-model serving: LLM + VLM + OCR + embeddings + reranker in one process, with LRU eviction, pinning, and per-model TTL
  • Native Swift/SwiftUI menu bar app (not Electron) plus a full web admin dashboard
  • Requires macOS 15.0+ (Sequoia), Python 3.11–3.13, Apple Silicon (M1/M2/M3/M4)

The author’s own framing, from the MLX project discussion: coding agents were taking 30–90 seconds per response a few turns into a session, and paged SSD caching brings that down to 1–3 seconds on long contexts.

Why This Matters Now

Local inference on Mac has had a good year. We covered Ollama’s switch to the MLX backend (1.6–2x faster) and Rapid-MLX (2–4x faster than Ollama on raw throughput). Both of those posts were about the same metric: tokens per second.

oMLX is interesting because it attacks a metric nobody was optimizing: time to first token in a long, mutating conversation.

Here’s the thing raw benchmarks hide. When you benchmark a local model you send one prompt and measure decode speed. Impressive numbers. But a coding agent doesn’t work like that. It sends dozens of requests where the system prompt, tool definitions, and file contents keep shifting in the middle of the context — a file gets edited, a tool result gets inserted, the conversation gets compacted. Standard prefix caching only helps when the entire prefix matches exactly. Change one token 40k tokens in, and every server on the market recomputes the whole thing.

That’s the gap between “my Mac does 108 tok/s” and “my Mac is unusable with Claude Code.” oMLX is the first widely-adopted MLX server built specifically to close it.

What It Actually Is

oMLX started from vllm-mlx v0.1.0 as its basic serving layer. Everything above that — SSD tiering, continuous batching, VLM support, the Anthropic API surface, the native macOS app — is original work, per the author.

The architecture is worth reading in full because it explains the design:

FastAPI Server (OpenAI / Anthropic API)
    │
    ├── EnginePool (multi-model, LRU eviction, TTL, manual load/unload)
    │   ├── BatchedEngine (LLMs, continuous batching)
    │   ├── VLMEngine (vision-language models)
    │   ├── EmbeddingEngine
    │   └── RerankerEngine
    │
    ├── ProcessMemoryEnforcer (total memory limit, TTL checks)
    │
    ├── Scheduler (FCFS, configurable concurrency)
    │   └── mlx-lm BatchGenerator
    │
    └── Cache Stack
        ├── PagedCacheManager (GPU, block-based, CoW, prefix sharing)
        ├── Hot Cache (in-memory tier, write-back)
        └── PagedSSDCacheManager (SSD cold tier, safetensors format)

The cache stack is the whole product. It’s block-based paged attention borrowed conceptually from vLLM — with prefix sharing and copy-on-write — but with a second tier underneath. When the hot RAM cache fills, blocks spill to SSD rather than being evicted into oblivion. On the next request with a matching prefix, they come back from disk.

Copy-on-write matters more than it sounds. It means two conversations that share a 30k-token system prompt share those blocks in memory until one of them diverges — and then only the diverging blocks get copied.

Getting Started

The friendliest path is the DMG from Releases — drag to Applications, done, with in-app auto-update. It also installs a ~/.omlx/bin/omlx CLI shim so you can script it.

Homebrew works too:

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx

# Run as a background service (auto-restarts on crash)
omlx start

From source:

git clone https://github.com/jundot/omlx.git
cd omlx
pip install -e .          # Core only
pip install -e ".[mcp]"   # With MCP support

Point it at a directory of MLX-format models and it auto-detects type — LLM, VLM, OCR, embedding, or reranker:

~/models/
├── Step-3.5-Flash-8bit/
├── Qwen3-Coder-Next-8bit/
├── gpt-oss-120b-MXFP4-Q8/
├── Qwen3.5-122B-A10B-4bit/
└── bge-m3/
omlx serve --model-dir ~/models

Any OpenAI-compatible client then talks to http://localhost:8000/v1, and there’s a built-in chat UI at http://localhost:8000/admin/chat.

The flags that actually matter

Defaults are conservative. These are the ones worth tuning:

# Enable the SSD cold tier — this is the headline feature, and it's opt-in
omlx serve --model-dir ~/models --paged-ssd-cache-dir ~/.omlx/cache

# Size the in-memory hot tier
omlx serve --model-dir ~/models --hot-cache-max-size 20%

# Raise concurrency (default: 8)
omlx serve --model-dir ~/models --max-concurrent-requests 16

# Memory guard: safe/balanced tiers, or a hard ceiling in GB
omlx serve --model-dir ~/models --memory-guard-gb 48

# Lock it down if you're not localhost-only
omlx serve --model-dir ~/models --api-key your-secret-key

Settings persist to ~/.omlx/settings.json, and CLI flags take precedence over the admin panel. Note the SSD cache directory is not enabled by default — if you install oMLX and don’t set --paged-ssd-cache-dir, you’re running without the feature you came for.

The Custom Kernel Footgun

This is the single most important practical detail in the project, and it deserves its own section because it will silently cost you 30x performance.

A plain pip install -e . does not build oMLX’s native Metal custom kernels. Affected model families — GLM-5.2, MiniMax M3, Qwen3.5 — then fall back to much slower generic paths, and use more memory doing it. The README quantifies it: for GLM-5.2 the fused DSA prefill is roughly 30x faster with the kernels — a measured 845 tok/s versus ~29 tok/s on an M3 Ultra.

Nothing errors. It just gets slow.

Building them requires the Metal toolchain, which Command Line Tools alone do not provide (you’ll hit xcrun: error: unable to find utility "metal"). You need full Xcode:

# From source
OMLX_WITH_CUSTOM_KERNEL=1 pip install -e .

# Homebrew (also needs full Xcode)
brew install jundot/omlx/omlx --HEAD --with-custom-kernel

Verify what you actually got:

python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_status())"

The official DMG ships the kernels precompiled — which is a strong argument for just using the DMG unless you have a reason not to.

Connecting a Coding Agent

Because oMLX exposes POST /v1/messages (Anthropic Messages API) alongside the OpenAI endpoints, it’s a genuine drop-in for Anthropic-native clients rather than requiring a translation proxy. The admin dashboard has one-click setup for OpenClaw, OpenCode, Codex, Hermes Agent, Copilot, and Pi, and generates the exact CLI command for each.

Two touches show real agent-workload experience:

  • Context scaling — oMLX scales reported token counts so that Claude Code’s auto-compact triggers at the right time when you’re serving a smaller-context model than the client assumes.
  • SSE keep-alive — prevents client read timeouts during long prefill, which is exactly when a local model is most likely to look “hung.”

Full endpoint surface:

EndpointDescription
POST /v1/chat/completionsChat completions (streaming)
POST /v1/completionsText completions (streaming)
POST /v1/messagesAnthropic Messages API
POST /v1/embeddingsText embeddings
POST /v1/rerankDocument reranking
GET /v1/modelsList available models

Tool calling uses mlx-lm’s built-in parsers, with auto-detection across Llama/Qwen/DeepSeek JSON <tool_call>, Qwen3.5 XML <function=...>, Gemma, GLM, MiniMax, Mistral [TOOL_CALLS], Kimi K2, and Longcat formats. The caveat: tool calling requires the model’s chat template to accept a tools parameter. If your quant’s template doesn’t, no amount of server config saves you.

Multi-Model Serving and Profiles

Running a coding model, an embedding model, and a reranker simultaneously is normally three processes fighting over unified memory. oMLX handles it in one, with:

  • LRU eviction when memory runs low
  • Model pinning to keep your daily driver always resident
  • Per-model TTL to auto-unload idle models
  • Process memory enforcement — a total ceiling defaulting to system RAM minus 8GB, which prevents the system-wide OOM that makes local LLM work genuinely painful on a Mac

The profiles feature is quietly excellent. You save a named bundle of per-model settings and it can be exposed as its own model ID: /v1/models then lists qwen3-8b:thinking alongside the base model. Same engine, same weights, settings overlaid per request — no extra memory, no reload. That’s a clean answer to “I want reasoning mode sometimes” without doubling your RAM footprint.

Community Reception

The reception in local-LLM circles has been strong. Threads in r/LocalLLaMA and r/LocalLLM comparing MLX engines repeatedly land on oMLX as the top choice for agentic workflows on Apple hardware, with users citing speed and efficiency over Ollama and LM Studio specifically for coding-agent use.

The consistent theme in user reports is the TTFT collapse: long sessions going from 30–90 second waits to a handful of seconds. That’s not a marginal benchmark win — it’s the difference between a tool you use and a tool you abandon.

Growth backs the sentiment. The project went from ~110 stars at the time of the author’s MLX discussion post to 19,977 stars today, and it’s still adding roughly 1,100 a week.

Honest Limitations

Apple Silicon only, macOS 15.0+. No Intel Macs, no Linux, no Windows. If you have mixed hardware this is not your inference layer.

984 open issues. For a six-month-old project moving this fast that’s not alarming, but it tells you the surface area is large and things break. The latest tag is a release candidate (v0.6.3rc1), and recent release notes mention fixing DeepSeek V4 prefill memory and restoring prefix-cache reuse in long Claude Code sessions — i.e. the core feature itself has regressed and been repaired before.

The custom kernel trap. Covered above, but worth repeating: the failure mode is silent slowness, not an error.

SSD cache is opt-in and writes real bytes. KV blocks in safetensors format on disk means genuine write volume on long sessions. One shipped fix stopped admin benchmarks from writing generated KV caches to tiered storage precisely because it caused long post-run delays. Budget disk space, and think about it if you’re precious about SSD wear.

Distributed multi-Mac inference is experimental and source-build only. Splitting a model across Macs over Thunderbolt RDMA/JACCL is a genuinely exciting feature with read-only peer discovery, unequal shard planning, and a live performance map — but the README itself flags security boundaries and a hardware-validation checklist. Don’t build production on it yet.

It’s one primary maintainer. Impressive velocity, real bus-factor risk. Apache 2.0 and 1,701 forks mitigate that, but go in clear-eyed.

Who Should Use It

Use oMLX if: you own a 32GB+ Apple Silicon Mac, you want to run coding agents against local models, and long-session latency is what’s been killing the experience. This is the most targeted fix available for that exact problem.

Use Ollama instead if: you want the largest model ecosystem, cross-platform support, and the smoothest onboarding, and your workload is short one-off prompts rather than agent sessions.

Use Rapid-MLX instead if: raw single-request throughput is your benchmark and you don’t need multi-model serving or persistent cache tiers.

Skip local entirely if: you’re on a 16GB machine trying to run a frontier-class coding model. No cache architecture fixes insufficient memory — check what your hardware can actually run first.

FAQ

What problem does oMLX’s SSD KV cache actually solve?

Coding agents constantly mutate their prompt prefix — editing files, inserting tool results, compacting history. Conventional prefix caching only helps on exact full-prefix matches, so any mid-context change forces full recomputation of the entire context. oMLX stores KV blocks in pages across RAM and SSD, so previously computed context is restored from disk rather than recomputed. Reported effect: TTFT dropping from 30–90 seconds to 1–3 seconds on long contexts.

Is oMLX faster than Ollama?

They optimize different things. Ollama’s MLX backend and Rapid-MLX target tokens per second on a single request. oMLX targets latency across a long multi-turn session with a shifting prefix. On a cold one-shot prompt you may see little difference; twenty turns into a Claude Code session, the cache architecture is what dominates the experience.

Does oMLX work with Claude Code?

Yes. oMLX implements the Anthropic Messages API at POST /v1/messages, so it’s a drop-in backend rather than needing a translation proxy. It also adds context scaling so auto-compact fires at the right time with smaller-context models, and SSE keep-alive to prevent read timeouts during long prefill. The admin dashboard generates the connection command for you.

What hardware do I need?

Apple Silicon (M1/M2/M3/M4), macOS 15.0+ (Sequoia), Python 3.11–3.13. Practically, 32GB+ unified memory for useful coding models, and free SSD space for the cold cache tier. Intel Macs are not supported.

Is oMLX free?

Yes — Apache 2.0 licensed and fully open source, including the native macOS app. There’s a Buy Me a Coffee link, but no paid tier or license gate.

Why does my install feel slow on GLM or Qwen models?

Almost certainly missing native Metal custom kernels. pip install -e . doesn’t build them and the affected families silently fall back to slower, more memory-hungry paths — roughly 30x slower prefill for GLM-5.2 in the README’s own measurement. Run python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_status())" to check, and use the official DMG (kernels precompiled) or build with full Xcode installed.

Sources