What benchmarks did Meta report for Muse Spark 1.1?

Meta reported Terminal-Bench 2.1 of 80.0, SWE-Bench Pro of 61.5, and DeepSWE 1.1 of 53.3 at Muse Spark 1.1 launch on July 9, 2026. Meta positioned these as competitive with Claude Opus 4.8 (82.7 Terminal-Bench) and GPT-5.5 (83.4). API pricing is $1.25/$4.25 per MTok, roughly one-quarter of Sol/Opus flagship rates. Meta framed the release as its first paid API and a competitive answer to OpenAI and Anthropic in the coding-agent market.

What did independent benchmarks find?

Independent testing from Artificial Analysis and TechTimes put Muse Spark 1.1 roughly 10 points below Meta's Terminal-Bench figure — closer to 70 than 80. Artificial Analysis Intelligence Index scored it at 51 (an 8-point gain from original Muse Spark), placing it below GPT-5.6 Terra (~58) and Claude Sonnet 5 (~62). Cost per completed coding task landed at ~$0.26, which is genuinely competitive — cheaper than GPT-5.4 at similar scores — but the gap between Meta's claims and independent numbers is real.

Is Muse Spark 1.1 actually a good model despite the benchmark gap?

Yes, with caveats. Muse Spark 1.1 is genuinely cost-efficient — ~$0.26 per task on Artificial Analysis' coding suite vs $0.40+ for GPT-5.6 Luna and $0.45+ for Claude Haiku 4.5. Its token efficiency is real: 94M output tokens for the eval suite vs 125M for Luna. But it's not the frontier coding model Meta's numbers suggest. Treat it as a strong cheap-tier option, not a Sonnet 5 replacement. For high-stakes work, benchmark it head-to-head against Sonnet 5 or Grok 4.5 on your actual workload.

Why do Meta's numbers differ from independent tests?

Common reasons for benchmark gap: (1) prompt engineering differences — model providers often use optimal prompt configurations that don't reflect real-world usage, (2) sampling parameters — temperature, top-p, and reasoning tokens can shift scores 5-10 points, (3) test-set contamination — some models see benchmark data during training, (4) selective disclosure — providers report subsets of runs. Meta hasn't disclosed which prompt configuration or sampling parameters produced 80.0 on Terminal-Bench, so the gap is likely a mix of optimal-config gaming plus fair-but-different eval methodology.

Quick Answer

Muse Spark 1.1: Meta Claims vs Independent Benchmarks (July 2026)

Published: July 13, 2026

The Numbers Side-by-Side (July 13, 2026)

Benchmark	Meta’s claim	Independent	Gap
Terminal-Bench 2.1	80.0	~70	-10
SWE-Bench Pro	61.5	Not independently reported	—
DeepSWE 1.1	53.3	Not independently reported	—
Artificial Analysis Intelligence Index	Not reported	51	—

Sources: Meta’s launch materials (July 9, 2026); Artificial Analysis coding evals (July 10-11, 2026); TechTimes independent analysis (July 11, 2026).

What Meta Reported at Launch

At the July 9, 2026 Muse Spark 1.1 launch, Meta’s positioning was:

Muse Spark 1.1 hits 80.0 on Terminal-Bench 2.1 — competitive with Claude Opus 4.8 (82.7) and GPT-5.5 (83.4)
SWE-Bench Pro at 61.5 — best-in-class among cost-efficient models
DeepSWE 1.1 at 53.3 — strong agentic coding
API pricing: $1.25/$4.25 per MTok — undercutting Sol ($5/$30) and Opus ($15/$75)
First paid Meta API ever — a strategic pivot from open-weight

Meta framed the release as a competitive answer to OpenAI and Anthropic in the coding-agent market. Zuckerberg’s positioning: “we can build closed-weight models that beat theirs on cost and match on quality.”

What Independent Benchmarks Found

Artificial Analysis (July 10-11, 2026):

Muse Spark 1.1 Intelligence Index: 51 (up 8 points from Muse Spark 1.0)
Cost per completed task: ~$0.26 — cheaper than GPT-5.4 at similar Intelligence scores
Token efficiency: 94M output tokens for the Artificial Analysis evaluation suite vs 125M for GPT-5.6 Luna and 141M for GLM-5.2

TechTimes (July 11, 2026):

Muse Spark 1.1 independent coding benchmark: ~71 on Artificial Analysis’ 100-point scale (vs Meta’s Terminal-Bench 80.0)
10-point gap between Meta’s claim and independent measurement
Article headline framed the launch as “at one-third rival cost”

Reddit r/singularity (July 10-12, 2026):

Community reports on Muse Spark 1.1 concentrated on US-only availability and comparisons to Grok 4.5
Consensus: “cheapest but not smartest” — reasonable for high-volume, not for frontier tasks

Why the Gap Exists (Four Likely Explanations)

1. Optimal prompt configurations. Model providers benchmark with prompt configurations optimized for maximum score. Real-world usage rarely replicates that setup. Meta likely reported Terminal-Bench with reasoning-token-heavy sampling and system prompts tuned for the benchmark; independent testing uses more realistic configurations.

2. Sampling parameters. Temperature, top-p, max tokens, and reasoning-token budget can shift benchmark scores 5-10 points. Meta hasn’t disclosed the exact sampling parameters that produced 80.0 on Terminal-Bench. Independent testers use their own standardized parameters.

3. Selection effects. Providers can run benchmarks multiple times and report best-of-N. Independent testers typically report single-shot or averaged-N runs.

4. Test-set contamination or eval-suite drift. Terminal-Bench 2.1 is relatively new. If any Terminal-Bench-adjacent data leaked into Muse Spark training, Meta’s numbers would be artifactually high. This is speculative but the 10-point gap is large enough to raise the question.

Is Muse Spark 1.1 Still a Good Model?

Yes, at the price point. Even taking the independent numbers as ground truth (~71 vs Meta’s 80.0 on Terminal-Bench), Muse Spark 1.1 is still:

The cheapest per-token frontier model at $1.25/$4.25 per MTok
The cheapest per-completed-task frontier model at ~$0.26 on the Artificial Analysis coding suite
Genuinely token-efficient — 25-30% fewer output tokens per task than comparably-priced peers
A real competitive move against Sonnet 5, Grok 4.5, and Luna in the cheap-tier fight

Treat it as a strong cheap-tier option, not a frontier replacement. For workloads where cost per token matters more than absolute quality, Muse Spark 1.1 wins the price/performance tradeoff. For frontier reasoning or hardest-task coding, route to Opus 4.8 or Sol Ultra.

What This Says About the AI Benchmark Ecosystem

Meta isn’t uniquely bad here. Provider-reported vs independent benchmark gaps have been a structural feature of the LLM market since 2023:

Anthropic — historically closest match between provider-reported and independent scores
OpenAI — small gaps, but reasoning-token-heavy configurations
Google Gemini — gaps of 3-8 points on some benchmarks vs independent
Meta — larger gaps historically (Llama 3 to Llama 4 both showed 5-10 point differences)
Chinese labs (DeepSeek, Kimi) — mixed; sometimes closer to independent, sometimes wider gap

Meta’s 10-point gap on Muse Spark 1.1 is on the higher end but not unprecedented. Always treat provider-reported benchmarks as ceiling, not typical performance.

Practical Recommendation

For anyone considering Muse Spark 1.1 in July 2026:

Ignore Meta’s 80.0 Terminal-Bench figure. Plan around the independent ~70 number.
Benchmark on your own workload. Cheap-tier models diverge more on real tasks than on standardized suites.
Test Muse Spark 1.1 vs Grok 4.5 on your actual coding agent. Grok 4.5’s token efficiency often flips the “cheaper” calculation.
Watch for Meta’s response — provider-side rebuttals usually appear within 2-3 weeks of an independent gap being called out.

Sources

Meta Muse Spark 1.1 launch (Meta AI, July 9, 2026): ai.meta.com/blog/muse-spark
Artificial Analysis Muse Spark benchmarks: artificialanalysis.ai/models/muse-spark
Meta Muse Spark 1.1 independent coding benchmark analysis (TechTimes, July 11, 2026): techtimes.com/articles/320182
Muse Spark 1.1 benchmarks and evals (Kingy AI, July 2026): kingy.ai/blog/muse-spark-1-1-benchmarks-specs-evals
DataCamp coverage of Muse Spark 1.1: datacamp.com/blog/muse-spark-1-1