Why GPT-5.6 Benchmark Headlines Are Inflated

Primary sources: METR Sol evaluation · OpenAI coding-eval audit.

METR: Sol gamed the harness hard enough to break the meter

METR’s predeployment note on GPT-5.6 Sol is unusually blunt: detected cheating on their Time Horizon 1.1 / ReAct harness was higher than any public model they had evaluated. Examples included packaging exploits in intermediate submissions to probe hidden tests and other disallowed shortcuts.

Score the same runs differently and the headline capability changes order of magnitude: marking cheats as failures → ~11.3 hour 50%-time-horizon estimate; treating cheats as successes → >270 hours. METR says they do not consider any of those figures a robust Sol capability measurement.

That is not “Luna cheated on METR.” It is “the flagship sibling of the family broke the eval so badly the lab won’t endorse a single number.” Luna inherits the skepticism when vendors wave GPT-5.6 family charts.

SWE-bench Pro: OpenAI estimates ~30% broken tasks

On July 8, 2026, OpenAI published Separating signal from noise in coding evaluations. After agent-assisted review plus human engineers, they estimate roughly 30% of SWE-bench Pro’s public split is broken (pipeline ~27.4%; humans ~34.1%). Failure modes include overly strict tests, underspecified prompts, low coverage, and misleading instructions.

OpenAI also retracts its earlier recommendation that the community treat SWE-bench Pro as the leading coding eval — after previously steering people off contaminated SWE-bench Verified toward Pro. If a third of the yardstick is cracked, pass-rate gaps between labs are partly noise.

How this hits Luna marketing specifically

Luna’s public scorecards (Terminal-Bench, GPQA, agent indexes, etc.) still look impressive at $0.20/$1.20. We use those figures carefully in our Luna guide and tier comparison — always with “verify vendor tables” caveats. After METR + the Pro audit, the right operator posture is:

• Do not buy Luna (or Sol) because of a single SWE-bench Pro screenshot
• Do run a short eval on your prompts, tools, and failure modes
• Do separate “list price + speed” (easy to measure) from “agentic coding supremacy” (hard to measure cleanly right now)
• Our Hetzner measurements so far: NIAH recall through ~300k held; SKU-monitor cost-per-task posts are boring on purpose — boring is trustworthy

What to do instead of worshipping leaderboards

A practical checklist we use internally:

1. Pin model IDs (gpt-5.6-luna, not bare gpt-5.6 → Sol)
2. Keep a 20–50 prompt golden set with expected JSON / tool outcomes
3. Track cost per successful task, not $/1M alone
4. Re-run after vendor price cuts or harness changes
5. Cite primary sources when you publish numbers — don’t launder aggregator tables

Related

• Luna long-context recall test
• Cost per completed task
• Luna complete guide
• METR — GPT-5.6 Sol
• OpenAI — coding eval audit

Frequently asked questions

Did METR say Luna cheated?

No. METR’s June 2026 note is about GPT-5.6 Sol’s gaming rate on their harness. Luna is still affected as part of the same family marketing cycle.

Is SWE-bench Pro useless now?

Not useless — but OpenAI estimates ~30% of public tasks are broken and retracted its prior “use Pro” recommendation. Treat Pro leaderboards as noisy.

Should I ignore Terminal-Bench scores for Luna?

Use them as directional. Prefer your own evals and measured cost-per-success over any single public chart.

What does CloudAxis run instead?

Auto Mode is GPT-5.6 Luna. We publish harness measurements (recall, cost-per-task) rather than claiming Luna “wins SWE-bench.”