Research report for the Little Control Room project, github.com/dpasca/LittleControlRoom. It summarizes LCAgent model benchmark runs against one fixed repository snapshot, prompt, turn budget, and tool harness.
This report benchmarks typical headless coding-agent work against a fixed repository snapshot: inspect source and docs, issue search and file-read tool calls, preserve enough context to compare implementation against a target document, and synthesize a structured final review. The task expected no file edits.
The benchmark is therefore about routing behavior inside this harness, not a universal model leaderboard. It measures final-answer calibration, tool discipline, structured-output reliability, cache behavior, latency, and cost under the same prompt, turn budget, timeout, and repository snapshot.
Cost matters because this is repeated agent-routing work, not a one-off demo. That is why low reasoning effort stays in the comparison as a first-class route, with medium effort sampled when there is a plausible quality or reliability return.
Notable mentions: Kimi K2.6 remains the best budget secondary. Gemini 3.1 Flash Lite is the fastest cheap scout, but shallow. Strict OpenRouter origin routing was used for Claude Sonnet 4.6 and MiniMax M2.7. Claude rows used explicit Anthropic cache controls; the cache reads were real, but cache writes are not free. That still did not make Sonnet a top candidate: low effort needed forced synthesis, while medium effort became too narrow. MiniMax completed reliably but was not accurate enough to pick as the main route.
Provider routing is recorded in the CSV. Completed Claude and MiniMax rows used strict OpenRouter origin-provider pins with fallbacks disabled and required parameter support enabled. Other completed OpenRouter rows used ordinary OpenRouter routing. Temperature was explicit at 0.2 for most OpenRouter and direct DeepSeek chat-completions runs; OpenAI Responses, direct Moonshot, and Claude Opus 4.7 runs omitted temperature.
The medium-effort runs did not change the recommendation. They kept Opus strong but more expensive, gave Sonnet strong cache reads without fixing answer focus, and made GPT-5.5 slower, pricier, and less calibrated than GPT-5.5 low.
Scores are 0-10 operator routing scores, not pure answer-quality scores. Each score was assigned after reviewing the final answer and the run metrics in the CSV.
Answer correctness, calibration, and concrete gap identification carried the most weight, roughly 40%. Tool discipline, evidence gathering, and selecting the right comparison source added roughly 20%. Structured-output and harness reliability added roughly 15%.
Cost and cache behavior contributed roughly 15%, while latency and practical routing ergonomics contributed roughly 10%. Invalid final output, reviewing the wrong document, confident false missing-feature claims, or forced synthesis after wandering received hard penalties.
X-axis uses estimated benchmark cost on a log scale. The quadrant thresholds are score 7.0 and $0.20 per benchmark run.
No model landed in the ideal quadrant in this batch. Claude rows used explicit Anthropic prompt caching; Sonnet remained too uneven and Opus remained premium. Kimi K2.6 is still the closest budget candidate; GPT-5.5 low remains the practical top pick; Claude Opus 4.7 remains the expensive audit pass. Rows with invalid or missing final output are excluded from this chart.
| Run | Score | Effort | Wall | Cost | Tok. | Cache | Resp. | Reads | Lines | Reason | Verdict |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.5 OpenAI API |
8.8 | low | 92s | $0.432 | 172,435 | 111,616 | 16 | 15 | 2,121 | 165 | Best answer and best overall pick despite not being the fastest. |
| Claude Opus 4.7 OpenRouter |
8.5 | low | 203s | $1.644 | 348,210 | 96,242 | 16 | 20 | 2,608 | 1,163 | Strongest audit quality with Anthropic cache reads, but it remains a premium verification route. |
| Claude Opus 4.7 OpenRouter |
7.8 | medium | 249s | $2.104 | 451,186 | 134,897 | 21 | 18 | 2,575 | 1,811 | Strong milestone calibration, but cache writes and longer context made it more expensive than Opus low. |
| GPT-5.5 OpenAI API |
7.4 | medium | 191s | $0.900 | 234,660 | 97,280 | 24 | 24 | 2,014 | 1,802 | Useful answer, but more expensive and less calibrated than GPT-5.5 low. |
| Kimi K2.6 Moonshot API |
6.8 | baseline | 143s | $0.155 | 323,374 | 222,049 | 24 | 26 | 2,798 | 0 | Best cheap secondary so far; more wandering than GPT. |
| Claude Sonnet 4.6 OpenRouter |
6.3 | medium | 430s | $0.986 | 415,109 | 227,415 | 21 | 11 | 1,304 | 9,810 | Strong Anthropic cache reads, but the final answer narrowed onto tool-surface gaps and missed broader doc drift. |
| GPT-5.5 OpenAI API |
6.2 | disabled | 59s | $0.511 | 117,441 | 32,256 | 11 | 19 | 2,546 | 0 | Fast, but weaker and worse cache economics than low. |
| Gemini 3.1 Flash Lite OpenRouter |
6.1 | low | 39s | $0.033 | 136,858 | 44,883 | 11 | 9 | 999 | 6,066 | Very fast and cheap; final was shallow but plausible. |
| GLM 5.1 OpenRouter |
5.9 | low | 388s | $0.308 | 277,963 | 102,720 | 21 | 8 | 1,001 | 9,757 | Read less raw source but searched heavily and was slow. |
| Gemini 3.1 Pro Custom Tools OpenRouter |
5.8 | low | 112s | $0.368 | 233,110 | 109,515 | 17 | 9 | 1,518 | 8,762 | Completed cleanly but made costly false claims. |
| Gemini 3.1 Flash Lite OpenRouter |
5.7 | medium | 30s | $0.036 | 224,738 | 94,556 | 18 | 12 | 1,601 | 0 | Still cheap and fast, but under-called real gaps and became less useful than low. |
| DeepSeek V4 Pro DeepSeek API |
5.6 | baseline | 365s | $0.065 | 323,884 | 184,960 | 24 | 30 | 3,364 | 4,426 | Extremely cheap, but weak quality and very slow. |
| DeepSeek V4 Pro DeepSeek API |
5.3 | medium | 331s | $0.069 | 308,450 | 159,232 | 19 | 30 | 2,818 | 4,717 | Cheap again, but medium did not fix false tool and architecture claims. |
| Claude Sonnet 4.6 OpenRouter |
5.2 | low | 488s | $1.094 | 407,487 | 195,190 | 24 | 12 | 1,662 | 11,465 | Completed with Anthropic cache reads, but still needed forced synthesis and included false missing-feature claims. |
| Gemini 3 Flash Preview OpenRouter |
5.1 | medium | 122s | $0.123 | 308,555 | 161,487 | 21 | 12 | 1,340 | 15,072 | More substantial than low, but included unearned verification claims and factual misses. |
| Grok 4.3 OpenRouter |
5.0 | low | 150s | $0.156 | 162,032 | 55,168 | 16 | 4 | 190 | 7,669 | Completed, but used too little evidence and reviewed the wrong doc. |
| Gemini 3 Flash Preview OpenRouter |
4.8 | low | 42s | $0.098 | 334,189 | 161,659 | 24 | 15 | 1,984 | 0 | Fast, but final included confident false missing-feature claims. |
| MiniMax M2.7 OpenRouter |
4.7 | baseline | 103s | $0.049 | 277,041 | 162,720 | 21 | 21 | 3,029 | 3,308 | Strict MiniMax routing fixed completion, but the final missed important gaps and added false tool/skill claims. |
| Gemini 3.1 Pro Custom Tools OpenRouter |
4.4 | medium | 201s | $0.520 | 331,810 | 186,282 | 24 | 12 | 1,729 | 17,695 | Medium effort increased cost and still made false missing-tool claims. |
| MiniMax M2.7 OpenRouter |
4.2 | low | 126s | $0.034 | 222,201 | 154,496 | 19 | 8 | 1,140 | 2,874 | Completed cheaply, but low effort produced broader false missing-feature claims than the baseline-effort MiniMax run. |
| GLM 5.1 OpenRouter |
3.8 | medium | 440s | $0.394 | 373,779 | 156,608 | 24 | 18 | 2,032 | 14,911 | Medium effort made GLM slower and more confidently wrong about implemented tools. |
| Grok 4.3 OpenRouter |
3.0 | medium | 174s | $0.227 | 220,245 | 59,200 | 16 | 5 | 247 | 9,167 | Read the right docs this time, but falsely marked core implemented tools and LCR wiring as absent. |
The lowest estimated cost is bolded. The cost column also uses bucket tints: green under $0.10, yellow $0.10-$0.30, orange $0.30-$1.00, and red above $1.00. Read and line counts are deliberately not highlighted because less reading is not automatically better in a source-review benchmark.
Scores are subjective benchmark-routing scores out of 10, not pure answer-quality scores. The Effort column is the requested reasoning-effort setting only, separate from reported reasoning tokens. Temperature is recorded per row in the CSV; most OpenRouter and direct DeepSeek chat-completions runs used 0.2, while OpenAI Responses, direct Moonshot, and Claude Opus 4.7 runs omitted temperature.
Kimi K2.6 was not rerun in the medium-effort pass because the direct Moonshot adapter does not accept the LCAgent reasoning-effort option.
Reasoning-token counts are provider-reported usage accounting, not harness estimates; a zero can mean the provider did not report a recognized reasoning-token field. For the direct Moonshot Kimi K2.6 row, it should be read as not reported rather than proof that no internal reasoning happened.
Provider routing is recorded in the CSV: completed Claude and MiniMax rows used strict provider pins, while other completed OpenRouter rows used ordinary OpenRouter routing. The Cache column records cache reads; Anthropic cache writes are called out separately below because they affect pricing but were not part of the original CSV schema. Rows with invalid or missing final output are excluded from the main scoring table because they are not comparable as route candidates.
OpenAI, Moonshot, DeepSeek, Gemini, Grok, and several OpenRouter routes can rely on implicit prompt caching. Anthropic Claude routes need cache_control to make prompt caching intentional. This benchmark used an explicit per-block cache breakpoint on the stable system message and pinned OpenRouter to Anthropic for Claude rows.
| Run | Score | Cost | Cache Reads | Cache Writes | Note |
|---|---|---|---|---|---|
| Claude Opus 4.7 low | 8.5 | $1.644 | 96,242 | 82,348 | Best Claude route: strong audit quality with substantial cache reads. |
| Claude Opus 4.7 medium | 7.8 | $2.104 | 134,897 | 120,938 | Completed cleanly, but cache writes and longer context made it more expensive than Opus low. |
| Claude Sonnet 4.6 medium | 6.3 | $0.986 | 227,415 | 52,008 | Cheapest Claude run, but final answer was too narrow. |
| Claude Sonnet 4.6 low | 5.2 | $1.094 | 195,190 | 67,135 | Completed with cache reads, but the run still needed forced synthesis and overclaimed. |
Anthropic cache reads are discounted, while 5-minute cache writes are billed at a premium. The strategy is useful but not automatically ideal: it works best when a stable prompt prefix is reused enough to amortize writes.
Short snippets from generated final answers, used only to make the subjective scores inspectable. They illustrate scoring rationale; they are not treated as ground truth.
| Run | Excerpt | Why It Mattered |
|---|---|---|
| GPT-5.5 low | "the MVP is largely implemented and in several places goes beyond the original first-slice handoff" | Correctly framed the task: not a missing-skeleton story, but a smaller set of harness-quality gaps. |
| GPT-5.5 low | "lcagent is not missing the core MVP skeleton" | Strong bottom-line calibration, with specific gaps called out after that. |
| Claude Opus 4.7 | "Most of the four milestones are implemented" | Best audit-style structure and lifecycle awareness, with substantial Anthropic cache reads. |
| Claude Opus 4.7 | "permission_denied event, the LCR-side launcher... future-context scaffolding" | Found nuanced follow-up gaps, but the run was still too expensive for primary routing. |
| Kimi K2.6 | "The biggest confirmed holes are the two missing outline tools" | Useful budget answer, but it also wandered into weaker claims like plan_item and piped stdin. |
| Grok 4.3 | "vs. `docs/ai_coding_agent_feasibility.md`" | Penalized because the user asked for the implementation handoff doc; this targeted the wrong comparison source. |
| Claude Sonnet 4.6 low | "`internal/lcagent/script/` missing as a distinct package" | False in the benchmark snapshot; cache reads did not fix overclaiming. |
| MiniMax M2.7 baseline | "`load_skill` implementation ... MISSING" | Also false in the benchmark snapshot; the run completed cheaply but missed important implemented behavior. |
| MiniMax M2.7 low | "`search` tool ... No `search` tool implementation" | A broader false-missing-feature claim than the baseline-effort MiniMax run. |
| Gemini 3 Flash Preview | "Tool-Call Markup Guardrail ... Missing" | Fast, but it confidently missed existing provider-markup guardrail work. |
| Claude Opus 4.7 medium | "Milestones 1-3 are essentially in place" | Strong medium-effort calibration, but still too expensive for primary routing. |
| GPT-5.5 medium | "`--dry-run` is documented but not implemented" | Penalized because dry-run was not actually in the benchmark docs; medium effort added a false headline gap. |
| Claude Sonnet 4.6 medium | "`search` tool missing `context_before` / `context_after` parameters" | A plausible harness-improvement note, but the answer over-focused on tool-surface details and missed broader doc drift. |
| Gemini 3.1 Pro Custom Tools medium | "`load_skill` tool logic ... appears to be missing or incomplete" | False in the benchmark snapshot; medium effort did not fix overclaiming. |
| GLM 5.1 medium | "no `apply_patch` or diff-application tool" | False in the benchmark snapshot and more damaging than the low-effort GLM answer. |
| Grok 4.3 medium | "`update_plan` ... absent" | False in the benchmark snapshot; it read the right docs but missed implemented core tools. |
It had a strong final answer, reliable structured-tool behavior, and good cache reuse. The none run was faster, but weaker and more expensive because cache reuse collapsed.
Claude Opus 4.7 gave a very strong, detailed audit-style answer, and explicit prompt caching reduced its cost. It still cost roughly 3.8x GPT-5.5 low in this benchmark, so it is best treated as a review/verification lane, not the highest overall routing choice.
It finished quickly, but it inspected more raw lines than GPT-5.5 low, made more doubtful claims, and cost more because cached input dropped from 111,616 to 32,256 tokens.
Gemini's speed is real, but the answers were too shallow or too wrong to displace Kimi. Grok 4.3 completed but inspected very little source and targeted the wrong doc. Strict provider routing fixed MiniMax's completion failure, but not its answer quality.