LCAgent Model Benchmark: score, cost, and routing trade-offs
Research artifact / Little Control Room
This is a field report from Little Control Room. We used LCAgent to run the same source-review assignment through a batch of current coding-capable models, then scored each run on practical agent-routing behavior: answer quality, tool discipline, cache behavior, latency, and estimated API cost.
The goal was not to rank models in the abstract. It was narrower and more practical: if LCAgent is doing repeated coding-agent work, which model should carry the main route, which one can scout cheaply, and which premium model is worth keeping for review?
Benchmark Method
The assignment asked the agent to review the LCAgent implementation against an implementation handoff document and identify missing functionality. This is a realistic source-review task: it rewards reading the right files, keeping track of implemented behavior, and avoiding confident false gaps.
- Each run used the same repository snapshot and prompt.
- Cost matters because this is repeated routing work, not a one-off demo.
- Low reasoning effort stays in the comparison because it is often the practical production lane.
The score is a subjective routing score out of 10, not pure prose quality. A model could write a good-looking answer and still lose points for reviewing the wrong document, overclaiming missing features, timing out, wasting calls, or failing to produce a comparable final answer.
- Answer correctness and coverage: about 50%.
- Evidence discipline and tool use: about 25%.
- Cost, cache behavior, latency, and routing ergonomics: about 25%.
Combined Results
Cost colors: green under $0.10, yellow $0.10-$0.30, orange $0.30-$1.00, red above $1.00. The cheapest row is bolded but not outlined, because the background tint already carries the cost signal.
| Run | Effort | Score | Wall | Cost | Cache | Reason | Verdict |
|---|---|---|---|---|---|---|---|
| GPT-5.5 | low | 8.8 | 92s | $0.432 | 111,616 | 165 | Best answer and best overall pick despite not being the fastest. |
| Claude Opus 4.7 | low | 8.5 | 203s | $1.644 | 96,242 | 1,163 | Strongest audit quality with Anthropic cache reads, but premium-priced. |
| Claude Opus 4.7 | medium | 7.8 | 249s | $2.104 | 134,897 | 1,811 | Strong milestone calibration, but longer context and cache writes raised cost. |
| GPT-5.5 | medium | 7.4 | 191s | $0.900 | 97,280 | 1,802 | Useful, but more expensive and less calibrated than GPT-5.5 low. |
| Kimi K2.6 | baseline | 6.8 | 143s | $0.155 | 222,049 | 0 | Best cheap secondary so far; more wandering than GPT. |
| Claude Sonnet 4.6 | medium | 6.3 | 430s | $0.986 | 227,415 | 9,810 | Strong cache reads, but the final answer narrowed too much. |
| GPT-5.5 | disabled | 6.2 | 59s | $0.511 | 32,256 | 0 | Fast, but weaker and worse cache economics than low. |
| Gemini 3.1 Flash Lite | low | 6.1 | 39s | $0.033 | 44,883 | 6,066 | Very fast and cheap; final was shallow but plausible. |
| GLM 5.1 | low | 5.9 | 388s | $0.308 | 102,720 | 9,757 | Read less raw source but searched heavily and was slow. |
| Gemini 3.1 Pro Custom Tools | low | 5.8 | 112s | $0.368 | 109,515 | 8,762 | Completed cleanly but made costly false claims. |
| DeepSeek V4 Pro | baseline | 5.6 | 365s | $0.065 | 184,960 | 4,426 | Extremely cheap, but weak quality and very slow. |
| MiniMax M2.7 | low | 4.2 | 126s | $0.034 | 154,496 | 2,874 | Cheap, but low effort produced broader false missing-feature claims. |
Reasoning-token counts are provider-reported usage accounting. A zero can mean the provider did not report a recognized reasoning-token field; for Kimi K2.6 it should be read as unavailable, not proof that no internal reasoning occurred.
What I Would Route Today
For this specific harness, GPT-5.5 low is the main route. It was not the cheapest and not the fastest, but it gave the best practical balance: enough reasoning, grounded source review, good cache reuse, and fewer false claims.
Kimi K2.6 is the interesting budget lane. It was much cheaper than GPT-5.5 low and produced a usable answer, but wandered more. That makes it attractive for scouting, not for final review.
Claude Opus 4.7 is the expensive audit pass. It produced the most detailed review and used Anthropic prompt caching, but the cost profile makes it a verification model rather than a routine route.
Sonnet and MiniMax both benefited from provider pinning, but in this assignment their completed answers still missed too much implemented behavior to win a main lane.
Artifacts
The full report includes the longer result table, Anthropic cache-write details, scoring excerpts, and source notes. The CSV is kept next to it so the table can be reworked without scraping the blog page.