Coding agent benchmark: how much the runtime layer changes the result

Four coding-agent sessions received the same repository and the same high-level assignment. Every session produced a repository change and passed its targeted tests. End-to-end duration still ranged from 6:04 to 13:57.

Codex with Luna Max was the shortest observed run. DSH with Qwen 3.8 27B was the shortest observed local-model run. Both OpenCode configurations took longer, even though OpenCode Qwen reported the lowest total token volume.

The central finding is therefore not a universal model ranking. A coding agent is more than its model: system instructions, skills, memory, context management, cache behavior, tools, user gates, review policy and final verification all contribute to the result the user sees.

What the benchmark measures

The four sessions started with the same repository and abstract assignment, but they were not forced to select the same concrete defect. The Luna runs converged on the same flashcard area while choosing different patch scopes. The two Qwen runs selected different defects.

This benchmark compares complete runtime profiles, not four models under a controlled same-patch condition. That boundary matters. A session may take longer because generation is slower, the runtime performs more tool work, reviewers add a second pass, or context compaction and continuation create additional turns.

Runtime and reported token volume

• Codex · Luna Max: 363.696 seconds, 2.031 million reported tokens, 11/11 targeted tests.
• DSH · Qwen 3.8 27B: 483.641 seconds, 1.984 million tokens, 57 tool calls, 14/14 tests.
• OpenCode · Luna Max: 773.632 seconds, 2.683 million tokens, 67 tool calls, 11/11 tests.
• OpenCode · Qwen 100K: 836.954 seconds, 1.687 million tokens, 39 tool calls, 17/17 tests.

OpenCode Luna took 2.13 times as long as Codex Luna. OpenCode Qwen took 1.73 times as long as DSH Qwen. At the same time, OpenCode Qwen was the longest session despite reporting the smallest total token volume. Lower token volume is therefore not automatically a speed or quality score.

The token schemas also differ. Codex reports reasoning as part of output. OpenCode exposes reasoning separately and combines fresh input with cache reads in its prompt total. DSH provides prompt and output totals. The normalized numbers are workload signals, not direct billing, energy or subscription-limit figures.

Cache, orchestration and review

More than 91 percent of prompt-side volume came from cache reads in both OpenCode sessions. OpenCode Luna reported 220,630 fresh-input tokens and 2,451,456 cache-read tokens; OpenCode Qwen reported 145,146 and 1,514,973 respectively.

DSH Qwen used 36 assistant rounds and 57 tool calls. OpenCode Luna used 32 rounds and 67 calls; OpenCode Qwen used 31 rounds and 39 calls. Codex did not expose directly comparable counters in the reviewed export, so it is omitted from that chart instead of being assigned a misleading zero.

OpenCode Luna activated brainstorming, writing plans, test-driven development, code review and verification before completion. The reviewer alone accounted for about 299 seconds, or 38.6 percent of the session. This process can be valuable for a large or risky change. For a small fix, its fixed cost became a substantial part of the run.

Codex Luna used Karpathy guidelines and prior working memory. Those inputs may have encouraged a smaller patch, explicit assumptions and direct verification, but a single run cannot quantify their independent effect. That requires a separate skills-on versus skills-off experimental arm.

Which runtime fits which task?

• For a small, bounded cloud-model fix, Codex Luna Max was the shortest observed path.
• For local Qwen execution, DSH Qwen was the faster runtime stack in this test.
• OpenCode remains useful when provider portability, visible planning and one model-agnostic interface matter more than minimum elapsed time.
• For a large refactor or a high-regression-risk change, an explicit planning and review chain may justify its runtime premium.
• For micro tasks, a lighter agent profile should avoid fixed process overhead unless the risk requires additional gates.

Limitations and the next benchmark

Each condition ran once, so there is no variance or confidence interval. The agents did not implement the same patch, token exports use different semantics, and Qwen ran locally while Luna used cloud infrastructure. Test counts are not a quality ranking because the patches covered different behavior.

A stronger next experiment should use one fixed issue, one acceptance test, identical tools and system instructions, and equal context and timeout limits. Every condition should run at least three times. Qwen reporting should include hardware, quantization, context limit, KV-cache format and generation throughput. Per-step timestamps would finally separate model latency from tool and reviewer time.

Evidence and brand sources

The measurements were reconstructed from four retained session exports: `lunamax-session.jsonl`, `session.jsonl`, `session-ses_fdda.md` and `session-ses_fdd8.md`.

• OpenAI official GitHub organization
• DeepSeek official logo in the DeepSeek-VL repository
• Qwen official GitHub organization
• OpenCode official repository