Four AI Agents Tested: Tokens, Time and Quality

AI in architecture is no longer limited to image generation. It is appearing in brief analysis, building-code and site research, layout studies, and recurring validation or documentation work. The RIBA AI Report 2025 describes these uses across pre-design, design and testing. At the same time, a 2025 AIA study found that only 8 percent of surveyed firms had formally implemented AI, while another 20 percent were actively working toward it.

During this transition, most discussions focus on the model and overlook the interface that actually directs it. Two agents can run the same model yet process different amounts of context, make different numbers of tool calls and take different amounts of time. For an architecture practice using AI for tender review, planning research, BIM-adjacent automation or internal quality control, that difference is operationally important. It affects cost and speed, but also traceability, automation and the amount of human supervision required.

I therefore compared four agent interfaces under the same conditions: Codex App, Codex CLI, Pi and OpenCode.

What the test actually measures

This was not a comparison of different local models. The same gpt-5.6-luna model at maximum reasoning effort ran in all 20 fresh sessions. The benchmark isolated the agent layer around it: context management, tool use, runtime and the ability to derive a defensible answer from a real repository.

The five tasks came from five different repositories and covered demanding read-only investigations: publishing state, approval workflows, 3D inventory evidence gates, CAD/DXF output consistency and concurrent notification delivery. Each task ran on all four interfaces with the same prompt. After normalising line endings and trailing whitespace, the four prompt copies for each task produced the same SHA-256 hash.

Every answer was then checked against the current source code. The rubric covered behavioural tracing, risk accuracy, evidence or tests, and response-contract compliance. All 20 final answers scored 12 out of 12. That does not prove that the agents are equally intelligent in general; it means these five research tasks reached the quality ceiling of this test.

Overall results

The weighted score assigned 60 points to quality, 15 to processed token volume, 15 to runtime and 10 to reliability.

• Codex App: 96.49/100 · 4,093,797 processed tokens · 494,181 non-cached tokens · 30.98 minutes · 66 tool calls
• Codex CLI: 95.77/100 · 4,088,140 processed tokens · 492,620 non-cached tokens · 30.86 minutes · 62 tool calls
• Pi: 95.75/100 · 3,370,229 processed tokens · 545,013 non-cached tokens · 37.48 minutes · 170 tool calls
• OpenCode: 93.65/100 · 4,198,888 processed tokens · 633,320 non-cached tokens · 36.13 minutes · 186 tool calls

The ranking is tighter than it first appears. Codex App won the weighted score; Codex CLI was only 6.85 seconds faster in total. Pi finished just 0.02 points behind the CLI but showed a distinctly different resource profile. OpenCode delivered the same measured answer quality, although it did not lead this set in tokens, runtime or tool-call count.

The Pi question: does it really use fewer tokens?

Pi is the newest and most deliberately minimal agent approach in this comparison. The official Pi documentation describes a small, extensible terminal environment, while its compaction documentation explains how older context can be summarised and large tool outputs constrained. That design has helped create a community perception that Pi is especially lean. Anecdotes, however, are not a controlled measurement.

Our five tasks produced a more precise answer. Pi had the lowest total processed token volume at 3.37 million: 17.7 percent below Codex App, 17.6 percent below Codex CLI and 19.7 percent below OpenCode. The expectation of leaner total context processing was therefore confirmed in this benchmark.

But only for that metric. Codex CLI and Codex App used fewer non-cached tokens. Pi also took roughly 21 percent longer than the CLI and made almost 2.7 times as many tool calls. “Fewer tokens” is therefore incomplete unless we state whether the goal is to minimise total context volume, newly processed content, actual API cost or human waiting time.

The word “token” itself needs care. The four export formats did not expose identical fields, so the common measure was normalised as a workload metric: input, cache activity, output and, where separate, reasoning. It is not a direct invoice amount or a statement about subscription limits.

Which interface fits which architecture workflow?

Codex App is the strongest default for interactive investigation. When drawings, screenshots, reports and intermediate results need to be reviewed together, and a human will regularly intervene, the app offers the best balance of visibility, speed and handoff.

Codex CLI fits repeatable workflows: standard repository audits, automated documentation runs, CI-adjacent checks or clearly defined quality gates. It was the fastest interface in this benchmark and used the fewest tool calls.

Pi is compelling when a slim, adaptable agent layer or total context volume matters most. That may be useful for experimental local-model stacks, constrained context windows or custom tool chains. This benchmark did not use a local model, however, so it does not establish how Pi behaves with local inference.

OpenCode remains useful for provider flexibility and as an independent second surface. It passed every quality check but showed no general efficiency advantage across these five tasks.

The practical question is therefore not “Which agent is best?” but “Which friction should this workflow minimise?” For a one-off visual planning review, interactivity may dominate. For nightly code, standards or documentation checks, reproducibility and runtime may matter more. For local experimentation, context budget and extensibility may be more important than raw speed.

Limits and the next benchmark

The five tasks were real and demanding, but read-only. This benchmark measures research quality and operating efficiency, not the correctness of code changes. It has not yet tested a targeted bug fix with tests, a small multi-file feature, repair of a deliberately broken test, a browser workflow with persistent readback, or safe continuation after an interrupted session.

Those tasks form the logical second phase. Only then can we see whether today's almost identical quality results persist when an agent changes a system and becomes responsible for regressions.

My current conclusion is pragmatic: Codex App as the interactive default, Codex CLI for automation, Pi for token- and harness-oriented experimentation, and OpenCode for flexibility or a second opinion. Pi did meet the expectation of lower total processed-token volume, but not for free. The saving came with longer runtime and many more tool steps.