Start with the operating model
A useful comparison begins before either agent writes code. Decide where the work should run, what repository context it may inspect, which commands require permission, and who owns the final release decision. Codex spans app, editor, and terminal surfaces; Claude Code emphasizes project work through terminal and desktop workflows. Those are verified product differences, not outcome evidence.
The right first candidate therefore depends on how the builder already works. A repository operated from an editor may value a different review loop from a terminal-first project. Neither interface proves that the resulting code is more correct, maintainable, secure, or economical.
Decision questions before testing
Context
Can the agent discover project rules, existing tests, and relevant files without broad unrelated edits?
Control
Are command permissions, network access, sensitive files, and destructive operations visible and bounded?
Review
Does the workflow produce a compact diff, clear assumptions, and fresh verification output a human can audit?
Recovery
Can the builder identify and reverse a wrong turn without losing unrelated work or restarting the entire task?
September 2026: define the run, not just the product
Astra's limited API release does not establish access in every Codex client. Anthropic distinguishes broad Fable 5.1 access from restricted Mythos 5.1 access, and documents different effort defaults across its surfaces. An old result cannot silently stand in for a new model or configuration.
Start with models both test accounts can actually use; a first baseline need not wait for restricted access. Record the exact model identifier, client version, date, account tier, effort, and execution surface. Matching effort labels across vendors does not establish equal compute: use the same task, acceptance checks, and declared time and spending limits instead.
Freeze repository instructions, network permissions, MCP tools, and available credentials. Log approvals, mid-run steering, retries, and human corrections. Report elapsed time and intervention time separately. For API-billed runs, itemize input, output, cache usage, and the dated rates; for subscriptions, report observed allowance use without inventing a per-task dollar cost. Unobservable cost remains UNKNOWN.
The benchmark ShipStackLab will run
- Start both agents from the same clean repository commit and dependency state.
- Provide the same feature brief, constraints, project instructions, and acceptance checks.
- Record prompts, permissions, commands, failed attempts, corrections, elapsed work, and attributable cost.
- Run the same test, lint, build, diff, accessibility, and production-smoke gates.
- Publish the artifacts and limitations before making any fit recommendation.
Until that controlled run exists, this page is a decision framework and test plan. Vendor descriptions establish capabilities; they do not establish which agent ships the better result.