Public Agent Benchmarks Lie About Your Codebase

We ran the same coding agent that had just posted a strong score on a well-known public benchmark against a real ticket in our monorepo: add a field to a GraphQL type, thread it through three resolvers, and update the two React components that consumed it. Nothing exotic. The kind of change a mid-level engineer finishes before lunch.

The agent burned through its context window reading files that turned out to be irrelevant, guessed at a resolver pattern that we'd deprecated two years ago but never fully removed, and produced a diff that compiled but silently duplicated a field name already used by a sibling type. On the benchmark, this same model was near the top of the leaderboard. In our repo, it needed three rounds of correction to do what a search-and-read of four files would have solved directly.

That gap isn't a scandal. It's the expected outcome of testing agents on the wrong kind of problem and then generalizing the result to a kind of problem the benchmark never included.

Benchmarks measure a shape your codebase doesn't have

Public agent benchmarks are, almost without exception, built from self-contained problems: a single repo, a bounded issue, a clear pass/fail test. That shape is easy to grade, which is why it's popular, but it's also nothing like how a multi-million-line monorepo actually works. Our GraphQL schema alone spans hundreds of files across a dozen packages, with resolvers that call into services owned by other teams, feature flags that change behavior at runtime, and naming conventions that evolved three times over the company's history and were never fully backfilled.

A benchmark task tells the model exactly which repo it's in and roughly what "done" looks like. A real ticket tells you neither. Half of solving it is figuring out which of six similarly-named UserPreferences types is the one actually rendered on the page the ticket is complaining about, and that search cost doesn't show up in any eval because eval repos are small enough that you don't need to search — you just read.

This is the same lesson the industry learned about human hiring interviews years ago and apparently has to relearn for agents: a task designed to be gradable at scale stops resembling the task you actually need done.

Context limits are a wall you don't see until you hit it

Every agent vendor advertises a context window in tokens, and every one of those numbers is true and also useless for judging fit against your repo. What matters isn't how many tokens the model can hold — it's how much of your codebase's relevant context an agent can assemble before the signal drowns in noise.

In a monorepo, the file the agent needs to understand a bug is often not the file with the bug in it. Our checkout flow's actual behavior lives partly in a React component, partly in a resolver, partly in a shared validation library imported by four other teams, and partly in a feature-flag config that overrides all of the above in production. An agent that reads the component in isolation — which is what a benchmark task trains it to expect as sufficient — will produce a plausible, wrong fix every time, because plausible is all it had the material to be.

We've found the failure mode is almost never "the model isn't smart enough." It's "the model was confidently wrong about which forty files mattered," and no amount of window size fixes a search problem.

Coupling is the real benchmark, and it's the one nobody publishes

The property that actually predicts whether an agent will help you isn't model capability, it's how tightly your code is coupled. A function that's called from one place, with a clear name and a colocated test, is legible to a model the same way it's legible to a new hire: the blast radius is visible in the file itself. A function reached through three layers of indirection, re-exported under a different name, and consumed by a component that infers its shape from a GraphQL fragment defined two directories away is illegible to both.

We noticed this concretely when comparing agent success rates across two parts of our platform. In a newer service with flat modules and consistent naming, the agent's first-attempt diffs were correct more often than not. In an older part of the monorepo — organically grown, heavily interdependent, three naming conventions deep — the same agent, same model, same prompt quality, needed supervision on nearly everything nontrivial. The variable wasn't the AI. It was the architecture.

That reframes the benchmark conversation. "How good is this model at coding" is the wrong question to optimize your workflow around. The better question is "how much of my codebase looks like a benchmark problem" — bounded, well-named, low-coupling — because that's the part where agent assistance actually compounds, and the rest is where you still need a human doing the archaeology first.

What to actually change

None of this means agents are a bust on real codebases — ours are useful daily, on the parts of the codebase that earn it. What it means is that the leaderboard number is a ceiling for a kind of task you may not have much of, not a prediction for your ticket queue.

The practical move isn't waiting for a bigger context window. It's the same discipline that makes a codebase easier for new engineers to onboard into: reduce indirection where you can, keep naming consistent instead of accumulating three eras of convention, and colocate the code with the tests and types that explain its contract. Every one of those changes makes the codebase more legible to a junior engineer on their first week — and it turns out that's exactly the same axis that makes it legible to an agent. We didn't set out to optimize for AI tooling when we started flattening our newer services; we did it because deeply nested indirection was already slowing down human reviewers. The agent performance gain was a side effect, and it's the more durable argument for doing it anyway.

The takeaway

A benchmark score tells you what a model can do on a problem shaped like the benchmark. Your monorepo is not shaped like the benchmark, and the gap between the two is mostly a function of coupling and naming, not model quality. If you want agents to actually help on your real code, the most valuable investment isn't picking a different model — it's making your codebase the kind of place where "which forty files matter" has an obvious answer, for the model and for the next person who joins your team.