AI Coding Agents on Real Enterprise Codebases: Why 70%+ Public Scores Collapse to 16–39% (Real-SWE / SWE-bench Pro)
The best coding agents clear 70%+ on public benchmarks but resolve at most 38.8% of tasks on authorized private production codebases — and as little as 16.2%. The gap isn't model strength; it's private context. Here's what the newest real-world benchmarks measure, why leaderboard scores don't transfer, and what it takes to make an agent useful inside your own code.
The short answer
If you are deciding whether an AI coding agent can do real engineering work inside your company's codebase, the honest number is lower than the headlines suggest. On SWE-bench Verified, a public benchmark, frontier models now clear 70%+. But on Real-SWE — the first benchmark that runs those same models against authorized, private production codebases — the best agent resolves just 38.8% of tasks, and several well-known models land between 16.2% and 33.8%. SWE-bench Pro, which also includes proprietary code, puts top models at roughly 23%.
The takeaway for enterprise teams: a strong public leaderboard score does not predict performance on your code. The gap is not mainly about which model is "smartest." It is about private context — the business logic, internal conventions, and real-world consequences that public benchmarks never see.
What are Real-SWE and SWE-bench Pro, and what does "real pass rate" actually measure?
Most benchmark numbers you have seen come from public, open-source repositories. That is exactly the problem: open code can leak into training data, and the tasks rarely carry business consequences. Two newer benchmarks were built to close that gap.
Real-SWE, released by Specific Labs in September 2026, is the first benchmark to evaluate frontier models on authorized private production codebases. Its tasks are drawn from real companies' tickets — billing, tax filing, customer migrations — the kind of work that has genuine business consequences if it goes wrong. Because the code has never been public, this substantially reduces the risk of public-training-data contamination. The benchmark frames the question bluntly: "Can a coding agent actually do the work of a software engineer in the real world?" (Real-SWE, Specific Labs).
The metric that matters here is the resolution rate — the share of real tickets an agent fully resolves, verified against the task's tests, not a similarity score or a partial credit. It is the closest thing we have to "did the agent actually ship the fix."
SWE-bench Pro takes a complementary angle: it spans 1,865 tasks across 41 repositories, including private and commercial proprietary code, and reports resolution rates per split (SWE-bench Pro Leaderboard, Scale). Together, the two benchmarks give enterprise readers something the public leaderboards cannot: a read on how agents behave on code that looks like theirs.
Why do models score 70%+ on public leaderboards but only two-to-three-tenths on real codebases?
The cleanest way to see the gap is side by side.
Table 1 — Public leaderboard vs. private enterprise benchmarks
| Benchmark | Type | Top-model resolution rate |
|---|---|---|
| SWE-bench Verified | Public | 70%+ |
| SWE-bench Pro (public / commercial split) | Includes private proprietary | ~23% |
| Real-SWE | Authorized private production codebases | 16.2% – 38.8% |
Within Real-SWE itself, even the leaders leave most tickets unresolved:
Table 2 — Real-SWE resolution rates by model
| Rank | Model (harness) | Resolution rate |
|---|---|---|
| 1 | Fable 5.1 (Claude Code) | 38.8% |
| 2 | GPT-6 Astra (Codex CLI) | 33.8% |
| 3 | Gemini 3.8 Flash | 31.2% |
| 4 | GLM 5.3 | 28.8% |
| 5 | Grok 4.6 / Muse Spark 1.3 | 23.8% |
| 6 | Kimi K3 | 18.8% |
| 7 | GPT-5.6 Sol | 16.2% |
There is also a reason to read any leaderboard critically, not just the private ones. A 2026 OpenAI audit found that roughly 30% of one widely used dataset contained overly strict or broken test cases — meaning some "failures" were the benchmark's fault, not the model's, and some "passes" were easier than they looked. As a practitioner-focused breakdown from CodeAnt argues, a headline SWE-bench percentage means little until you know how the tasks were constructed and scored. The lesson cuts both ways: public scores can be inflated by contamination and easy tasks, and depressed by broken tests. Neither tells you what will happen inside your repo.
What makes private codebases hard — and how do you actually get an agent to do the work?
The reason the numbers drop is not that enterprise code is written worse. It is that real tickets carry context a general model was never given. Real-SWE's tasks — billing, tax filing, customer migration — demand knowledge of your private data models, your internal conventions, and your business semantics. A migration that is "correct" in the abstract can still be wrong if it violates a compliance rule encoded nowhere in the public internet. An out-of-the-box model cannot infer what it has never seen.
This is not a fringe concern. When Real-SWE launched, it reached the front page of Hacker News with 272 points and 154 comments in two days — a strong signal that engineering leaders are actively asking whether their tooling budgets are buying real capability or leaderboard theater.
So what closes the gap? In our field-engineering work, the difference between "strong public model" and "useful inside our codebase" comes down to a few things the benchmark scores make visible by their absence:
- Private context, made available to the agent. Retrieval over your own code, data schemas, and internal docs — so the model reasons about your system, not a generic one.
- Enterprise conventions, captured as durable scaffolding. The unwritten rules (naming, review gates, compliance constraints) turned into explicit guardrails the agent can follow.
- A verification loop with business stakes in mind. Because the tasks that matter have consequences, the agent's output needs the same checks a human PR would get, not a single green run.
- Forward-deployed engineering (FDE). Someone embedded closely enough with the team to encode all of the above, and to tell the difference between a fix that passes tests and a fix that is actually safe to ship.
Cost and reliability are the other two enterprise veto points, and they compound this one — but they are separate topics; see our notes on why enterprise AI agents fail in production and on production reliability and observability for how they interact, and on the hidden token costs of running agents for the budget side.
If you want a grounded read on where your own codebase and workflows sit on this curve, you can book a free AI implementation diagnosis — we will look at your actual code, conventions, and tickets rather than a leaderboard.
FAQ
What are Real-SWE and SWE-bench Pro?
They are benchmarks that evaluate AI coding agents on private, proprietary codebases rather than public open-source ones. Real-SWE (Specific Labs, Sept 2026) is the first to use authorized private production code, with tasks drawn from real business tickets; top models resolve at most 38.8% of them. SWE-bench Pro spans 1,865 tasks across 41 repositories including proprietary code, with top models around 23%.
Why do public leaderboard scores fail to predict performance on enterprise code?
Public benchmarks like SWE-bench Verified use open-source code (which can leak into training data) and low-stakes tasks, producing 70%+ scores. On private production code — Real-SWE at 16.2%–38.8%, SWE-bench Pro at ~23% — the same models drop sharply. A 2026 OpenAI audit also found ~30% of one common dataset had broken or overly strict tests, so leaderboard percentages should be read alongside how the tasks were built.
How do enterprises actually get AI to work on their own private codebase?
By supplying what the benchmarks show is missing: private context (retrieval over your code and data), your internal conventions captured as explicit scaffolding, a verification loop that accounts for business consequences, and forward-deployed engineering to encode all of it. The bottleneck is rarely raw model strength — it is the private, business-specific context a general model was never given.
