Introducing Enterprise-Bench

What benchmarks are – and why enterprise AI was crying out for one

A trustworthy benchmark levels the playing field, putting platforms through a standardized testing model that uses the same tasks, same data, same scoring. That means you can compare platforms with full transparency and confidence. That's how the database industry solved the "everyone claims they're fastest" problem in the 1990s, when TPC introduced common, auditable standards.

AI models have plenty of benchmarks that test reasoning, coding, tool use – mostly at single-user scale. What none of them test is whether or not an AI agent can operate inside the complexity of a real company.

Enterprises pose AI unique problems: data is sprawling and siloed; permissions and safety protocols are vital; answering a straightforward question might require connecting support tickets to sales records to engineering issues.

Until now, there was no standardized, trustworthy benchmark for enterprise AI.

How Enterprise-Bench works

We developed Enterprise-Bench with Laude Institute, and it was validated by Alexandros Dimakis, UC Berkeley professor, co-founder of Bespoke Labs, and DevRev board member. It's published through Harbor, with full GitHub access for inspection and contribution.

Every task in Enterprise-Bench has one correct answer. That answer is always the same – whether the dataset is small, or 256× larger. What changes is how buried it is.

At the smallest scale, 40% of the data is relevant to a given task. At the largest (256×), just 0.16% is. The question doesn't get harder – but finding the right answer in a growing sea of irrelevant data absolutely does.

We call this "answer-preserving data scaling". If an agent's accuracy drops as data grows, that's a retrieval architecture problem, not a model problem.

Enterprise-Bench evaluates on three axes simultaneously (because production-ready AI needs them all):

Tasks are classified across an L1–L4 autonomy progression, from deterministic lookups through multi-step synthesis to autonomous operation.

The key insight is that even "simple" lookups become architecturally demanding when they span multiple systems with different schemas – what we call "wide L1" – and that moving from curated tools to realistic API surfaces costs 18–19 points of accuracy on its own.

The finding that underpins all of this: retrieval architecture is a stronger predictor of production performance than the choice of foundation model.

The results are in

We ran Computer, by DevRev, head-to-head against Claude Code. On identical L1–L2 tasks, modeled on a realistic B2B company (42 customer accounts, 40 product parts, 5 interconnected enterprise systems). Same Opus 4.8 model, same data, same independent judge.

An important nuance: Claude Code performed well on simpler, single-user lookups and extraction. But the gap really opened up on enterprise-level work – e.g. joining tickets to engineering issues, connecting support records to opportunity risk, synthesizing information that lives across systems.

The proof. And the challenge.

If you want the full deep-dive into how we built it – and why – the full methodology is here.

The full methodology, test dataset, evaluation harness, scoring criteria, and all of our traces are publicly available.

And we are saying loud, and clear: any vendor, researcher, or customer should run their AI platform on Enterprise-Bench, and submit results to the leaderboard on Harbor. Because we truly believe that Computer is unbeatable.

Enterprise-Bench, including the full dataset, evaluation harness, queries, judging criteria, and initial results, is publicly available here.