Measuring AI on real-world work

Our benchmarks draw on work performed inside Current and Shield, with the context and tools needed to complete each assignment. Models run through the Holdings proprietary evaluation system under consistent conditions. Tax runs use the same client documents, prior-year information, and tax engine version. IT runs replay recorded tool responses in an isolated environment, with write actions mocked and the starting state reset between runs.

The overview shows the Pareto frontier: configurations offering the best score for their cost. Detailed charts and tables include all reported configurations. Tax cost per return is the source cohort total divided by 28; time is average extraction time. IT reports cost and elapsed time per task. Completion counts, run identifiers, and turn counts are not reported. Computer-use results and human-cost comparisons remain illustrative. Human-cost scenarios add an assumed practitioner-review allowance to execution cost and compare it with a fixed labor baseline, using the assumptions stated beneath each chart. They do not represent measured savings.

Practitioner corrections become focused evaluations that help us improve prompts, tools, and our evaluation system. Repeating tasks under consistent conditions helps us understand whether those changes improve performance. Results describe a particular system and task set, not every job in an industry. We plan to extend these benchmarks as our systems improve and we build in additional industries, helping practitioners spend more time on what customers value most. Read how we use practitioner feedback to improve products like Tax AI in Building self-improving tax agents with Codex (opens in a new tab).