Why the best AI model still scores 56% on accuracy

TLDR

The latest accounting benchmark shows frontier AI models completing real accounting work at just 56.4% accuracy. That’s not an indictment of AI. It’s evidence that accounting is a systems problem, not a model problem.

Models provide intelligence. Reliable accounting requires everything around them: domain expertise, transaction context, company-specific policy, deterministic execution, controls, and human review. The next leap in accounting AI won’t come from prompting a better model. It will come from building the accounting system that makes models reliable enough for production.

Context

Last week, researchers at Mercor published something the accounting profession has needed for a long time: a benchmark that measures AI models against real accounting work instead of exam questions.

The result made headlines: The best frontier model in the world scored 56.4%.

We poured over the research paper, and our reaction may surprise you. This is the most useful piece of AI research an accounting leader could read this year, not because it shows AI failing, but because it measures the right thing precisely. It also puts a number on a question we get asked in nearly every conversation: if a raw model scores 56% on month-end work, how does Max, our accounting agent, run the same categories of work in production at 99%+ accuracy?

The answer in accounting is not a better model. It is the accounting specific platform we built around the model. This piece explains the gap.

What APEX-Accounting actually measured

Most AI benchmarks in accounting have measured knowledge. Models have been acing those for years. GPT-4 was scoring 85% across CPA, CMA, CIA, and Enrolled Agent exam sections back in 2023. If passing the CPA exam made software audit-ready, this would be a solved problem.

APEX-Accounting measured work.

The benchmark consists of 160 tasks built by 42 accounting experts with a median of 11 years of experience, over half of them from Big Four firms. Each task lives inside a realistic company frozen at month-end: an ERP, bank statements, subledgers, agings, payroll exports, an average of 7.5 source files per task. The tasks are the ones your team recognizes on sight:

• Reconciliations: tie sources together, identify differences, explain and correct breaks

• Data entry: post journal entries, vendor bills, invoices

• Variance analysis: compare actuals to budget and prior periods, explain the drivers

• Schedules and accruals: build supporting schedules, calculate accruals, carry balances forward

Every task was solved by an expert first, then graded against a rubric averaging roughly 14 binary pass or fail criteria. No partial credit for a rec that almost ties.

Nine frontier models ran every task eight times each, more than 11,000 attempts in total. The best performer met 56.4% of criteria on average. Even on reconciliation, the strongest category, the top model reached 59.4%. On schedules and accruals, it managed 51%.

The number that should get more attention

The 56% headline is what traveled. But the finding that matters more to a controller is about consistency.

Because every model attempted every task eight times, the researchers could measure something benchmarks usually hide: how often a model produces a fully correct result every time. The answer: no model achieved a perfect result across all eight runs on more than 2.6% of tasks.

Read that again through an accountant's eyes. A process that produces a correct reconciliation on Tuesday and a subtly different one on Wednesday, from identical inputs, is not 56% of the way to done. It is not a control at all.

This is the distinction the accounting profession has always understood and the AI industry is only now measuring. Simply put, accounting is not an intelligence problem but a reliability problem. The challenge is not producing the right answer once. It is producing accurate, consistent, and auditable outputs across thousands of interconnected workflows, month after month, without errors compounding over time.

Why raw models miss: the hard part is accounting system and context, not calculation

The paper's failure analysis is worth sitting with. When the researchers annotated thousands of failed attempts, reasoning failures dominated. Better prompting and better tooling, the usual fixes, accounted for only about a quarter of failures combined. When the team swapped in a purpose-built execution harness, average scores moved by roughly one percentage point.

There is no single fix, and that matches what two decades in accounting taught us before we wrote a line of code: the hard part of accounting has never been the calculation. It is the context around the model. Accurate, auditable agentic automation needs context in three forms, and a general-purpose model has none of them.

• Financial and cross-system context. Every reconciliation, accrual, and journal entry involves judgment. You validate inputs, resolve inconsistencies, and account for edge cases that do not fit a predefined workflow. A model working from a stack of exports sees balances. It does not see the transactions, documents, and system state behind them.

• Company-specific policy. A general-purpose model does not run on your company's reality. Accounting is full of local rules: how your team estimates an accrual when the invoice has not arrived, which expenses route to which department, what is your materiality thresholds. A confident-sounding guess that touches the GL is a material risk.

• Domain knowledge. A general-purpose agent can read a spreadsheet and write a plausible journal entry. It won’t know that a vendor accrual should be estimated from historical run-rate when the invoice has not arrived, that a fixed asset should depreciate on a waterfall schedule from its in-service date, or that a commission must be capitalized under ASC 340-40 and amortized over the expected customer life rather than the contract term.

None of this makes the models unimpressive. A general-purpose system half-solving expert-built month-end tasks would have been unthinkable three years ago. But a model, however capable, is not a system of work. So what does a system of work look like? That is what we built Max to be.

From generic LLMs to accounting system

The benchmark naturally raises a question. If the best frontier model scores 56% on realistic accounting work, how are accounting teams successfully deploying AI in production?

The answer isn’t a different language model. Most modern accounting platforms use the same frontier models available to everyone else. The difference is everything wrapped around the model.

A production accounting system doesn’t ask a language model to “do accounting.” It combines the model with transaction-level context, company-specific accounting policies, deterministic calculations, native integrations, validation checks, human review, and complete audit trails.

The model provides intelligence. The system provides reliability. That’s the difference between an impressive demo and accounting work that can be reviewed, approved, and posted.

What a production accounting system requires

Six mechanisms, each one addressing a failure mode the benchmark exposed.

• Native tool calling, not prompting over data. Max actively performs accounting work by calling proprietary tools across the Maxima platform. Rather than relying on a single prompt, it decides which tools to use, what information it still needs, and what should happen next to move a workflow from preparation through review. The benchmark's models had to reason their way through 7.5 loose files per task. Max operates a system built for the work.

• Grounded in source data. Max works directly off live system records, not copied, exported, or stale data. The Maxima platform unifies financial data across banks, billing platforms, credit cards, and contracts through native integrations and a unified finance knowledge graph. That gives Max transaction-level context, not just balances, so the work reflects what is actually in your systems at the moment it runs.

• Deterministic where it matters. Calculations, mappings, validations, and accounting logic execute directly rather than being predicted by a language model. Inputs are validated before posting, and outputs are checked against your accounting rules and controls. When a schedule ties, it ties because arithmetic tied it. This is how the drift that compounds across a benchmark task gets engineered out entirely.

• Policy-driven by design. Max learns your company's accounting logic instead of hard-coding it. Your team teaches it in natural language: the CEO's T&E above $1,000 goes to Sales & Marketing, not G&A. For webinar opex accruals, ping the buyer twice, and if no response, use the average of the last three months. Once the logic is encoded, Max applies it consistently across entities, periods, and transaction volumes, and adapts as policies evolve. This is the answer to the consistency problem: the same policy, enforced the same way, every run.

• Human-reviewed when judgment is required. Max operates as a co-preparer, not an autonomous decision-maker. On deterministic workflows where the treatment is known and repeatable, like transaction matching, reconciliations, prepaids, and recurring journal entries, it prepares end to end. On workflows involving estimation, like payroll accruals or inventory rollforwards, it prepares the work, surfaces its assumptions, and routes to an accountant for review and approval. It does not go rogue or make silent assumptions. It knows when to prepare and when to involve an accountant.

• Fully auditable. Every action Max takes is captured in a complete audit trail: the source data used, the accounting logic applied, the supporting documents referenced, the validations performed, the exceptions encountered, and the reviewer approvals. Any output can be re-performed step by step. Instead of reconstructing how work was performed weeks later, your auditor reviews a complete record of what happened, why, who approved it, and the evidence behind it. The result is often a stronger control environment than the spreadsheet-driven process it replaces.

  
  

The layer the benchmark could not test: deployment

There is one more ingredient, and it is people.

During implementation, Maxima's forward deployed team, which includes former accountants and auditors, works with your team to encode the policies and procedures Max should follow: how to estimate each accrual, how to classify expenses, when to chase support, even what tone to use in flux explanations. Every workflow is tested against your prior-period actuals and tied out to what your team produced manually before it runs live.

And the evaluation never stops. We continuously test Max against real accounting workflows, edge cases, and expected outcomes, so performance improves where it matters most: real accounting work, not benchmark tasks.

We are equally clear about where Max stops. Complex equity accounting, unusual transactions, and situations where multiple accounting treatments may be appropriate remain the accountant's domain. Max prepares the work. Accountants exercise judgment and approval. That boundary is not a limitation we are waiting to remove. It is how the profession's oldest control, segregation of duties, extends to a new kind of preparer.

How to use this paper if you lead an accounting team

If your CEO or board has been asking about AI, and at most companies we talk to they have, APEX-Accounting is the most useful framing device you could bring to that conversation.

Let it recalibrate the discussion rather than end it. The wrong takeaway is that AI cannot do accounting. The right takeaway is that raw model access, a chat window and a stack of exports, is the wrong deployment shape for work that must be exact, repeatable, and evidenced.

Then borrow the benchmark's standard when you evaluate vendors, including us. Ask how outputs stay consistent run to run. Ask to see the lineage behind a prepared entry. Ask what happens when the agent hits an exception. And ask to run it on your own data, tied out against what your team produced by hand. Any vendor building a real system will welcome the test.

The model provides the intelligence

Fifty-six percent is not a verdict on AI in accounting. It is a precise measurement of the distance between a model and a system of work. Accountants, of all professionals, are equipped to appreciate that distance. The profession has never trusted a preparer on intelligence alone. It trusts the process around the preparer: the support attached, the review performed, the evidence retained.

That standard should not be lowered for AI. It should be engineered for. The model provides the intelligence. Maxima provides the accounting system that makes that intelligence domain-specific and reliable. That is the distance between 56% and 99%+, and it is measured in engineering, controls, and accounting judgment, not in model releases.

The models will keep getting better. We are counting on it, and so should you. But the last miles between a benchmark score and a close you can sign will always be built, not prompted.

Read the APEX-Accounting paper here [link: https://arxiv.org/html/2607.27189v2].