Evaluating Document AI Vendors on Accuracy Commitments

Judging document AI vendors on their accuracy commitments means asking who pays when extraction goes wrong. Benchmark claims cannot answer that question. A per-field accuracy commitment in a service-level agreement, or SLA, can. Without a defined measure and remedy in the contract, the vendor keeps the claim and the customer takes the extraction risk.

Why benchmark accuracy numbers fail in production document workflows

Document AI buyers face more accuracy claims but less useful signal. Headline figures converge at the top of the market, so the numbers that appear to distinguish vendors increasingly look the same. Forrester VP Boris Evelson observed that generative and agentic AI is “becoming an equalizer that challenges vendors' ability to differentiate”. When any capable model can perform well on a clean document, raw extraction accuracy stops being a useful separator. The real distinction lies in what happens when the document is not clean.

Blended averages hide that distinction. A system can extract the vendor name correctly nearly every time while failing systematically on line-item quantities or tax fields. Its overall score can still look strong because success on easy fields masks failure on a critical column. The same problem appears across document types, where clean files conceal poor results on handwritten annotations, rotated scans, or a table that spans a page break. Those failures matter more than the average when they affect the fields a workflow depends on.

Template-based approaches also struggle when scan geometry or labels change. A label that reads “Invoice #” on one supplier’s form reads “Factura No.” on another’s, and regex grows into a maintenance monster. A clean benchmark does not show how much work those changes will take. Choosing on headline accuracy thus bases the choice on the vendor’s strongest data. A useful evaluation has to reverse that choice.

Why the absence of a contractual accuracy commitment transfers all extraction risk to the customer

Refusing to put per-field accuracy in a contract is a decision about risk. The vendor is declining to share the downside of extraction errors. The customer absorbs every mismatch, downstream correction, and compliance exposure. Who bears that risk may be easy to overlook during a polished demo, but it becomes clear when a wrong value enters a working system. The lack of a commitment is not neutral.

A claim costs the vendor nothing when it fails, while an SLA defines a remedy. Standard platform SLAs rarely cover failures of the underlying model, so having an SLA is not the same as having an accuracy commitment. Without an AI-specific addendum, the buyer can accept the model’s failure risk in writing while believing the platform agreement covers it. The contract has to address extraction performance itself.

That distinction matters in accounts payable, mortgage lending, insurance, and healthcare. In regulated, high-volume workflows, an undetected extraction error can propagate into downstream systems, audit records, and compliance filings before anyone catches it. Ask the vendor to put its quoted accuracy figure into the SLA with a defined remedy, because its response reveals more about its willingness to share that risk than a demo does.

How to run an evaluation that produces a reliable signal on your data

Start with your own documents and write the grading rule before seeing any output. That order matters because an evaluation should test a defined requirement. The proof of concept must include the edge cases that break other systems. Handwritten annotations, rotated scans, and tables split across pages belong in the test where they occur in your work.

Then apply the same conditions to every vendor in a standardized environment. When the vendor picks the dataset, environment, and grading rule, the whole test plays to its strengths. Measure per-field accuracy for each document type, and measure table-level accuracy on its own. Good results on headers must not hide failed line items, and getting one document type right must not hide errors on another. This separation produces a result that can support a specific contractual commitment.

Before interpreting accuracy scores, run two basic sanity checks. A character-count check should flag a parsed document whose extracted content is a tiny fraction of the source file size, because that signals a broken parse. A structural check should flag a document right away if it should contain tables but returns zero tables. These checks catch catastrophic parsing failures before a reassuring average can hide them. They expose cases where the parsing process has failed more fundamentally.

Finally, ask for the confidence interval on results for your specific data category. An exceptionally high accuracy claim can indicate an easy test set or data leaked into evaluation. A useful answer explains the test and its limits. Refusal to run the proof of concept on your documents leaves generalization beyond the curated test set unproved. That refusal should stop the evaluation.

What a contractually adequate accuracy commitment looks like in practice

An adequate accuracy commitment specifies the field, document type, measurement method, and remedy. A headline rate attached to an unspecified test corpus leaves the buyer without a clear connection between the promised result and the documents being processed. The negotiated term should make clear what is measured and what happens when the commitment is missed.

The supporting evidence matters as much as the wording of the rate. The questions should cover what was tested, on what data, who graded it, and what the confidence interval is. A strong answer describes the test set and its limits. Those limits define the scope of the commitment and give the buyer a way to distinguish statistical understanding from sales confidence.

A no-training clause belongs in the same contractual review. The commitment that customer data will not be used to train or fine-tune the vendor’s models must appear in the DPA or contract. An FAQ is not a substitute for that term. Confidence scores also need scrutiny, because an operational threshold depends on scores that are calibrated and traceable to the source document. A high-confidence flag attached to a hallucinated value can suppress the human review that would otherwise catch it.

The contract needs to state who owns the outputs and address liability when an extracted value supports a downstream decision that turns out to be wrong. For regulated buyers, that clarity belongs before signing. Together, the accuracy measure, data-use restriction, confidence requirements, and output terms define what the vendor is actually standing behind.

Data security and compliance terms that belong in the same vendor review

An accuracy SLA does not settle the risk of handing over sensitive documents. Document AI vendors process financial records, payslips, mortgage files, insurance claims, and healthcare documents. The terms for how they handle data therefore matter to regulators regardless of how well extraction works. Accuracy and data protection need to be reviewed together when choosing a vendor.

Zero data retention must be an explicit contractual guarantee. A provider’s standard terms often do not guarantee that processed documents are discarded after inference, so a buyer cannot safely treat disposal as the default. The guarantee belongs in the MSA or Data Processing Addendum, with audit rights included. It also needs to remain distinct from the no-training clause. A restriction on training does not itself state that documents will be discarded.

The no-training promise deserves the same evidence standard as the accuracy promise. If it appears only in marketing materials, it carries no contractual weight and can be changed unilaterally. Subprocessor transparency is equally important because the model often belongs to a third party. Ask for the complete subprocessor list, including the model provider, its region, and its data handling terms. Otherwise, the review ends at the platform vendor even though document data may not.

These terms are practical controls over the document-processing relationship. The immediate task is to establish who receives the documents, what they may do with them, and whether they must discard them. An accuracy commitment cannot compensate for a gap in those answers.

The build-vs-buy question and where an accuracy SLA fits in that calculus

Building on raw AI APIs moves responsibility for accuracy inside the company, where the engineering team becomes the vendor without an SLA on itself. The team still has to deal with field-level failures, difficult documents, and downstream corrections. Control over the pipeline is therefore not the same as relief from its risks. The build decision changes who owns the work.

The hidden cost comes in month six, when a new supplier format breaks extraction and the team has moved on. Edge cases get harder to handle faster than most teams expect, so a build that works at first needs ongoing maintenance. That burden belongs in the comparison with a purchased system. Comparing only the initial implementation misses the part of the work most likely to persist.

Building can still be the right choice when document types are proprietary, data residency prohibits third-party platforms, or extraction logic encodes genuine competitive advantage. For standard processes such as accounts payable, mortgage, and insurance claims, the practical default is to buy. Vendors have spent years building in ways to handle edge cases, and a single team cannot reasonably match that work in the time it takes to decide whether to build or buy. A purchased platform with custom API integrations offers a middle path, preserving the accuracy SLA and compliance architecture while allowing bespoke downstream workflow logic.

The RFP questions that expose where a vendor's accuracy commitment actually ends

Before any demo, every vendor should receive the same questions in writing, with the answers collected in one sheet. The evidence column matters most, because an answer without a supporting document is a promise. Written questions also keep the comparison focused on what you need. The goal is to find the boundary of the commitment before negotiating around it.

For accuracy, ask the vendor to state performance on your specific document types and field set, measured against your evaluation criteria. Ask whether it will put that figure in the SLA with a defined remedy if the figure is not met. Require the confidence interval and a description of the test set behind the claim. Then ask whether it will run a proof of concept on your documents, including the worst cases, before signing.

For data handling, ask whether zero data retention is contractually guaranteed in the DPA or merely a default that can change. Ask whether the no-training clause appears in the contract or only in marketing materials. Require the names of the subprocessors that receive your data, their regions, and the terms that govern their handling of it. Each answer needs a document behind it.

For model behavior, ask whether extraction is deterministic and whether the vendor can demonstrate that the same input produces the same output. Ask how exceptions and low-confidence extractions get to human review so they do not pass through unchecked. Ask whether the vendor can produce a per-call audit log showing timestamp, model version, input, output, and retention settings.

Stop the evaluation for refusal to test your data, an undescribed test set, unexplained confidence calibration, marketing-only data terms, or no named incident owner. How the vendor responds is itself evidence. Clear documents, stated limits, and defined remedies demonstrate more operational maturity than polished answers without backing. The vendor must make its accuracy claim an obligation and share responsibility for its failure.