Ground Truth Construction for Document Extraction Evaluation

Document extraction pipelines get evaluated against a number, and that number gets treated as fact. It shouldn't be. The number only means something if the ground truth behind it was built correctly, an assumption that produces most teams' comparisons of models, tuning of prompts, and sign-offs on accuracy claims for production use without ever being interrogated first.

The stakes are not small. Grand View Research put the intelligent document processing market at $3.0 billion in 2025, with a projected climb to $29.7 billion by 2033, a 33.8% compound annual growth rate (forecasters vary widely here depending on how they scope the category, so treat the exact multiple with some caution, but the direction is not in dispute). BFSI, banking, financial services, and insurance, is expected to hold 32.7% of that market in 2026, the largest single segment. Loan files, KYC forms, claims documents, financial statements: these are exactly the places where a wrong field value doesn't just look bad in a report, it triggers a wrong loan decision or a missed compliance flag. At that volume, a measurement error in your evaluation isn't an academic footnote. It's a production risk multiplied across every document that flows through the pipeline afterward.

Most teams spend their energy on model selection and prompt tuning, treating those as the levers that determine accuracy. They're levers, sure. But the ceiling on what any of that tuning can prove is somewhere else entirely: the quality of the ground truth doing the measuring. A pipeline can't be held accountable to a standard that the ground truth itself can't represent. What follows is a walk through how that ground truth actually gets built, and where it breaks.

What ground truth means at the field level, and why document-level thinking produces misleading results

Ground truth isn't a stamp of "correct" or "incorrect" slapped on a whole document. It's a structured record of the right value for every named field, on every document, in the set being evaluated. That distinction changes what a single accuracy number can actually tell you.

Field-level accuracy answers one question: what fraction of individual extracted fields matched their ground truth value? Document-level accuracy answers a different one: what fraction of documents had every single field correct? These two numbers are related, but not in a straight line, and the relationship surprises almost everyone who hasn't run the math themselves. A field-level accuracy study using 200 documents found overall average accuracy at 94.72%, with individual field accuracy ranging from 87.94% (a field called Season) up to 100% for the strongest fields. That spread, nearly 13 points between the weakest and strongest field, is itself a finding. Document-level accuracy would have flattened all of that into one pass/fail number and hidden exactly where the pipeline is weak.

A sample of 200 records carries a margin of error around ±4% at a 90% confidence level. Teams often report a single accuracy figure from a small evaluation set as though it were exact, when the honest version of that number comes with a range attached. Smaller samples aren't wrong to use, they're just narrower in what they can claim.

There's also a distinction that gets collapsed too often: OCR accuracy and extraction accuracy are not the same thing. OCR reads characters off a page. Extraction assigns values to named fields. A system can read every character on a form correctly and still put the right number in the wrong field, or misinterpret which line item a dollar figure belongs to. Ground truth has to be defined at the level you actually care about, or the evaluation ends up measuring something adjacent to the real question.

Then there's schema compliance versus value correctness, a gap that trips up automated evaluation constantly. A model can return perfectly valid JSON, structurally sound, every key present, every type correct, while still extracting the wrong values into every field. Benchmarks like ExtractBench and LLMStructBench evaluate on value correctness rather than schema validity, which is the right instinct, but each is built on a single source domain and doesn't test how accuracy holds up across different document types. That's a gap ground truth construction has to account for directly, not assume away.

None of this is a natural, discoverable object waiting to be found in the documents. Field-level ground truth is something a team builds, with explicit decisions about field definitions, normalization rules, and what counts as an acceptable equivalence, made before a single document gets annotated.

The core decisions in ground truth design before annotation starts

Every field that will be scored needs a name, a type, and a boundary, decided before anyone starts labeling. "Vendor address": does that include the country? The zip code? If two annotators answer that question differently, the disagreement will appear later in the data and look like annotation error. It isn't. It's a design failure that happened earlier and got discovered late.

Normalization rules need to be written down, not assumed, and they need to cover several recurring fights. Case normalization matters for any string field (title accuracy in table evaluation, for instance, gets defined only after trimming whitespace and normalizing case). Date formats need equivalence rules, since "03/04/2025" and "4 March 2025" might be the same date or might not be, depending on locale. Numbers need rules too: is "1,000" the same as "1000" or "1.000"? That depends entirely on which country's convention is in play, and an evaluation that doesn't decide this in advance will silently punish correct extractions that happen to use the wrong locale format. Missing values versus not-applicable values need separate handling as well; conflating them quietly erases a real distinction.

Acceptable equivalence decisions carry more weight than they look like they should. Is "USA" the same as "United States" for scoring purposes? That's not a trivial edge case; it's a decision that determines whether a genuine production error gets counted as one or gets waved through as a formatting quirk.

All of this has to be finalized and frozen before annotation starts. Ground truth built while the extraction rules are still being revised ends up reflecting an earlier, already-obsolete version of what's supposedly being measured, which contaminates every result that follows.

Annotate a first batch (the VERITAS framework, for archival document evaluation, uses 100 pages as a first round), run a full evaluation pass against it, surface every disagreement, and update the rules to resolve them. Only after that should additional samples get annotated, under the now-frozen rule set. This catches ambiguity in the field definitions while it's still cheap to fix, rather than after thousands of documents have already been labeled under a shaky standard.

These are editorial decisions with engineering consequences, and they deserve to be documented as part of the evaluation artifact itself, not left as tribal knowledge sitting in an annotator's head.

Sampling strategy: how the documents you annotate determine what your evaluation can see

A convenience sample, the clean text-layer PDFs sitting in a shared drive, with predictable layouts and no scanning artifacts, will produce an evaluation that's blind to almost everything that goes wrong in production. That's the uncomfortable truth about sampling: the evaluation can only see what got put into it.

Stratified sampling from the real document corpus is the fix, and it needs to cover the actual distribution production sees: dense prose, footnotes, tables, multi-column layouts, mixed text and image, handwriting, scanned pages, embedded fonts, and low-quality scans. The VERITAS archival framework uses stratified sampling specifically to guarantee representation across dense prose, footnotes, title pages, illustrations, and tables. The same logic applies whether the domain is archival records or commercial loan files.

OmniDocBench, a CVPR 2025 benchmark now at version 1.7 as of April 2026, offers a useful reference point for what adequate coverage looks like at scale: 1,651 PDF pages spanning 10 document types, 5 layout types, and 5 language types. Teams building a private evaluation set don't need to match that scale, but the category breakdown is a useful checklist to study. Version 1.6 of that same benchmark added 296 new pages specifically to cover harder cases: complex nested tables, dense mathematical formula layouts, unconventional structures. The fact that expansion was necessary says something on its own, namely that the earlier version's coverage of hard cases wasn't good enough.

Failure modes cluster by domain, and that clustering is the real argument against a one-size-fits-all evaluation set. Finance and certificate documents tend to concentrate failures around annotation contamination, loss of formula semantics, and seal recognition. STEM pages concentrate failures around notation fidelity and formula formatting. A generic evaluation set built for one domain will not surface the failure modes that matter in another, no matter how large it is.

The practical move is to identify hard-case categories before sampling, not after looking at results. If a layout type is under-represented in the evaluation set, the pipeline will look better on that type than it actually performs, and no amount of downstream analysis fixes a blind spot baked into the sample itself.

Synthetic and public ground truth carry a separate risk: training data contamination. If the documents used for evaluation are the same ones (or close variants) a model saw during training, the model may be recalling rather than extracting. One arXiv benchmarking framework (arXiv:2603.18652) deliberately sources tables from December 2025 arXiv papers specifically to sit outside the training window of established models, reducing the odds that strong performance is memorization rather than genuine extraction capability.

Manual annotation: the annotator agreement problem and how to control it

Manually labeled ground truth gets treated as correct by definition, which creates a quiet problem: any disagreement between two annotators doesn't get flagged as a quality issue, it gets absorbed silently into whichever label won, and from that point forward it's treated as fact.

The standard defense against this is multiple independent annotators. A representative methodology has independent reviewers review a random sample of 200 records, then resolves disagreements through a fixed adjudication rule, never by averaging or splitting the difference. The VERITAS framework uses three annotators working from shared transcription guidelines, and the guidelines are the part that actually does the work. Annotators without a written, shared rule set will diverge on ambiguous cases in ways that look, on the surface, like the document was just hard. It wasn't the document. It was a gap in the rules.

Inter-annotator agreement should get measured and reported alongside the ground truth itself, not treated as a private quality check that stays internal to the labeling team. Low agreement on a specific field is a signal about that field's definition, not a signal that the field is inherently difficult. Conflating those two things means a team might spend engineering hours trying to fix an extraction model for a field that was never well-defined to begin with.

Fields where annotators disagree with each other are exactly the fields where the extraction model gets evaluated against noise. Whatever the ground truth's internal disagreement rate is on a given field, that becomes a noise floor, and no evaluation result on that field can be more meaningful than the ground truth that produced it.

Generalist annotation programs often lack the domain expertise needed to catch errors in specialized fields such as tax IDs, account numbers, IBAN formats, and medical terminology. Tax IDs, account numbers, IBAN formats, medical terminology: these require specific domain knowledge that a generalist crowdworker often doesn't have, and errors in these specialized fields tend to go undetected without a reviewer who actually knows the domain. Annotation tooling should also enforce whatever normalization rules got set during the design phase; a free-text entry box for annotators just recreates the exact ambiguity the rules were supposed to eliminate in the first place.

Synthetic and automated ground truth: where it is legitimate and where it introduces its own bias

Synthetic ground truth sidesteps the cost of manual annotation entirely, when it's possible to generate. If a document gets rendered from a LaTeX source or a known template, the correct field values are exact by construction and require zero human judgment to confirm.

The arXiv table benchmarking framework mentioned earlier embeds real tables pulled from arXiv papers into synthetically generated PDFs, which gives exact LaTeX ground truth without any manual labeling step. Across 100 synthetic documents containing 451 tables total, this produces a precision of measurement that would be slow and expensive to reproduce by hand.

The limitation is structural, not incidental. Synthetic documents represent the rendering path, the process of turning clean source data into a clean PDF. They don't represent the degradation path: scanned pages, phone-camera photographs of paper forms, embedded fonts that fail to render correctly, documents that have been copied and re-scanned multiple generations down the line. A system evaluated only against synthetic ground truth has never been tested against the failure modes that actually dominate a real production queue.

A hybrid approach splits the difference sensibly: synthetic ground truth for the easy-case baseline, manual annotation reserved for the hard-case stratum. That gives exact measurement where it's cheap to get (the clean documents) and puts the more expensive human effort where it matters most, in the long tail of difficult cases that determine whether a pipeline is actually production-ready.

For semantic evaluation, particularly of tables, LLM-as-a-judge scoring has shown a notably stronger correlation with human judgment than older rule-based metrics. One comparison found LLM-based evaluation reaching a Pearson correlation of r=0.93 against human ratings, compared to r=0.68 for TEDS and r=0.70 for GriTS, validated against 1,554 human ratings across 518 table pairs. That gap between r=0.93 and r=0.68/r=0.70 argues for LLM-based judging specifically in cases where semantic equivalence takes precedence over exact string matching.

The markdown-to-markdown comparison used in OmniDocBench, where a model's full-page markdown output gets compared against a markdown ground truth, is a document-level evaluation, not a field-level one. It's useful for what it measures, but teams shouldn't mistake it for per-field accuracy, because it isn't built to answer that question.

Training data contamination deserves a second mention here specifically because automated benchmarks built on public datasets are the most exposed to it. A model scoring artificially high on a public dataset it was partly trained on isn't demonstrating extraction skill, it's demonstrating memory. That's precisely why the December 2025 sourcing choice in the arXiv benchmark matters as much as it does.

Metrics that actually hold extraction accountable: WER, CER, per-field accuracy, and table-specific scores

Word Error Rate counts the minimum number of word-level insertions, deletions, and substitutions needed to turn a prediction into the ground truth, normalized by total word count. It's a reasonable fit for text-heavy fields and OCR transcription quality, but it's a poor fit for structured fields, where getting one digit wrong in an account number isn't a small proportional error, it's a total failure of that field.

Character Error Rate works the same way at the character level, and it's the sharper tool for fields where character-level precision is the whole point: account numbers, postal codes, tax identifiers. The VERITAS archival framework uses both WER and CER together as complementary measures rather than picking one.

Per-field accuracy, a binary match after normalization, aggregated across fields and documents, is the metric that exposes the field-level versus document-level gap most directly, and it's arguably the single most important number for structured extraction evaluation.

Table extraction gets its own composite scoring approach in several evaluation setups, defined with sub-metrics that include title accuracy, checked after trimming whitespace and normalizing case, as well as structural and content dimensions, how many ground truth rows show up in the extraction, regardless of order. Cell accuracy checks fine-grained individual cell matches, and it's the most sensitive of the four to errors buried in dense tables.

Rule-based structural metrics like TEDS and GriTS have a specific weakness: they penalize a table that's structurally different from the ground truth even when it's semantically identical. A parser that correctly captures every cell value but organizes them in a non-standard layout can score poorly on these metrics despite being right, which is exactly why LLM-as-a-judge scoring tends to track human judgment more closely.

Schema compliance is not a metric, worth restating plainly. Valid JSON output isn't correct extraction, and value correctness has to be checked against ground truth directly, never against the schema alone. No single metric covers every failure mode. WER and CER for text fields, binary per-field accuracy for structured fields, semantic table metrics for tabular content: the evaluation suite needs to match the actual mix of document types running through production, not default to whichever metric is easiest to compute.

Confidence scores and how they interact with ground truth in production evaluation

A well-designed extraction system attaches a confidence score to every field it extracts. High-confidence fields route straight downstream; low-confidence fields get pulled into a human review queue. That routing decision is only as good as the confidence score behind it.

Calibration is the word for whether that score actually means anything. A model reporting 95% confidence on fields it gets wrong one time in five is worse than having no confidence score at all, because it actively suppresses the review that would have caught those errors. False confidence doesn't just fail to help, it does active harm by hiding the exact cases that needed a second look.

One 2026 research result, from a multi-signal confidence engine called EXTRACTCONF, found that at 80% coverage (meaning 80% of fields get auto-approved without human review), the system reached 99.1% automated accuracy. This gets measured through a metric called AURC, and a system posting an AURC of 0.042 delivers close to perfect accuracy at the coverage level that actually matters for production throughput. That's the real tradeoff confidence thresholds control: push coverage higher and accuracy drops, pull coverage down and more documents sit in a human queue.

Calibration quality isn't uniform across field types. Calibration behavior tends to vary across field types, with some categories proving more reliable than others: the model claims certainty it hasn't earned. How to set appropriate routing thresholds for different field types remains an open practical question, and a single blanket threshold applied across every field type is likely leaving accuracy on the table somewhere.

Extraction errors in document processing are mostly caused by the document, not the model. A frontier language model reading a smudged, poorly scanned page will generate high-probability tokens describing what it thinks the OCR noise says, confident and wrong at the same time. Confidence calibration that only accounts for model uncertainty and ignores input quality is measuring the wrong variable.

None of this calibration work happens without ground truth. Without labeled examples where the right answer is already known, there's no way to check whether a confidence score actually predicts accuracy or is just noise dressed up as a number. Confidence calibration is an evaluation task in its own right, not a feature that ships and takes care of itself. And per ConfBench (arXiv:2608.01792, 2026), no systematic benchmark currently exists for evaluating confidence calibration in key information extraction within intelligent document processing. That means vendor-reported calibration numbers can't be taken on faith. They have to be measured against a team's own domain-specific ground truth, because right now, nobody else is doing it for them.