Benchmarking PDF Extraction Tools

Abstract

Document ingestion is the foundation of every downstream capability at Needl.ai — retrieval, financial question-answering, report automation, and analyst workflows all consume the markdown produced by an upstream PDF-to-text extraction step. The fidelity of that step bounds the quality of everything built on top of it. This report establishes a rigorous, reproducible benchmark for measuring extraction quality and applies it to 42 extraction libraries and vision–language models, spanning classical CPU parsers (Apache Tika, PyMuPDF, Docling, Marker) and modern vision models (the Gemini 3 family, Claude, GPT, Qwen, Kimi, Llama 4, Nova, and others).

We define seven evaluation criteria derived from a survey of the leading academic document-extraction benchmarks (OmniDocBench, SCORE-Bench, ParseBench, olmOCR-Bench, DocVQA, OCRBench v2, and the PubTabNet/TEDS line of work), and adapt them to the financial-document workload that dominates Needl's production traffic. Each criterion is scored against a single human-verified golden markdown using either deterministic code metrics or a golden-grounded LLM judge with majority voting. The criteria cover text fidelity, content coverage/omission, hallucination, table detection, table structure, table cell accuracy, and chart-to-data understanding.

The central finding is consistent across every criterion: a compact group of high-quality vision models — gemini-3-flash-preview (with and without thinking), kimi-k2.5, qwen3-vl-235b, claude-sonnet46, claude-opus-4.8, and the gemini-3.1-flash-lite pair — leads on all dimensions simultaneously. Classical CPU text extractors recover the words (high coverage) but lose document structure (low fidelity, near-zero table scores). The weakest tools paraphrase or restructure content so heavily that they drop 40–53% of it and emit chart data trapped inside image references. This report documents the methodology in full detail so that the benchmark can be re-run, audited, and extended.

We additionally measure the cost and latency of each library and combine them with quality into a value frontier. Cost spans more than three orders of magnitude (from free local parsers to ~$1.78 per document for GPT-5.5), and the analysis identifies gemini-3.1-flash-lite as the standout value option — near-top quality at roughly half the cost of the leader and the fastest turnaround of any high-quality model — while gemini-3-flash-preview leads the cost-effective cloud tier where maximum quality justifies a modest premium. Claude Opus 4.8 posts the highest composite quality of any library, while GPT-5.5 pairs top-tier fidelity with systematic financial-table value errors.

Model catalogue

Every extractor benchmarked in this report, with the category it belongs to and the colour used for it in the charts. The six categories collapse to four chart colours: closed and open-weight frontier LLMs and the cloud OCR APIs share the frontier-LLM colour; self-hosted OCR/VLM models get their own colour; and the GPU and CPU parsing libraries each get theirs.

1. Introduction and motivation

1.1 The problem

Needl.ai ingests a heterogeneous stream of financial documents: BSE filings and rating reports, annual reports, 10-K and 10-Q filings, earnings-call transcripts, press releases, sell-side broker/research reports, investor presentations, and Word documents. Each of these is converted into markdown before any model reasons over it. A large number of extraction libraries exist — both classical CPU-based parsers and vision–language models (VLMs) — and they vary enormously in how faithfully they preserve the content and structure of the source PDF.

The governing question is deceptively simple: which extraction library produces the highest-quality markdown for our document mix? Answering it rigorously requires a precise, defensible definition of "quality" — one that is decomposed into measurable dimensions rather than collapsed into a single opaque judgment.

1.2 How the evaluation criteria were derived

Rather than inventing criteria from scratch, we studied how extraction libraries themselves substantiate their quality claims. Every major library reports scores against one or more dedicated academic benchmarks, each of which defines its own evaluation dimensions. We surveyed those benchmarks and adopted the dimensions that are both consistently measured and directly relevant to a financial-document workload.

The benchmarks surveyed, and the criteria each contributes, are summarised below.

Table 1. Academic benchmarks surveyed and the evaluation dimensions each contributes to the Needl criteria set.

1.3 The distilled criteria

Distilling the survey against the financial-document workload yields the following candidate criteria. The final column records the measurement modality chosen for each.

Table 2. The candidate criteria distilled from the benchmark survey, with the measurement modality ultimately adopted for each.

1.4 The dropped criterion: KIE / field-level accuracy

The survey originally included Key Information Extraction (KIE) — whether named fields such as EPS, revenue, reporting date or issuer were extracted correctly. It was dropped as a standalone criterion for three reasons:

• Redundancy. When text fidelity is high, hallucination is low, and table structure and cell accuracy are good, field values are already correct by construction. KIE rarely separates two libraries that those three do not already separate.
• Wrong layer. An extraction library's job is "PDF → faithful markdown"; it does not know which fields are special. KIE is a downstream QA/retrieval metric, not an extraction-fidelity metric, and folding it in muddies what is being ranked.
• High maintenance, low signal. KIE requires a hand-defined field schema and a golden value per field, per document type — costly, arbitrary annotation for little additional discriminating power.

The intent of KIE — getting the revenue/EPS number right — is preserved by cell spatial accuracy together with the high-severity tier of the hallucination check (wrong or invented numbers). Nothing of value is lost.

2. Scoring function design

Defining what to measure (Section 1) is only half of the design. The second half is deciding how the per-criterion scores are combined into a comparable judgment, and how the evaluation dataset is composed so that strengths and weaknesses are not averaged into invisibility.

2.1 How major benchmarks compute a composite score

Every benchmark reduces multiple criteria to a single comparable number, and they fall into a small number of patterns:

Table 3. Composite-scoring strategies used by the surveyed benchmarks, and the limitation each carries for a financial-document workload.

The common structure is a weighted (or unweighted) average of per-criterion scores, where each criterion resolves to either a deterministic metric (TEDS, NED, F1), a binary pass/fail unit test, or an LLM-judge score. The Needl scoring function adopts this structure, with two deliberate refinements described next.

2.2 The Needl scoring function

The per-document score is the mean of only the criteria that apply to that document. A text-only BSE filing has no charts, so the chart criterion is excluded from its average rather than scored as a free 1.0. This "applicable-criteria-only" averaging is implemented in the scoring module and prevents libraries from being rewarded for capabilities the document never exercised.

applicable       = { c ∈ criteria : c applies to document d }
score(lib, d)    = ( 1 / |applicable| ) · Σ_{c ∈ applicable} s_c(lib, d)

leaderboard(lib) = Σ_{b ∈ buckets} w_b · mean_{d ∈ b} score(lib, d)
                   where Σ_b w_b = 1   (workload weights, §2.4)

Two measurement-design decisions distinguish the Needl function from the off-the-shelf benchmarks:

• Dual table-structure measurement. Table structure is computed both deterministically (a lightweight TEDS variant) and, where divergence is suspected, cross-checked by the judge. Divergences greater than two points are flagged for inspection rather than silently averaged.
• Multi-run LLM judging. Every LLM-judge criterion is run multiple times with majority voting (three runs for hallucination, two for visualizations), and the standard deviation is monitored so that an unstable judgment is caught rather than trusted.

2.3 Criterion measurement modalities

Each criterion is computed by exactly one of three mechanisms. The split between deterministic code and LLM judgment is the most important design choice in the benchmark, because deterministic metrics are free, reproducible and run on every library, whereas judge calls are reserved for the dimensions where code cannot perceive meaning.

Table 4. Measurement mechanism for each criterion. Deterministic metrics run on all 42 libraries at zero marginal cost; LLM-judge calls are reserved for meaning-level perception.

The metrics referenced throughout Sections 2.2 and 4 are defined once here for reference. Each row gives the acronym, the criterion it serves, how the score is computed, and its range and direction (higher- or lower-is-better).

Table 4a. Metric and acronym definitions — what each score measures and how it is computed.

2.4 Dataset composition by document type

Averaging across a random document mix hides failure modes: a library that excels at plain text but fails at tables looks merely "average." To prevent this, the evaluation corpus is bucketed by document type, and the overall leaderboard weights each bucket by its share of Needl's production ingestion volume. Bucketing both isolates per-type strengths and makes the leaderboard representative of real traffic.

Table 5. The eight document-type buckets, the criteria each is designed to exercise, the estimated share of Needl production volume, and the target document count per bucket.

The workload weights drive the leaderboard: a library's overall score is the bucket-weighted mean of its per-bucket scores. High-weight buckets (BSE filings ~30%, annual reports ~25%, broker reports ~20%) therefore dominate the ranking, while low-weight buckets (10-K/10-Q ~2%, DOCX ~1%) contribute marginally. The current results in this report were produced on a single representative benchmark document; the bucketed multi-document corpus is the basis for the production leaderboard.

3. Experimental setup

3.1 Benchmark document and golden reference

All scores in this report are computed on a single representative financial document (17 pages, ~3.7 MB), which combines dense prose, ~40 tables (of which ~20 are scorable real tables), and 18 distinct visualizations. The ground truth is a single human-verified golden markdown representing the correct extraction of that PDF.

Human verification focused primarily on numeric correctness, with a combination of manual checking and several LLM-as-judge passes used to surface and correct errors before the golden was frozen. As with any hand-built reference, the golden is not perfectly infallible: a small number of residual issues were found during the study (Section 7), including a misspelt plant location and isolated cases where the extracted text was actually more correct than the golden. These were handled by recomputation and are documented for transparency.

Approximately 42 libraries each produce their own markdown for this PDF — CPU tools such as Tika, PyMuPDF and Docling, and vision models such as the Gemini, Claude, GPT, Qwen and Kimi families — and each is scored against the golden. Raw per-library outputs and result JSONs are version-controlled.

3.2 Shared text normalisation

To ensure that only genuine content and ordering differences are penalised — not cosmetic formatting — every metric is computed after a shared normalisation pipeline. This is applied identically to the golden and to each library output before scoring.

• Strip markdown and HTML markup.
• Fold smart punctuation to ASCII (curly quotes → straight quotes; en/em dashes → hyphen).
• Unicode NFC normalisation, lowercasing, and whitespace collapse.

Word-level metrics apply two additional steps: numbers are normalised (e.g. the Indian-grouped 1,41,793 becomes 141793), pure-symbol tokens are dropped, and — for coverage only — stopwords are removed.

4. Evaluation criteria, formulas, and results

This section presents each criterion in turn: what was done, the formula in computational notation, the per-library scores, and the observations. Unless explicitly noted, all scores follow the convention higher = better.

4.1 Text fidelity — "did it transcribe the words correctly?"

What we did

Text fidelity measures how accurately the words of the document were transcribed. The primary ranking metric is character-level Normalised Edit Distance (NED) against the golden, reported as a similarity. Two secondary signals are reported alongside it: ANLS (the DocVQA metric, with a 0.5 acceptance floor) and word-level WER. All three are computed on the shared-cleaned text so that only real content and ordering differences register. The design follows OmniDocBench (NED), DocVQA (ANLS) and the OCR/ASR literature (WER).

Formula

NED             = Levenshtein(golden, library) / max(|golden|, |library|)
text_similarity = 1 − NED                       # primary rank metric, higher = better
ANLS            = (1 − NED) if (1 − NED) ≥ 0.5 else 0
WER             = word_edit_distance(golden, library) / golden_word_count   # lower = better

Results

Table 6. Text-fidelity scores. ANLS equals text similarity above the 0.5 floor and drops to 0 below it (pymupdf4llm sits exactly at the floor: 1−NED ≈ 0.49 → ANLS 0). WER is the only lower-is-better column and can exceed 1 when output is far longer than the golden (ministral3-14b, 1.14).

Observations

• Strong vision models lead (0.85–0.90): the Gemini-3 family, kimi-k2.5, qwen3-vl-235b, claude-sonnet46, claude-opus-4.8 and gpt-5.5 reproduce wording, order and layout closely.
• Low fidelity ≠ missing content. CPU text tools (pymupdf4llm 0.50, liteparse 0.43, tika 0.59) score low despite high coverage — they extract the words but in a different order/layout, and NED is character-aligned. These must be cross-read with the coverage metric (Section 4.2).
• Bottom cluster genuinely garbles text: ministral3, nemotron, markitdown, textract, gemma3, got-ocr, internvl2 and pixtral-12b heavily restructure or corrupt the transcription (0.21–0.35 for the worst).
• OCR/VLM parsers span a wide range: olmocr-2-7b is competitive with the strong tier (0.85), while paddleocr-vl, rolmocr and qwen2.5-vl-7b land in the mid band (0.72–0.77).

4.2 Coverage / omission — "how much content did it keep?"

What we did

Coverage measures retention of content independent of order, complementing fidelity. It is an order-independent bag-of-words multiset overlap on content words only. We report the multiset coverage rate, a distinct-term coverage rate, the omission rate, and a list of missed tokens for inspection. Keeping coverage separate from fidelity is essential: a dropped page can cancel out under character-aligned NED but shows up immediately as lost coverage. The metric follows SCORE-Bench's "Tokens Found." It is computed over the entire document's bag of words.

Formula

coverage_rate          = (matching content words) / (total golden content words)   # repeats counted
distinct_coverage_rate = (distinct golden terms found) / (distinct golden terms)
omission_rate          = 1 − coverage_rate

Results

Table 7. Coverage, distinct-term coverage, and omission rate. Omission is the only lower-is-better column.

Observations

• Same leaders as fidelity (>0.95): the top vision models — including claude-opus-4.8 (0.99) and gpt-5.5 (0.97) — capture nearly everything.
• CPU text-dumpers rank high here (pymupdf4llm/liteparse 0.93, tika 0.88) — the opposite of their fidelity rank. They do capture the text; they simply do not preserve the structure.
• Bottom cluster drops 40–53%: gemma3, mineru, mistral-ocr, nemotron and got-ocr summarise or paraphrase instead of transcribing (got-ocr sheds ~76%).
• OCR/VLM parsers vary widely: olmocr-2-7b retains almost all content (0.93), while the weaker open parsers shed 15–76% (paddleocr-vl 0.85 down to got-ocr 0.24).

The fidelity–coverage diagnostic

Reading fidelity and coverage together separates two fundamentally different failure modes. A library can lose points on fidelity either because it dropped content (which also lowers coverage) or because it preserved the content but reordered it (which leaves coverage high). The scatter below makes this explicit: points near the diagonal preserve structure, while points far below it — the CPU text tools — are the structure-loss cases.

4.3 Hallucination (including meaning-changing misplacement)

What we did

This criterion combines two originally separate concerns — invented or wrong content, and meaning-changing misplacement — into one measure, scored two complementary ways. A deterministic signal (free, run on every library) penalises extra tokens, repeated 8-grams, and junk character sets. A golden-grounded LLM judge (gemini-3-flash-preview, three runs with majority vote) flags unsupported assertions at high or low severity; it is instructed to treat the golden as the sole ground truth and to use no outside knowledge or arithmetic. The two signals are intended to disagree — code catches mechanical artefacts, the judge catches semantic errors — so both are reported. Reading order was folded into this criterion, since out-of-order content that changes meaning is a form of misplacement.

Two-model cross-check. For a subset of the libraries — the open OCR/VLM parsers together with claude-opus-4.8 and gpt-5.5 — the golden-grounded audit was run as a two-model pipeline rather than a single judge call. Two independent LLM judges — gemini-3-flash-preview and Claude Opus 4.8 — are each given only the library's extracted markdown and the human-verified golden; the judge is blind to which library produced the output, and is asked simply to report every value, label, date or fact the extraction asserts that the golden does not support. The two judges run separately and produce their own finding lists. A third LLM agent then receives both lists together with the golden, re-verifies each finding against it (dropping any where the extraction actually agrees with the golden), and merges and de-duplicates the remainder into one consolidated list — collapsing the same error that the two judges worded differently (e.g. "2,285 → 2,202" vs "2,285 → 2,202 (Freehold Land)") into a single finding, which naive exact-string matching would otherwise double-count or split apart. Each surviving finding is scored with the same severity-weighted formula used for every other library, so the numbers stay directly comparable; chart and diagram mismatches are counted once per chart to match table-row granularity.

Formula

deterministic = 1 − min( 1, tokens_added_rate + repetition_rate + junk_charset_rate )

LLM judge:      severity_weight(high) = 1.0 , severity_weight(low) = 0.2
                judge_score = 1 / ( 1 + Σ severity_weight per 1000 output words )

Results

Table 8. Hallucination scores. "High" and "Low" count findings by severity; "Total" is their sum. gemini-2.5-flash-lite's judge run is pending (errored on token limit); internvl2-8b is shown with its deterministic score only, as its LLM-judge run did not complete.

Observations

• Cleanest: claude-opus-4.8 (0.76, only 3 findings) and gemini-3-flash-preview (thinking 0.72, non-thinking 0.68) — 5–6 findings each.
• Worst are the heavy-restructuring models: mistral-ocr 0.20, gemma3-27b 0.23, nova-lite 0.25 (40 findings, 32 high), nemotron 0.28; among the open OCR/VLM parsers pixtral-12b and internvl2-26b are noisiest (28–29 findings) while glm-ocr (0.66) is cleanest.
• GPT-5.5 is a striking outlier: despite top-tier fidelity it systematically alters financial-table values (25 findings, mostly across one PPE note table) for a 0.28 score, a pattern that reproduced across two independent extractions.
• Deterministic and judge often disagree. Code catches extra/repeated tokens; the judge catches semantically wrong values and misplacement. Both are kept because each is blind to what the other detects.

Two audits were run on the judge's findings and are documented in full in Section 7 (Reliability): a golden typo correction around the plant-location spelling, and the removal of chart-reading false positives discovered by auditing all 522 findings.

4.4 Tables — detection, structure, and cell accuracy

What we did

Tables are scored on three sub-criteria. A table manifest marks the ~20 scorable real tables out of ~40 golden tables. Detection matches golden tables to library tables by content similarity (the better of character-similarity and token-F1, with a 0.5 threshold), so a correctly extracted table is credited regardless of position. Structure compares the dimensions and filled-cell occupancy pattern of matched tables — a lightweight TEDS that avoids fragile HTML-tree construction. Cell accuracy is a set-based comparison of (row-label, column-header, value) triples, which is tolerant of reordering and rendering differences and replaced an earlier brittle positional match. Structure and cell accuracy are only defined for tables that were found. The design draws on OmniDocBench, SCORE-Bench and PubTabNet.

Formula

detection_recall    = cell-weighted fraction of scorable golden tables found   # primary
detection_precision = match quality over ALL golden tables
detection_F1        = 2 · P · R / (P + R)
structure           = f( row/col dimension similarity, filled-cell occupancy pattern )
cell_accuracy       = matched (row-label, col-header, value) triples / total golden triples

Results

Table 9. Table detection (F1), structure, and cell accuracy. Structure and cell accuracy apply only to tables that were detected.

Observations

• Best all-round: claude-opus-4.8 (0.92 / 0.96 / 1.00) and gemini-3-flash-preview (1.00 / 0.96 / 0.91), with kimi, gemini-3-flash-preview-thinking, gemini-3.5-flash-thinking and qwen close behind — they find every table and place cells correctly.
• Plain-text tools score 0 (tika, liteparse, mistral-ocr; markitdown 0.14): table text is emitted as prose, so detection — and everything downstream of it — is 0.
• Found ≠ correct: gemma3 (0.23–0.24), mineru (0.32), docling/marker (0.36), pymupdf4llm (0.26) and internvl2-26b / pixtral-12b detect tables but mangle cell placement.
• OCR/VLM parsers are mixed on tables: olmocr-2-7b detects tables well (F1 0.94) though its cell placement is weaker (0.51), gpt-5.5 finds every table (F1 0.90) but corrupts cell values (0.88), and got-ocr fails entirely (0/0/0).
• Read cell accuracy next to detection: gemini-2.5-flash has the highest cell accuracy (0.98) but only 0.55 detection — it is accurate on the subset it found, not overall.

A reviewer note (Section 7) records that detection is treated as a binary found/not-found decision per table; cases where a found table is rendered with broken internal structure are penalised by the structure and cell-accuracy sub-criteria rather than by detection.

4.5 Visualization / chart-to-data understanding

What we did

This criterion asks whether numeric values locked inside charts were recovered, not merely whether the chart was described. A hand-authored visualization golden enumerates 18 visualizations (bar, line, pie, donut, KPI cards, an org chart, a matrix and a timeline), each as {id, type, data} with the data taken verbatim from the golden report. An LLM judge (gemini-3-flash-preview, two runs with majority vote) returns one holistic verdict per visualization — correct, partial, wrong, or not-detected — with a reason. The prompt enforces semantic meaning: trend arrows, value-to-label association, structure and hierarchy, multi-series matching, and signs. Deterministic RMS-style scoring and blind transcription were both tried first; each either leaked or over-rejected on heterogeneous outputs, so the holistic verdict was adopted as the reliable signal.

Formula

verdict ∈ { correct, partial, wrong, not-detected } (per visualization)
points: correct = 1.0 , partial = 0.5 , wrong = 0 , not-detected = 0
score   = ( 1.0 · #correct + 0.5 · #partial ) / 18

Results

Table 10. Chart-to-data verdicts and resulting scores. gemini-2.5-flash-lite is unscored — its extraction exceeds the judge's ~1M-token input limit (re-run with truncation pending).

Observations

• Strong VLMs read charts: gemini-3-flash-preview 0.944 (17/18), claude-opus-4.8 and gpt-5.5 both 0.889 (16/18); the Gemini-3 family, kimi, qwen, claude-sonnet46 and gpt-5.4-mini all 0.75–0.83.
• Text extractors land mid-table (0.61–0.69): they capture tabular chart data but lose pies and trend arrows — many "partial" verdicts mean "values right, but ↑/↓ dropped."
• Figure-as-image tools bottom out: docling 0.167, marker/mineru/mistral-ocr 0.222, and paddleocr-vl / got-ocr (0.08–0.14) — charts are emitted as image references or dropped, so chart data is absent (high "not detected").
• Open OCR/VLM parsers vary: olmocr-2-7b reads charts best of them (0.75), the rest capture little chart data.
• textract is distinctive (2/13/0/3): almost all partial, zero wrong — it captures data incompletely but never fabricates.

5. Cost and latency

Extraction quality is only half of the deployment decision. Because every document Needl ingests must pass through this step, the cost and latency of each library matter as much as its accuracy, and they vary by several orders of magnitude. This section reports the measured per-document cost and wall-clock latency for each library on the same 17-page benchmark document, then combines them with the quality results to identify the value frontier. The figures come from a single benchmarking run; cloud costs are approximate and reflect provider pricing at run time.

Two caveats apply to the raw numbers. First, latency here is single-stream wall-clock — every model processes the 17 pages strictly one at a time (concurrency 1), so the figures are directly comparable across cloud APIs, self-hosted OCR on a single GPU, and local CPU tools. Run at production concurrency (5–10 parallel pages) the cloud APIs are several times faster. Second, the local CPU tools were run on an Apple M2 and incur zero marginal API cost, but that is not free in the operational sense — it consumes compute the organisation still pays for, and marker in particular is dominated by a one-time model download on first run.

5.1 Measured cost and latency

Table 11. Per-document cost and latency on the 17-page benchmark. Cost is approximate and provider-priced at run time; latency is single-stream wall-clock — pages processed one at a time (concurrency 1) — so it is comparable across cloud APIs, single-GPU OCR and CPU tools. Local CPU tools incur no API cost; the self-hosted OCR/VLM models are priced as GPU-rental time (extraction seconds × the HF Inference Endpoint hourly rate for the serving GPU).

Cost

• Four cost tiers. Free local CPU tools ($0.00); a broad low-cost band ($0.003–$0.11) holding the Nova, Llama, Gemma, gemini-3.1-flash-lite, kimi and qwen cloud models plus the cheaper self-hosted OCR (glm-ocr $0.04, paddleocr-vl/got-ocr ~$0.11); a mid band ($0.09–$0.60) with the gemini-3-flash-preview pair, gpt-5.4-mini, claude-sonnet46, textract, gemini-2.5-flash and the pricier self-hosted GPUs (olmocr/rolmocr ~$0.26, internvl2-8b $0.51, internvl2-26b $0.56); and a top tier with pixtral-12b ($0.78), Claude Opus 4.8 ($1.19) and GPT-5.5 ($1.78).
• gemini-2.5-flash is a cost outlier among the general-purpose cloud VLMs ($0.601) — nearly 7× the gemini-3-flash-preview pair — while scoring lower on quality, making it hard to justify.
• textract is expensive for its tier ($0.26) despite being a managed OCR service rather than a generative model, and delivers only mid-table quality.
• API tokens versus GPU-rental time: claude-opus-4.8 costs ~$1.19 per document and gpt-5.5 ~$1.78 on token-billed APIs, while the self-hosted OCR/VLM models are priced as GPU rental — extraction time × the AWS us-east-1 hourly rate for the serving GPU (T4 $0.50, L4 $0.80, L40S $1.80, A100 $2.50, H200 $5.00). These range from ~$0.04 (glm-ocr, 164s on an L4) to ~$0.78 (pixtral-12b, 1123s on an A100), with the big-GPU or slow models costing most (internvl2-26b ~$0.56 on an H200, internvl2-8b ~$0.51 on an L40S).

Latency

• Latency spans three orders of magnitude — from ~2s (markitdown, liteparse) to ~1831s (marker). Measured single-stream, most cloud VLMs and self-hosted OCR fall in a 135–560s band, while the gemini-2.5 pair and the largest OCR models run 1000–1815s.
• Fast and good is achievable: among the high-quality models, gemini-3.1-flash-lite is the quickest single-stream at 221s — about 2× faster than gemini-3-flash-preview (484s) and well ahead of claude-opus-4.8 (909s) — while staying in the top quality band. gpt-5.4-mini is the fastest cloud VLM overall (157s), though only mid quality.
• Slow does not imply good: the gemini-2.5 pair (flash-lite 1815s, flash 1343s) is slow without commensurate quality, and marker's 1831s buys only mid-table accuracy; among the self-hosted models pixtral-12b (1123s) and internvl2-8b (1011s) are both slow and weak.
• Heavier models cost time: claude-opus-4.8 runs in ~909s and gpt-5.5 in ~840s per document single-stream — the slowest of the strong models; the self-hosted OCR/VLM models on single GPUs range from 164s (glm-ocr on an L4) to 1123s (pixtral-12b on an A100). Run at production concurrency (5–10 parallel pages) the cloud APIs are several times faster.

5.2 The value frontier: quality versus cost

The decisive view overlays quality on cost. For this analysis a single composite quality score is computed per library as the unweighted mean of its five applicable headline metrics — text fidelity, coverage, the hallucination judge score, the mean of the three table sub-metrics, and chart understanding. This composite is a convenience summary for the trade-off plot, not the production leaderboard score (which will apply the document-type workload weights of Section 2.4).

composite(lib) = mean( fidelity, coverage, halluc_judge, mean(table_3), chart )
                 # unweighted summary for the cost trade-off only

Table 12. Libraries ranked by composite quality, with cost and latency alongside. The composite is the unweighted mean of the five applicable headline metrics (a trade-off summary, not the workload-weighted leaderboard).

Reading the frontier

• Best value overall: gemini-3.1-flash-lite (and its thinking variant) sits at composite ~0.78–0.84 for only $0.043 — roughly half the cost of gemini-3-flash-preview ($0.089) and a sixth of claude-sonnet46 ($0.28) — while returning in the fastest single-stream time of any high-quality model (221s). It is the standout cost/quality pick.
• Highest quality, premium price: gemini-3-flash-preview (± thinking) sits at the very top of the composite (~0.89, just behind claude-opus-4.8) for $0.089–$0.093. The thinking variant costs marginally more and runs ~60% longer for a slightly lower composite — the non-thinking version is the better default at the top end.
• Strong mid-cost alternatives: kimi-k2.5 ($0.055) and qwen3-vl-235b ($0.047) deliver composite ~0.82–0.83, competitive with the leaders at lower cost.
• Hard to justify: claude-sonnet46 is the most expensive of the high-quality models ($0.28) for composite 0.79; gemini-2.5-flash is the priciest of the general-purpose cloud VLMs ($0.60) for only 0.70; textract is costly ($0.26) for 0.62.
• Free tools are not free quality: the zero-cost CPU parsers cluster at composite 0.43–0.59 — acceptable only where structure and charts are irrelevant and raw text recall suffices.
• Top of the frontier: claude-opus-4.8 posts the highest composite quality of any library (~0.90) at a modest $1.19, placing it at the top of the value frontier; gpt-5.5 sits mid-frontier (~0.79 at $1.78), pricier than several higher-scoring models. The self-hosted OCR/VLM models deliver weak value — their GPU-rental cost ($0.04–$0.78) buys unremarkable quality-per-dollar (olmocr-2-7b best at ~0.77 for ~$0.26; glm-ocr cheapest at ~$0.04 for ~0.66), so none reach the frontier.

Quality versus latency

6. Cross-criterion synthesis

The individual criteria are most useful read together. Two libraries with identical text-fidelity scores can differ sharply in table or chart handling; the multi-criterion view exposes those differences and supports the eventual combined decision the team needs to make.

6.1 Multi-criterion profile of the leading models

The radar profile below plots the six leading libraries — including claude-opus-4.8, which posts the highest composite of any library — across five axes (text fidelity, coverage, hallucination-cleanliness, mean table score, and chart understanding). Their profiles are large and well-rounded — they do not trade one capability for another — which is precisely why this group leads the leaderboard regardless of how the criteria are weighted.

6.2 Performance by library family

Grouping the 42 libraries into families clarifies the structural story. The cloud VLM providers (Gemini, OpenAI, Anthropic and other VLMs) are strong on all four headline metrics; the self-hosted OCR/VLM parsers form a broad mid-to-low tier; and the CPU text tools retain content (coverage) but collapse on tables and lag on fidelity, while the weakest models sit near the floor on tables and charts.

6.3 The three structural findings

• A compact group of high-quality vision models leads on every criterion. gemini-3-flash-preview (± thinking), kimi-k2.5, qwen3-vl-235b, claude-sonnet46, claude-opus-4.8 and the gemini-3.1-flash-lite pair are at or near the top of fidelity, coverage, hallucination, tables and charts simultaneously — with claude-opus-4.8 posting the highest composite of all (0.88).
• CPU text extractors capture words but lose structure. tika, pymupdf4llm and liteparse post high coverage (≈0.88–0.93) yet low fidelity, ~0 table scores, and only mid chart scores — they dump text without grids or chart semantics.
• The weakest tools restructure heavily. gemma3, mineru, mistral-ocr, nemotron and nova-lite drop 40–53% of content, hallucinate or misplace the most, and (docling/marker/mineru) leave chart data trapped inside image references.
• A distinct self-hosted OCR/VLM tier sits between the leaders and the CPU parsers. olmocr-2-7b approaches the mid-tier (composite ~0.67); glm-ocr, rolmocr, qwen2.5-vl-7b and paddleocr-vl land 0.47–0.59; and got-ocr, pixtral-12b and internvl2 collapse. GPT-5.5 is the cautionary case — strong fidelity but heavy financial-table hallucination pulls its composite to ~0.72.

6.4 Composite quality — full library ranking

Composite quality across all benchmarked libraries. Weights: hallucination 0.25, tables 0.25, coverage 0.20, fidelity 0.15, visualization 0.15. 38 of 42 models are scorable on all five criteria (claude-sonnet46, gemini-2.5-flash-lite, internvl2-8b and liteparse are excluded — incomplete criteria).

7. Reliability, audits, and known limitations

A benchmark is only as trustworthy as the scrutiny applied to its ground truth and its judge. Three audits and several known limitations are documented here so that the results can be interpreted with appropriate caution and reproduced faithfully.

7.1 Golden typo correction

The golden markdown misspelt a plant location (Singhtarai instead of the correct Singhitarai, per the source PDF). Because the evaluation treats the extraction as OCR output — any text not present in the golden is penalised as invented — libraries that spelled the location correctly were wrongly flagged for hallucination. Those false-positive findings were removed and scores recomputed. Net effect: gemini-3-flash-preview (0.68→0.68, 7→6 findings), marker (0.43→0.44, 17→16, a high finding removed), gemini-3.5-flash-thinking (0.47→0.48, 15→14), docling (0.40→0.40, 18→17). gpt-5.4-nano and qwen3-vl-235b kept their findings, as their spelling was genuinely garbled.

7.2 Chart-reading false positives

An audit of all 522 hallucination findings checked whether the golden-grounded judge had wrongly flagged faithful chart and figure readings as hallucinations. It found exactly one library affected — gemini-3-flash-preview-thinking — with two such findings: a correct Y-axis scale reading (−1 to 9 for the GDP chart, whose data spans −0.3 to 7.8) and a page-number addition (10). Both were removed; its score rose 0.71→0.72 (7→5 findings). No other library showed the pattern — every other absent-in-golden finding was a genuinely invented or wrong value, not a faithful visual reading.

7.3 Independent reviewer findings

During review, a spot-check of the hallucination set surfaced two findings in which the extracted text was actually correct and the golden markdown was wrong — a reminder that the golden, while carefully verified, is not infallible. No score action was required, but the cases are recorded. Reviewers also confirmed two methodological points: coverage is computed over the entire document's bag of words, and table detection is a binary per-table decision, with broken internal rendering of a detected table penalised by the structure and cell-accuracy sub-criteria rather than by detection.

7.4 Known limitations and pending work

• Judge coverage gap. gemini-2.5-flash-lite is unscored on hallucination and visualizations because its extraction exceeds the judge's ~1M-token input limit; internvl2-8b is likewise unscored on the LLM-judge criteria, as its judge runs did not complete. A re-run is pending for both.
• Single-document scores. All numbers here are for one representative document; the bucketed multi-document corpus (Section 2.4) is the basis for the production leaderboard.
• Same-provider judge bias. The primary judge is gemini-3-flash-preview, which is also among the libraries under test. A subset of the libraries were scored with a two-model cross-check (gemini-3-flash-preview and Claude Opus 4.8, Section 4.3); extending that cross-check across all libraries is planned to detect any same-provider favouritism.
• Imperfect golden. The golden is human-verified but not infallible (Sections 7.1, 7.3); audits and recomputation mitigate but do not eliminate this.

7.5 What changed from the original criteria set

• Reading order was removed as a standalone criterion and folded into hallucination (meaning-changing misplacement).
• Hallucination and misplacement were combined into a single criterion.
• KIE was dropped (Section 1.4); its intent is preserved by cell accuracy plus high-severity hallucination.
• Chart-to-data was newly built as the visualization criterion (Section 4.5).

8. Conclusions and recommendations

This benchmark provides a rigorous, reproducible basis for selecting an extraction library for Needl's financial-document workload. The methodology decomposes "extraction quality" into seven measurable criteria, grounds each in established academic practice, and computes them against a human-verified golden using deterministic code wherever possible and a golden-grounded, multi-run LLM judge only where meaning must be perceived.

The empirical conclusion is unambiguous and stable across criteria: the gemini-3-flash-preview family, kimi-k2.5, qwen3-vl-235b, claude-sonnet46 and claude-opus-4.8 are the strongest extractors on this workload, leading on fidelity, coverage, hallucination, tables and charts at once. For deployments where cost or latency favours a lighter model, the gemini-3.1-flash-lite pair offers the best balance, trailing the leaders only modestly. CPU text tools remain viable only where structure is irrelevant and raw text recall is all that matters; they should not be used where tables or charts carry the data. Claude Opus 4.8 is the single highest-quality library overall (composite 0.88) — narrowly ahead of the gemini-3-flash-preview pair and the cleanest on hallucination of any library; GPT-5.5, by contrast, pairs top-tier fidelity with systematic financial-table value corruption (25 findings), a cautionary profile for numeric documents.

Bringing cost into the picture (Section 5) sharpens the recommendation rather than changing it. gemini-3.1-flash-lite emerges as the clear value pick — composite quality ~0.79–0.85 at $0.043 per document and the fastest turnaround of any high-quality model (221s single-stream), roughly half the cost of gemini-3-flash-preview and a sixth of claude-sonnet46. Where maximum quality justifies the premium, the non-thinking gemini-3-flash-preview is the top performer among the cost-effective cloud models at $0.089 (Claude Opus 4.8 edges it on raw composite quality, but at ~14× the cost). The premium options that are hardest to justify are gemini-2.5-flash ($0.60 for mid quality) and claude-sonnet46 ($0.28), which a cost-aware deployment can safely pass over.

8.1 Recommended next steps

1. Finalise the bucketed multi-document corpus and workload weights against production ingestion data, then produce the weighted leaderboard.
2. Close the judge-coverage gap by re-running gemini-2.5-flash-lite with input truncation.
3. Add the planned OpenAI cross-check judge to quantify and correct any same-provider bias.
4. Define the combined decision score (criterion weights per document type) that converts these per-criterion results into a single deployment recommendation.

References

1. Ouyang, L. et al. OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations. arXiv:2412.07626.
2. SCORE-Bench: A Benchmark for Structured Content Extraction (Unstructured). arXiv:2509.19345.
3. ParseBench: Document Parsing Evaluation (LlamaIndex). arXiv:2604.08538.
4. Poznanski, J. et al. olmOCR-Bench: Deterministic Unit Tests for OCR and Document Conversion (Allen AI). arXiv:2510.19817.
5. Mathew, M., Karatzas, D., Jawahar, C.V. DocVQA: A Dataset for VQA on Document Images. arXiv:2007.00398.
6. OCRBench v2: Evaluating Large Multimodal Models on Visual Text Localization and Reasoning. arXiv:2501.06916.
7. Zhong, X., ShafieiBavani, E., Jimeno Yepes, A. Image-based Table Recognition (PubTabNet / TEDS). arXiv:1911.10683.