AskNeedl on OfficeQA Pro

Abstract

We report the performance of AskNeedl, Needl.ai's document question-answering agent, on OfficeQA Pro, a 133-question benchmark for grounded, multi-document reasoning over a corpus of U.S. Treasury Bulletins spanning roughly a century (89,000 pages; over 26 million numerical values), published by Databricks AI Research (arXiv:2603.08655). Evaluated on the full corpus in raw PDF format, AskNeedl achieved 64.66% correctness at the benchmark's default 0.0% allowable absolute relative error threshold, with an average latency of 2 minutes 22 seconds, 22 tool calls, and $0.98 in model cost per question.

Under the same corpus and document-format configuration, the published baselines for frontier agent frameworks are 48.12% (Claude Opus 4.8, Claude Agent SDK), 36.09% (GPT-5.4 High, Codex CLI), and 18.05% (Gemini 3.1 Pro Preview, Gemini CLI), while the subsequently reported Claude Fable 5 result is 57.90%. We present these results alongside the published figures and describe the methodological caveats that apply when comparing across evaluation runs.

1. Background

The OfficeQA Pro benchmark

OfficeQA Pro evaluates AI agents on end-to-end grounded reasoning: locating relevant material within a large, heterogeneous document corpus, extracting evidence from unstructured text and complex nested tables, and performing multi-step quantitative analysis to produce a single verifiable answer. The corpus consists of U.S. Treasury Bulletins published from 1939 onward, transitioning from scanned physical documents to digital-native PDFs in 1996. Questions may require information from one to over twenty pages; 11% require data from three or more bulletins, 22% involve web search for external values (e.g. historical CPI), and 62% require data analysis beyond basic arithmetic.

Evaluation is deterministic. Since 99% of answers are numerical, correctness is reported across allowable absolute relative error thresholds of 5.0%, 1.0%, 0.1%, and 0.0%, with 0.0% (exact match after normalization) as the default reporting threshold. In the benchmark paper's hardest published configuration — agents given the full corpus in raw PDF format — the strongest frontier agent (Claude Opus 4.8 in the Claude Agent SDK) reaches 48.12% correctness, and the three evaluated frontier agents average 34.1%.

2. Evaluation setup

AskNeedl is Needl.ai's custom agent for document question answering. It was run on the OfficeQA Pro question set (133 questions) against the full Treasury Bulletin corpus in raw PDF format — i.e. the same corpus and document-format configuration as the "Full Corpus / PDF" rows of Table 1 in the benchmark paper.

Unlike the baseline agents, which either rely on pre-parsed document representations (Databricks' ai_parse_document) or invoke external OCR and PDF-parsing libraries at query time, AskNeedl ingests the raw PDF corpus through its own built-in PDF parsing pipeline: each document is converted to full text, indexed, and made available for retrieval-augmented generation (RAG) at query time.

This means AskNeedl's results reflect end-to-end performance inclusive of its own parsing, retrieval, and reasoning stages, as shown in the diagram below. Scoring follows the benchmark's published evaluation protocol across the four error-tolerance thresholds.

3. Results

3.1 Correctness

Table 1 reports AskNeedl's raw results by error tolerance. Table 2 places the 0.0%-threshold and per-threshold results next to the published full-corpus PDF baselines and the subsequently reported Claude Fable 5 result, and Figure 2 plots the corresponding curves.

Table 1. AskNeedl results on OfficeQA Pro (full corpus, raw PDF format), by allowable absolute relative error threshold. N = 133 questions.

Table 2. Correctness (%) by error-tolerance threshold, full corpus, raw PDF format. Baseline values at the 0.0% threshold are from Table 1 of the benchmark paper; values at the 5.0%, 1.0%, and 0.1% thresholds are transcribed from Figure 6 (left panel, PDF series) of the same paper. † Claude Fable 5: the 0.0% value (57.90%) is as reported by Databricks (June 9, 2026); the 5.0%, 1.0%, and 0.1% values are estimated by applying the same threshold-to-threshold ratio observed for Claude Opus 4.8.

At the default 0.0% threshold, AskNeedl's 64.66% correctness is 6.8 percentage points above the reported Claude Fable 5 result (57.90%), 16.5 points above the strongest published baseline from the original benchmark paper (Claude Opus 4.8, 48.12%), 28.6 points above GPT-5.4 High (36.09%), and 46.6 points above Gemini 3.1 Pro Preview (18.05%). The ordering is preserved at all four thresholds.

Notably, while Fable 5's estimated curve runs above AskNeedl at the more relaxed 5.0% threshold, AskNeedl overtakes Fable 5 at the stricter thresholds that matter most for enterprise use cases requiring exact numerical answers. AskNeedl's correctness rises to 84.96% when a 5% relative error is tolerated; a substantial fraction of residual errors are near-misses (e.g. rounding, revision selection, or unit-level imprecision) rather than wholly incorrect retrievals.

3.2 Correctness with multiple interpretations

The benchmark paper additionally reports an oracle-retrieval configuration, in which agents are provided directly with the exact PDF page(s) required to answer each question, isolating downstream extraction and reasoning from corpus-scale retrieval. This also narrows the context given to the agents to only one interpretation for questions that could have multiple ways of answering them.

With AskNeedl, instead of supplying the oracle pages, we instead allowed the agent to output multiple answers, each for a different interpretation of the question. When the golden answer is one of the multiple answers output by the AskNeedl agent, we count it as a correct answer.

Table 3 reports the results by error tolerance; Table 4 and Figure 3 compare against the published oracle-PDF baselines.

Table 3. AskNeedl results on OfficeQA Pro with multiple interpretations, by allowable absolute relative error threshold. N = 133 questions.

Table 4. Correctness (%) by error-tolerance threshold. AskNeedl uses the multi-answer configuration; baselines use oracle pages in raw PDF format. Baseline values at the 0.0% threshold are from Table 1 of the benchmark paper; intermediate-threshold values are transcribed from Figure 6 (right panel, PDF series) of the same paper.

At the default 0.0% threshold, AskNeedl reaches 85.71% correctness with multiple answers, 24.8 percentage points above the strongest published oracle-PDF baseline (Claude Opus 4.8, 60.90%), 30.8 points above GPT-5.4 High (54.89%), and 46.6 points above Gemini 3.1 Pro Preview (39.10%).

AskNeedl's threshold curve is comparatively flat in this setting (a 6.8-point spread between the 0.0% and 5.0% thresholds, versus 20.3 points in the full-corpus setting), suggesting that most residual errors are categorical — wrong value, formula, or revision — rather than near-miss numerical imprecision.

3.3 Latency, tool usage, and cost

Table 5 and Figure 4 compare efficiency metrics in the full-corpus configuration. AskNeedl averaged 2 minutes 22 seconds per question, 22 tool calls, and $0.98 in model cost. The published baselines in the full-corpus PDF setting range from 13.1 to 31.2 minutes, 57.0 to 82.4 tool calls, and $1.79 to $6.21 per question.

We note that the published baseline latencies were obtained with a suite of PDF-parsing and OCR dependencies pre-installed; the benchmark authors report that latencies without pre-installation were substantially higher.

Table 5. Correctness at the 0.0% threshold and per-question efficiency metrics, full corpus, raw PDF format. Baseline rows are from Table 1 of the benchmark paper. AskNeedl figures are from our evaluation run; cost figures reflect provider pricing at the time of each respective run.

4. Limitations and methodological notes

5. Conclusion

On OfficeQA Pro's full-corpus, raw-PDF configuration, AskNeedl reaches 64.66% correctness at the benchmark's default exact-match threshold, exceeding the subsequently reported Claude Fable 5 result (57.90%) by 6.8 percentage points and the original benchmark baselines by a wider margin, while using fewer tool calls and lower per-question latency and cost than each published baseline. With multi-answer output, AskNeedl reaches 85.71%, again above each published oracle-PDF baseline.

Consistent with the benchmark authors' conclusion, substantial headroom remains: the 20.3-point spread between AskNeedl's 0.0% and 5.0% threshold results in the full-corpus setting indicates that answer precision — revision selection, rounding discipline, and unit handling — is a significant remaining error source alongside retrieval and parsing.

References

1. Databricks AI Research. OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning. arXiv:2603.08655, March 2026. github.com/databricks/officeqa
2. Databricks. Claude Fable 5 is now available on Databricks, fully governed through Unity AI Gateway. Databricks Blog, June 9, 2026.
3. Needl.ai. AskNeedl evaluation runs on OfficeQA Pro (full corpus and multi-answer, PDF format). Internal results, June–July 2026.