Context Infrastructure

1. The problem — the context bottleneck in agentic enterprises

Reasoning gets the public attention. Context — whether the model is working off the right evidence in the first place — gets discussed less. It is the layer that decides whether the work is correct.

Enterprises run on information that is fragmented by design. The latest 10-K lives on EDGAR. Internal due-diligence notes from past deals sit in SharePoint. The firm's house view on credit cycle positioning is buried in a research email thread. The legal team's prior commentary on covenant language is in a redlined PDF. The most recent risk update is a thread in Slack. Every analyst already knows this geography. What is new is that AI systems are now being asked to operate against the same fragmented surface, autonomously, at speed.

Agentic systems are language-model-driven workflows that take a goal, decompose it into a multi-step plan, and execute each step with whatever context they can retrieve. They have two distinct ways to fail.

• Reasoning failure. The model plans incorrectly, picks the wrong tool, runs a step out of order, or hallucinates a conclusion. This is the failure mode public benchmarks track, and it is improving fast through longer reasoning chains and better post-training.
• Context failure. The model receives input that is incomplete, irrelevant, or wrong. It then reasons precisely about the wrong evidence and produces an answer that is fluent, well-cited, and incorrect. This failure mode is quieter, harder to spot, and largely unaddressed at the architecture layer.

In multi-step workflows the two compound, and the math gets sharper than it looks. A single retrieval step that is correct 95% of the time looks reliable in isolation. Across five sequential retrieval steps, the probability that all five returned the right context is 0.95⁵, about 77%. Across ten steps it falls to 60%. For credit memos, deal screens, and regulatory filings, an error budget of one in four does not work.

The gap is not a function of the language model used. It is a function of what gets surfaced before the model reasons. The rest of this paper explains how that gap is engineered, layer by layer.

2. Current architectures — why common patterns fall short

Two architectural patterns dominate the current discussion of enterprise retrieval. Both have visible limits that surface only at scale, on real corpora, under real query distributions.

LLMs wired to MCP servers, one per source

The shortcut. Expose each enterprise SaaS application through an MCP server, hand the language model a directory of those servers, and let the model plan its way to the right answer. The pattern is appealing because it needs minimal infrastructure, fits the LLM's tool-use idiom, and feels composable. It does not work at scale, for two reasons.

Heterogeneous retrieval is lowest-common-denominator retrieval. Different vendors expose fundamentally different search capabilities. Slack search ranks recent, conversational matches against terse phrases. SharePoint search is built for keyword-and-metadata lookups across formal documents. Salesforce search is built for record retrieval keyed on object types and field values. None of these were designed to feed an analytical LLM. Each has its own scoring quirks, its own definition of relevance, and its own ceiling on result quality.

When a query crosses three or four such systems, the answer that reaches the language model is whatever each underlying search engine happened to return, glued together. It is weighted by nothing, normalized against nothing, and reconciled only by whatever heuristics the orchestration layer imposes after the fact. The result is a system whose accuracy is bounded by the weakest indexer in the stack, with no operator knob to improve it.

A uniform indexing pipeline and a single, consistent scoring layer are not arbitrary preferences. They are the only architecture in which retrieval quality is bounded by the system you control, not by the weakest third-party search engine in the customer's stack.

Vector-only retrieval

The other dominant pattern is vector RAG. Chunk every document at ingest time, embed each chunk into a high-dimensional vector space, and retrieve at query time by cosine similarity. The pattern has two structural problems that show up as corpus size grows.

A hard accuracy ceiling at scale. A 2025 DeepMind paper, Theoretical Limitations of Embedding-Based Retrieval (Weller et al.), establishes a fundamental ceiling on the corpus size over which embedding-based retrieval can stay reliable. Past that ceiling, the dimensionality required to distinguish documents grows faster than fixed-size embeddings can hold, and accuracy degrades regardless of the model. The same paper shows that BM25-style lexical retrieval keeps scaling through that ceiling without degrading. In practical terms: vector RAG can be made to work on a small, curated knowledge base. It cannot be made to work, with the accuracy a credit committee requires, on a hundred-terabyte enterprise corpus.

Vector retrieval is opaque by design. When a vector-RAG result is wrong, the failure is hard to diagnose. Cosine similarity scores in a high-dimensional space are not human-readable. There is no scoring trace to read, no term-level signal to tune, and no way to bridge "this chunk matched" with "here is why it matched". For a system whose first commitment is auditability, that is disqualifying. Engineers cannot fix what they cannot inspect, and auditors cannot trust what cannot be reconstructed.

Bigger context windows do not solve retrieval

A frequent counter-argument: as context windows grow into the millions of tokens, the retrieval problem disappears. Just give the model everything and let it figure out what matters. It does not work. Context window growth helps with synthesis, not retrieval. A typical enterprise has hundreds of terabytes of unstructured content, and no current or foreseeable context window holds that volume of input. Even if it did, putting the model in charge of locating the relevant fragments inside an enormous undifferentiated input is a strictly harder version of the problem retrieval is built to solve, with worse latency, higher cost, and lower accuracy.

3. System architecture — the Needl context engine

Needl is organized in three layers, each handling a distinct concern. The indexing plane produces a uniform substrate from heterogeneous sources. The query-time engine retrieves and ranks against that substrate. The agentic routing layer decomposes mixed queries and verifies completeness end to end. A permission layer mediates every result.

1. Indexing plane. The ingestion layer crawls, parses, normalizes, and indexes content from heterogeneous data sources under explicit freshness, fairness, and observability SLAs. Unstructured content goes into a lexical index and an object store with full provenance. Structured systems contribute only their schema and metadata, never their rows. The indexing plane runs continuously and independently of any query.

2. Query-time retrieval and ranking. The engine that takes a natural-language query and returns ranked, explainable results. Lexical-first hybrid retrieval over the unstructured index. Runtime SQL generation against structured sources. A query understanding layer that adds semantic awareness without depending on per-customer behavioral data.

3. Agentic routing. The orchestration layer decomposes mixed queries into structured and unstructured sub-queries, dispatches each to the right sub-agent under an inner ReAct loop, and verifies completeness through an outer ReAct loop before returning to the caller.

Permission layer. Every result, structured or unstructured, is mediated by a permission layer that enforces the access controls of the source system at runtime. There is no precomputed denormalization of permissions in the index. ACL state is authoritative at the source, and Needl joins against it on every query.

4. Unstructured retrieval — lexical-first hybrid retrieval

The first design decision in Needl's retrieval engine is to not use embedding-based retrieval as the primary mechanism. Lexical retrieval is inspectable, scales without an accuracy ceiling, and stays stable across a corpus that grows by tens of thousands of documents per week. Each of those properties matters for what Needl does.

The decision rests on three observations: the information-theoretic ceiling on embedding retrieval at scale (Section 2), the opacity of vector similarity for engineering and audit purposes (Section 2), and the absence of any stable supervisory signal in enterprise environments to train a domain-specific embedding. Needl is built on a lexical index. Semantics are added at query time through a separate query understanding layer, described later in this section.

Stable scoring across changing corpora

A naive lexical retriever uses BM25, which scores each document based on term frequency, inverse document frequency, and document length. In a static corpus this works fine. In an enterprise corpus, where new documents arrive every hour and the term-frequency statistics shift continuously, BM25 ranking drifts. A query that returned the right document in March returns a different document in September. Nobody changed the query. The corpus statistics changed beneath it.

Needl's scoring layer sits on a lexical index but uses proprietary scoring mechanisms that do not key off corpus-level statistics like term frequency, document frequency, or document length. Where these signals are used at all, they enter as weak inputs rather than primary ranking factors. The resulting ranker stays stable as the corpus grows, needs no per-customer retuning as content arrives, and behaves consistently over months and years of operation. On top of this stable substrate we layer firm-specific business logic, which makes the ranking personalized to each enterprise's vocabulary, document hierarchy, and authority signals.

Dynamic chunking, at query time, not at index time

A second core decision: documents are not chunked at indexing time. Chunking at ingest is universal in vector RAG systems, and it is one of the biggest reasons accuracy drops in those systems. The reason is straightforward. The right span of context for a question depends on what the question is, and the question does not exist when the chunk boundaries get drawn.

The mechanism is direct. Documents are stored intact in an object store and indexed whole into the lexical index, with positional information preserved. At query time, the scoring layer locates high-relevance documents, and a dynamic chunking module assembles a span from each high-scoring document that contains the answer with the surrounding context it needs. The span is shaped by the query, not by an arbitrary token budget chosen months earlier. Tables come back with their headers. Management commentary comes back with the data it references. Section breaks fall where the document put them, not where a chunker cut.

Query understanding without behavioral data

Consumer search systems like Google rely on enormous query-and-click datasets to train query understanding. When users search for "iphone charger" and click on results matching "lightning cable", the system learns the synonym. Inside an enterprise, that signal is unavailable. Query volumes are too low, terminology is firm-specific, and the same intent is expressed in dozens of different ways across teams. What the credit team calls a "watchlist note" the research team calls a "credit memo update" and the compliance team calls a "covenant flag".

Needl substitutes a different approach. Two inputs feed the query understanding layer.

• Pretrained world knowledge from frontier language models. Frontier models already encode substantial general and domain knowledge. They resolve "credit memo" to its synonyms, recognize "MD&A" as a section of a 10-K, and disambiguate "the Acme deal" against the right ticker, without any per-customer training.
• Customer-specific ontology where one exists. For firms that maintain a controlled vocabulary (internal product codes, deal codenames like "Project Phoenix" that map to specific borrowers known only to the M&A team, sector taxonomies), Needl ingests the ontology and uses it to resolve queries that would otherwise be too firm-specific for a general model to pick up.

The two inputs work together at query time. A natural-language query is expanded into its semantic neighborhood, mapped to the firm's vocabulary where applicable, and then matched against the lexical index. The expansion is fully traceable. An engineer can read which synonyms were considered, which ontology nodes activated, and how each contributed to the final result set.

5. Structured retrieval — schemas, not rows

For structured data sources (relational databases, warehouses, transactional systems), Needl makes a different but related choice. The system does not ingest the underlying rows. It ingests only the schema and metadata.

User queries that target structured data are translated into SQL at query time and executed directly against the source system. The translation is performed by a sub-agent that has access to the schema, table descriptions, sample values, and relationships, but never to the data itself. This is a deliberate separation. The agent knows the shape of the world. The source database holds the world.

Why this is sufficient

A SQL query is a relational-algebra expression. It is computable, deterministic, and verifiable. When the right query is generated against the right schema, the answer it returns is exact — not an approximation produced by similarity matching, but the literal result of the expression as evaluated by the source database. Reindexing structured data into a separate retrieval layer adds no accuracy. It would, however, introduce three serious costs: data freshness lags whenever the source updates, duplicate storage of often-sensitive transactional data, and the operational burden of keeping two systems consistent. None of those costs buys anything in return.

SELECT origination_year, AVG(loan_amount / property_value) AS ltv
FROM loans JOIN properties USING (loan_id)
WHERE sector = 'auto_parts'
GROUP BY origination_year;

Permissions come for free

Because the SQL executes inside the source database with the requesting user's authenticated identity, the database's native access controls (row-level security, column masking, view-based permissions) apply automatically. Needl does not need to model these permissions separately, replicate them, or risk drift between its model of access and the source system's reality. The database is the source of truth for both data and permissions, and stays that way.

6. Agentic routing — routing, decomposition, verification

A real enterprise question rarely targets only structured data or only unstructured data. The interesting ones cross both. "Show me our top ten exposures in the auto-parts sector with the most recent risk commentary on each" needs a structured query (top exposures by sector) and an unstructured query (recent commentary per borrower), joined on a common key.

The agentic routing layer handles three things: classifying the incoming query, decomposing it into sub-queries that each target the right retrieval mode, and verifying that the combined result actually answers the question that was asked.

Classification and decomposition

When a natural-language query arrives, the router first determines whether it is purely structured, purely unstructured, or mixed. The classifier uses the query understanding layer's output (entities, intents, schema affinity) along with the firm's ontology to make this call. Mixed queries are decomposed into their structured and unstructured components, each formulated as a self-contained sub-query. Decomposition is not done greedily. The router uses the schema and ontology to identify a join key (a borrower identifier, a deal code, a date range) that will let the results be reconstructed coherently downstream.

Inner ReAct loops per sub-agent

Each sub-query is dispatched to its specialized agent, which runs an inner ReAct loop until it is satisfied that it has the information needed to answer its part of the question. ReAct here means the agent can issue a query, inspect the result, decide whether it is complete or needs refinement, and reissue with a refined sub-query if so. For the unstructured agent, refinement might mean expanding synonyms, broadening the recency window, or relaxing an author filter. For the structured agent, it might mean rewriting a SQL query that returned an empty result set because of a misnamed column, or joining an additional table to enrich the result.

Inner-loop refinement runs against the same retrieval layer, with the same scoring and the same permission enforcement. The agent's view of the data never violates the user's actual access rights, regardless of how many iterations it takes.

Outer ReAct verification

When each sub-agent reports completion, results return to an outer ReAct loop. The outer loop's job is to verify that the combined structured and unstructured answers, taken together, address the original natural-language question. If gaps remain (a borrower appears in the structured result but no commentary was retrieved for it, or a piece of commentary references a borrower not in the structured result), the outer loop dispatches follow-up sub-queries to close the gap. Only when the outer loop is satisfied that the full answer is assembled, with citations to source content for every claim, does the engine return to the caller.

The architectural payoff is that each sub-agent handles one well-defined retrieval mode and can be tuned, monitored, and improved independently. The outer loop owns answer-level invariants and is the natural place to enforce them. Errors are localized, traceable, and addressable. There is no single opaque agent making every decision.

7. Permission awareness — ACL enforcement, end to end

A retrieval system that returns a document the user was not entitled to see in the source system is broken. In regulated industries it is also a compliance event. Permission awareness in Needl is not a layer added on top of retrieval. It is built into the result-generation path itself.

Unstructured: runtime ACL join

For unstructured content, every retrieved candidate is joined at runtime against the role-based access controls of the application from which it originated. SharePoint files are joined against SharePoint's ACLs. Slack messages are joined against Slack's channel and DM membership. Google Drive against Drive's sharing model. And so on for every connected source.

This join is performed at query time, not at index time. There is no precomputed denormalization of permissions in the index, and there is no opportunity for the index to drift out of sync with the source system's ACL state. When an administrator revokes a user's access to a SharePoint folder, the next query from that user will not see content from that folder — with no pipeline rerun, ACL backfill, or eventual-consistency window.

Structured: identity passthrough

For structured sources, permission enforcement is simpler. The SQL generated by the structured agent is executed under the requesting user's authenticated identity. Row-level security, column masking, view-based access — every native security primitive of the underlying database applies automatically. The architectural advantage is that the database's security infrastructure does the work it was designed to do. Needl does not need to model, replicate, or shadow the database's security logic. There is one source of truth for both data and access, and the retrieval system inherits both correctly without extra effort.

Sub-document permissions

In some cases, access is more granular than the document level. A deal memo may be readable by an entire team, but the section discussing personnel decisions may be restricted to a smaller group. Needl supports sub-document access controls where the source system exposes them, with the same runtime-join semantics. The candidate span is checked against the user's effective access at the section level before being included in the response.

8. Explainability — inspectable retrieval, end to end

Because the unstructured retrieval engine operates over a lexical index and the structured engine operates through SQL, every retrieval decision is composed of primitives that a human engineer can read, trace, and modify. There is no high-dimensional similarity score to interpret, no embedding space to debug, and no opaque relevance model to reverse-engineer.

In practice, this shows up as three concrete capabilities.

• Scoring traces. For any retrieved result, the system can show the term-level signals that surfaced it: which tokens matched, what synonym expansions fired, which ontology nodes contributed, and how the proprietary scoring function combined them. Engineers can read this trace and tune behavior with the precision of code rather than the imprecision of model retraining.
• Citations on every claim. Every assertion in a generated answer is linked to the specific document and span it was derived from. End users see not just the answer but the evidence. Downstream reviewers can verify each claim independently.
• Audit reconstruction. For any historical query, the system can reconstruct what was retrieved, when, against what permission state, and how the answer was assembled. In regulated workflows, this is a baseline capability, not a premium feature.

Deduplication at search time, not index time

In a real enterprise, the same document is duplicated across drives, channels, inboxes, and shared folders. A naive system might dedupe at indexing time, discarding byte-identical copies and keeping only one canonical version. Needl does not. The reason is that provenance and access controls vary across copies even when the content is identical. A draft memo in a personal drive has different permissions from the same memo in a deal room. An attached file in an email thread has different reviewers from the same file in SharePoint. Discarding duplicates at indexing time discards legitimate provenance and risks surfacing the wrong copy under the wrong permissions.

Instead, Needl deduplicates at search time, using near-duplicate hashing to identify clusters of equivalent content. The user receives one canonical result with the most relevant provenance for their context. The system retains the full provenance graph beneath. If the user asks where else the document lives, the system can answer.

9. Indexing pipelines — from raw sources to ranked context

Most of the operational complexity in Needl lives in the indexing plane. Query-time retrieval is the visible half of the system. The indexing pipelines are the half that determines whether the visible half has anything useful to retrieve from.

Three SLAs

The indexing plane targets three explicit service-level requirements.

• Freshness. Most enterprises require new content to be searchable within 15 to 30 minutes of arrival. A retrieval system that lags the source by hours is not retrieving the current state of the firm. It is retrieving a snapshot.
• Burst tolerance. Information arrives in bursts. A deal closes and 400 documents land in the data room over an afternoon. A regulatory deadline triggers a flood of internal correspondence. No user's data should be starved when another user's burst saturates the system.
• Immediate utility. A new user has to find value within minutes of onboarding, not after a multi-day historical backfill completes. For a firm with five years of accumulated SharePoint history, the backfill alone can take days.

Stateful crawlers and per-user fairness

For each unstructured source, Needl runs stateful crawlers that maintain per-user state (a queue, a bookmark, an ACL snapshot, and a backfill cursor) so that crawling can be paused, resumed, and rate-limited per user without losing place. This is what makes burst tolerance possible. When one user's mailbox suddenly grows by 50,000 messages, the crawler for that user is scaled up independently of every other user, and the rest of the firm continues to see normal indexing latency.

The two-crawler pattern for new users

When a new user is onboarded, the system runs two crawlers in parallel against their content. The first, the recent crawler, pulls only the last sixty days of content and pushes it through the indexing pipeline as quickly as possible, so the user finds value within minutes of logging in. The second, the backfill crawler, walks back through historical content at a steady, fairness-bounded rate, so the user's full history becomes searchable over the following hours and days without ever blocking the recent crawler.

Object store, metadata queue, and complexity-tiered parsing

Crawled content is written to an object store with full provenance: which user's view it was crawled from, what ACL snapshot applied, what version was seen, and what checksum the content carried. Metadata describing each object is enqueued in a separate queue, decoupling crawling from processing so that either side can scale independently.

A fair scheduler picks up unprocessed objects and routes them to the appropriate parsing pipeline. The routing decision is based on the complexity of the document, not its file extension. A simple text-only PDF goes to a lightweight parser that extracts text in milliseconds. A PDF with embedded tables, multi-column layouts, charts, and footnotes goes to a heavier pipeline that uses vision models to reconstruct the layout, extract structured tables as data, and preserve the relationships between figures and the text that references them. A PDF that turns out to be a scan goes through OCR before either of the above. Routing on complexity rather than file type is what keeps indexing costs bounded at hundreds-of-terabytes scale. Most enterprise content is simple, and routing it through a vision pipeline because it happens to be a PDF is wasteful. The vision pipeline is reserved for content that actually needs it.

Observability throughout

Every stage of the pipeline emits telemetry: crawl latency per source per user, queue depth, parser tier distribution, parser failure rates, scheduler fairness metrics, and end-to-end indexing latency. Operators can answer questions like "why is User X seeing stale content?" or "which source is producing the most parser failures?" in real time, rather than reconstructing the answer from logs after a customer complaint.

10. Benchmarks — measured performance

Needl benchmarks against configurations designed to expose retrieval, not bypass it. The two results that matter most are the S&P Global Kensho Long-Document QA leaderboard and a full-corpus rerun of FinanceBench, both described below.

S&P Global Kensho Long-Document QA

On S&P Global's Kensho Long-Document QA benchmark, which evaluates retrieval and answer quality over long, complex financial documents, Needl ranks #1. The benchmark is operated independently by S&P Global, and the ranking is verifiable through their published leaderboard.

FinanceBench, full-corpus configuration

FinanceBench, the financial-domain QA benchmark, is conventionally run by pairing each question with the SEC filing that contains the answer. That configuration tests synthesis, but it bypasses retrieval entirely. Needl reruns the benchmark in a different configuration. Each question is posed against the entire SEC filings corpus, with no pairing. The system has to locate the relevant filing first, then answer. This is the configuration that mirrors how the system actually operates in production.

The 15-point gap is not a function of the language model used. Both systems can be run on the same frontier model. The gap is a function of the retrieval architecture — what gets surfaced before the model reasons. It widens, rather than narrows, as the corpus grows, because vector retrieval's information-theoretic ceiling tightens with corpus size while lexical retrieval does not.

Closing — context is the infrastructure

Test-time scaling will keep making reasoning models stronger. That progress is real and it matters. But it does not free the enterprise from the requirement to put the right context in front of the model in the first place. Reasoning over the wrong evidence produces fluent, confident, well-cited errors, and the wrongness is invisible to anyone who does not already know the right answer.

Building the context layer correctly is a discipline of its own. It needs accuracy that holds at scale, retrieval that respects every existing permission boundary without drift, explainability that lets engineers and auditors trace every result, and indexing pipelines that operate reliably under the freshness, fairness, and burst constraints of real organizations. None of those properties come from a vector database, an MCP wrapper around a SaaS search box, or a sufficiently large context window.

They come from an architecture built specifically for them: lexical-first hybrid retrieval, schema-only structured ingestion, runtime permission enforcement, dynamic query-time chunking, and a uniform scoring layer the operator controls end to end.

Needl.ai is that architecture, deployed in production, at hundreds of terabytes, inside the most demanding workflows of the enterprises that have already adopted it. It is the context infrastructure layer that turns enterprise documents and structured data into decision-ready intelligence, for humans and for the agents that will increasingly act on their behalf.