Every answer Needl.ai gives starts with reading. Before a document can be searched, quoted or cited, each of its pages has to become text. We moved that reading step from a pool of up to 14 GPU workers to hundreds of short-lived AWS Lambda workers, and a document of more than 300 pages now takes a median of 1 minute instead of 10.6 minutes.
PDFs, Office files and spreadsheets
Up to 14 GPU workers
Started when work arrived
10.6 min
500 Lambdas at a time
Started per file, stopped when done
1 min
Median time to extract a document of more than 300 pages
Part one: the plain-language version. Four short sections on what changed and why. Every technical term is underlined where it first appears; hover over it, or tap it on a phone, for a definition.
What document extraction is
Document extraction turns a file into text a computer can work with: every word, table and chart label on every page, in reading order. Search, citations, red-flag checks and memos all work from that text, so nothing else can start until extraction is done. When a team connects a data room or a shared drive, extraction is the first thing that happens to each file.
Older systems used OCROptical character recognition: software that recognises the letters in a picture of a page., software that recognises the letters in a picture of a page. Today, AI Vision modelAn AI model that takes an image of a page and returns its contents as text, tables and charts included. read a page much as a person does, tables, charts and multi-column layouts included. Our pipeline sends a picture of each page to one of these models through an APIA way for one program to send a request to another over the internet and get an answer back. and gets the page's text back. Which model reads best is a separate question, covered in our benchmark of 42 PDF extraction tools. This post is about running that reading across thousands of pages at once.
What changed
Our first pipeline ran OCR on GPUA graphics processor. It is the usual chip for running AI models and the most expensive part of a server to rent. servers: large machines with graphics chips, priced at $0.80 an hour and started when work arrived. We ran up to 14 of them. A long document was spread across several of them, but at most 14 pieces could be read at once, and every other file waited in line. Small files waited a median of 4.5 minutes before any work on them started.
Vision models changed the shape of the work. When a model reads the page, our worker sends the page and then waits for the answer, 2.9 seconds at the median. Waiting needs very little computing power, so a few large machines are the wrong tool for it. Many small, cheap workers that wait side by side are the right one.
Watch one 300-page document on both pipelines
Median time to searchable text | each block is one chunk
AWS LambdaAmazon's serverless service. It starts a copy of a function for each task, runs many copies at once and bills per millisecond. provides that. It starts a copy of our code for each piece of work, up to 500 at a time in this pipeline, and stops each copy when its piece is done. We pay for the milliseconds the code runs and nothing when no files arrive.
That matters because files arrive in bursts. Someone connects a shared drive, hundreds of files land at once, and then nothing happens for hours. A pool capped at 14 workers is too small for the burst, so files queue behind it.
Files arrive in bursts
Illustrative day | the same arrivals, two responses
MorningMiddayEvening
With a pool capped at 14 workers, work under way is held flat at the cap while the burst of arrivals waits in line, and the backlog is worked off over the following hours.
MorningMiddayEvening
With serverless workers, work under way follows the arrivals exactly and nothing runs between bursts.
Files waiting Work under way Cap of 14
Illustration of the arrival pattern, not measured data
How a document is read now
Every file goes through the same five steps, whether it has 3 pages or 800.
- A file arrives. Our ingestion service records a new job and puts one message on a QueueA managed to-do list of tasks. Workers take tasks from it, and a task that isn't finished goes back on the list.. It doesn't open the file.
- The file is split. A worker counts the pages and creates one task for every five pages, so a 300-page report becomes 60 tasks. Word and PowerPoint files are converted to PDF first. Spreadsheets are split by sheet.
- Every piece is read at the same time. Each task starts its own Lambda worker, which turns its five pages into images, sends them to the vision model and saves the text of each page.
- A checker watches the job. Every five seconds, a small program checks whether all of a document's pieces are done.
- The document becomes searchable. When the last piece is in, the pages are joined in order and added to the search index, where the document can be searched and cited.
Because the pieces run side by side, a document takes about as long as its slowest piece, and length matters much less than it used to. On Lambda, documents over 300 pages took only four times as long as documents under 20 pages.
What this means for your team
- Documents are ready sooner. A new file becomes searchable and citable soon after it lands in a data room, whether it has 3 pages or 800.
- One long file no longer holds up short ones. Every file is split and read in parallel, so a single 500-page agreement does not keep a two-page letter waiting.
- One pipeline for a mixed room. PDFs, Word and PowerPoint files and spreadsheets go through the same pipeline, so a data room is handled as one batch.
See it on a data room. The AI Due Diligence page shows what Needl.ai does with the text once it is extracted: a first pass, red flags and a cited memo.
See AI Due DiligenceTen terms to know
Most of part two builds on these ten. The other terms the post uses are defined in the glossary at the end.
- Document extraction
- Turning a file into plain text a computer can search, quote and analyse, page by page.
- Vision model
- An AI model that takes an image of a page and returns its contents as text, tables and charts included.
- GPU
- A graphics processor. It is the usual chip for running AI models and the most expensive part of a server to rent.
- Server
- A computer in a data centre that you rent, start and manage yourself.
- Serverless
- Running code without renting servers: the cloud provider starts machines when there is work and bills only for the time the code runs.
- AWS Lambda
- Amazon's serverless service. It starts a copy of a function for each task, runs many copies at once and bills per millisecond.
- Worker
- A program whose only job is to take a task, do it, save the result and take the next one.
- Queue
- A managed to-do list of tasks. Workers take tasks from it, and a task that isn't finished goes back on the list.
- Chunk
- A small slice of a document that one worker handles: five pages here, or one sheet of a spreadsheet.
- Concurrency
- How many workers are running at the same moment. Our peak was 507.
Part two: the engineering detail. From here on, the post follows a file through the pipeline, then covers what we learned, what we measured and where this design stops being the right one.
Where the GPU setup hit its ceiling
In an earlier write-up, we described scaling OCR for long PDFs with Ray on Kubernetes. A 463-page document that took 38 minutes on one GPU finished in about 10 minutes across 10 GPUs.
From 38 minutes to about one
Time to extract one long document, by pipeline generation | earlier generations | today
Source: our GPU write-up and our measurements. Different documents, so read the trend, not the exact ratio
That was a large win at the time. A year of running it showed three limits.
- Waiting for a GPU. Small files waited a median of 4.5 minutes for a GPU before any work started.
- A fixed ceiling. Parallel work was capped by the number of GPUs: 14 workers.
- Rising cost per page. Each page used 3.5 seconds of GPU time, and 10.8 seconds after an OCR model upgrade.
Bursty arrivals made all three worse: a pool capped at 14 workers was too small at the peak (Figure 4).
With a vision model doing the reading, a worker's job comes down to four steps: render a page as an image, send it to the model, wait for the answer and save the text. The wait is most of the time, 2.9 seconds at the MedianThe middle value: half the documents finished faster than this and half slower. and 7.7 seconds at the 90th percentile (p90)90% of documents finished within this time. It shows the slow tail that a median hides., and it needs almost no processor, memory or GPU. A GPU server waiting on replies from the network is an expensive way to wait.
Most of a worker's time is spent waiting
One page, one worker | processor and memory are almost idle during the wait
Only the wait is measured, and the other widths are illustrative
So the new design had four requirements: many small workers that can wait in parallel, workers that appear when files arrive and go away when they finish, parallel work inside a single document as well as across documents, and one pipeline for PDFs, Office files and spreadsheets.
Why Lambda and SQS fit the work
Lambda starts one worker per waiting message. It peaked at 507 workers running at once, with no ThrottlingA service limiting how many requests it accepts and asking callers to slow down., no servers to manage, no autoscaler to tune and no GPU quota to request. Billing is per millisecond, so a quiet hour costs nothing.
Amazon SQSAmazon's queue service. Each Lambda function in this pipeline reads from its own SQS queue., the queue in front of each function, takes care of the parts that are easy to get wrong. RetryRunning a failed task again, usually by putting its message back on the queue., Visibility timeoutHow long a message stays hidden from other workers while one works on it. Unfinished work reappears after it. and Dead-letter queueWhere a message goes after failing too many times, so a person can look at it. come with the service. Each function also gets its own memory, TimeoutThe longest a task may run before it is stopped. and share of the account's capacity.
How the account's 1,000 workers are shared
Reserved concurrency per function, split by queue | the rest stays free for everything else in the account
| Function | Queue | Memory | Timeout | Workers |
|---|---|---|---|---|
| ExtractionSplit queue | Split queue | 2 GB | 5 min | 100 |
| ExtractionChunk queue | Chunk queue | 2 GB | 5 min | 500 |
| Office conversionConversion queue | Conversion queue | 3 GB | 15 min | 100 |
| SpreadsheetsSpreadsheet queue | Spreadsheet queue | 4 GB | 15 min | 100 |
| Everything else in the account | 200 |
The platform was only half the answer. How the work is split mattered more than where it runs.
Every file becomes small, independent pieces
Whatever the file type, it is broken into units that one worker can finish in seconds. PDFs become five-page ChunkA small slice of a document that one worker handles: five pages here, or one sheet of a spreadsheet.. Word and PowerPoint files are converted to PDF first and then split the same way. Excel workbooks become one unit per sheet, with no model involved.
A 300-page report becomes 60 chunks running at the same time, and a three-sheet workbook becomes three. How long a document takes now depends on its slowest chunk.
Chunk size sets the ceiling
The first version used 25-page chunks. Fewer messages and less overhead seemed sensible. But a test batch of 1,618 pages produced only 73 chunks, so it could never use more than 73 of the 500 workers available. With five-page chunks, a larger batch kept all 500 busy. Chunk size decides how much parallel work is possible at all.
Chunk size sets the ceiling
500 workers available | each square is one worker | busy workers in teal
73of 500 workers busyA 1,618-page test batch made only 73 chunks, so 427 workers had nothing to do.
500of 500 workers busyWith five pages per chunk, a larger batch gave every available worker something to read.
The pipeline end to end
Every document we process ends up in Elasticsearch, the Search indexElasticsearch, the engine that makes extracted text searchable. A document reaches it only when every page is ready. behind Needl.ai. The service that writes documents there, our indexer, normally extracts the text itself. For files that go through Lambda, it waits until every page is back. The numbered parts of the diagram are the sections that follow; pick a file type to trace its path.
One message in, many Lambdas out, one poller to finish
The extraction pipeline | pick a file type to trace its path
- 1
The ingestion service only plansCreates the job and sends one small message. It never opens the file.
- 2
The splitter fans outCounts the pages and sends one message per five pages.
- 3
Paying for each page onceWorkers save each page to S3 before marking the chunk done in Postgres.
- 4
Office files are converted firstLibreOffice turns Word and PowerPoint files into PDFs, then the same split step runs.
- 5
Spreadsheets skip the modelOne worker per sheet reads the cells directly.
- 6
One checker decides when a document is doneThe poller releases a job only when every chunk is done.
- 1Plan
Ingestion serviceCreates the job and marks the document as extracting in one transaction, then sends one message carrying ids only.
Ingestion serviceCreates the job and marks the document as extracting in one transaction, then sends one message carrying ids only.
Ingestion serviceCreates the job and marks the document as extracting in one transaction, then sends one message carrying ids only.
- 2Prepare
Nothing to prepareA PDF goes straight to the splitter.
Conversion LambdaConverts the file to PDF with LibreOffice and saves the PDF to S3. 3 GB of memory, 15-minute timeout.
Spreadsheet LambdaOpens the workbook once, detects the real format from the file's contents and lists the sheets in 0.7 seconds.
- 3Split
Splitter LambdaCounts the pages, writes one row per five-page chunk and sends one message per chunk, in batches.
Conversion LambdaRuns the same split step in the same run: one message per five-page chunk.
Spreadsheet LambdaRecords one unit per sheet, queues sheets 2 onwards as separate messages and keeps sheet 1 for itself.
- 4Read
Extraction LambdasUp to 500 at once. Each renders its five pages, sends them to the vision model and falls back to a second model if needed.
Extraction LambdasUp to 500 at once. Each renders its five pages, sends them to the vision model and falls back to a second model if needed.
Spreadsheet LambdasOne per sheet. Each reads the cell values directly with python-calamine. No model call.
- 5Save
S3, then PostgresOne Markdown file per page, then the chunk is marked done.
S3, then PostgresOne Markdown file per page, then the chunk is marked done.
S3, then PostgresOne text file per sheet, then the unit is marked done.
- 6Finish
Completion poller and indexerThe poller releases the job once every unit is done. The indexer merges the files in order and writes them to Elasticsearch.
Completion poller and indexerThe poller releases the job once every unit is done. The indexer merges the files in order and writes them to Elasticsearch.
Completion poller and indexerThe poller releases the job once every unit is done. The indexer merges the files in order and writes them to Elasticsearch.
The ingestion service only plans
The ingestion service receives every new file, but it doesn't download the file or count its pages. In one database transaction it creates a JobOne run of extraction for one version of one document, with its own id. record and marks the document as extracting, so the indexer skips it. Then it sends one small MessageOne task on a queue. Ours carry ids only, such as the job, the document and the page range. that carries ids only. Everything else, such as where the file lives and which model to use, sits on the job record.
That keeps the ingestion service fast whatever the size of the file. Writing the job and the document's status in one transaction also closes a gap in which the indexer could pick up the document halfway through.
The splitter fans out
The splitter downloads the PDF once, counts its pages, writes one row per five-page chunk to the database and sends one message per chunk, in batches. This is the Fan-outSplitting one task into many that run at the same time, such as one document into 60 chunks.. Each message names the job, the document, where the file is stored, the page range and the model provider to use.
Two messages, both small
What travels on the queues | ids and page ranges, never file contents
Split message
One per file, from the ingestion service
Extraction message
One per five pages, from the splitter
Ids only. Where the file lives and which model to use stay on the job record, so the message never goes stale.
Everything one worker needs for its five pages, so it starts work without a lookup.
Source: the messages in our pipeline, ids shortened
Paying for each page once
Each chunk worker downloads the PDF, renders its five pages as images and sends them to the vision model five at a time. If the model times out, blocks the content or returns nothing, the worker retries the page on a fallback model hosted on AWS Bedrock. In a two-week sample, that happened on 8% of pages. The worker then saves each page as its own MarkdownA plain-text format that keeps headings, lists and tables. Each extracted page is saved as one Markdown file. file in S3Amazon's file storage. The original files and the text of every page are kept there. and marks its chunk done in the database.
The order of those last steps matters more than anything else in this design: model first, then storage, then the database. SQS uses At-least-once deliveryA queue's promise that every message is delivered, sometimes more than once, so workers must expect repeats., and Lambda workers do get stopped in the middle of a run. Before calling the model for a page, a worker checks whether that page's text is already in S3 and skips it if so. Because the database is updated last, it can always catch up from what is in storage.
What a retry costs
A worker stops while reading pages 41 to 45 | pick the moment it stopped
What the retry does
The retry skips pages 41 to 43, reads 44 and 45 once and pays nothing extra.The answer for page 44 was never saved, so the retry reads it again and pays for page 44 twice.Every page is already in S3, so the retry pays nothing and marks the chunk done.
Every chunk is therefore IdempotentSafe to run twice. Running the task a second time gives the same result.: running it twice gives the same result, and each page is paid for once. The only exception is a page whose answer arrived in the moment before a worker stopped. S3 is the record of what was paid for, and the database only tracks progress.
The job id is part of every page's storage path. If a changed version of the same document arrives later, it gets a new job and a clean folder, so pages from the old version are never merged into the new text.
One folder per job
Where page text lives in S3 | upload a new version to see what changes
results
a43e9a…Annual report.pdf
7f3c9e…Job for version 1
- 00001.md
- 00002.md
- 00003.md
- ⋮
- 00300.md
Kept apart, never merged into the new text
b81d20…Job for version 2
- 00001.md
- 00002.md
- 00003.md
- ⋮
- 00312.md
Zero-padded names sort in page order, so the indexer merges a document by listing its folder.
The job id is in the path. A changed file gets a new job and a clean folder.
Source: our pipeline's storage layout, ids shortened
Office files are converted first
Word and PowerPoint files go to a separate conversion function: a Container imageA packaged copy of a program with everything it needs to run, such as LibreOffice for Office files. that runs LibreOffice with no screen attached. It converts the file to PDF, saves the PDF to S3 and runs the same split step in the same run. From then on, an Office file is handled exactly like a PDF.
It is a separate function for two reasons. It needs more memory (3 GB) and a longer timeout (15 minutes) for large decks, and its own share of capacity means a pile of slide decks can't hold up PDF extraction. Conversion and splitting together take 2.9 seconds at the median and 12.3 seconds at the 90th percentile.
Spreadsheets skip the model
Spreadsheets aren't pages. One sheet in a workbook we tested held 128 million characters of text. Rendering that as page images for a vision model would be slow, expensive and worse at keeping the table's structure. For search, the raw cell text is enough, and any analysis that needs the real numbers reads the original file.
So spreadsheets take their own path, through python-calamine, a Python library built on a fast spreadsheet reader written in Rust. It reads .xlsx, .xls and .ods files, and it replaced a general-purpose document parser on this path. The output is plain text: a header for each sheet, then one line per row with the cells separated by tabs.
Each sheet gets its own worker. The first worker opens the workbook, records one unit per sheet, sends sheets 2 onwards back to the queue as separate messages and reads sheet 1 itself.
Splitting by sheet roughly halves the time
324 MB workbook with 14 sheets | one run reads every sheet | split by sheet, slowest sheet
Time to finish
Peak memory
One run Listing the sheets, 0.7 s Slowest single sheet
Four details that caught us out
- Older .xls files are read in full on open. There is nothing to gain by splitting them, so they finish in the first run, as single-sheet files do.
- The file extension can be wrong. A .xls file renamed to .xlsx turns up more often than you would expect, so we detect the format from the file's contents.
- Whole numbers gain a decimal. A cell showing 8337 comes back from the reader as 8337.0. We print whole numbers as integers, because someone will search for "8337".
- One very large sheet can't be split further. An .xlsx sheet is stored as a single stream of XML, and there is no way to start reading it in the middle. One sheet per worker is as fine as the split gets.
One checker decides when a document is done
Workers never decide whether a document is finished. With 60 chunks completing at around the same moment, "am I the last one?" turns into a race very quickly. Instead, one small PollerA small program that checks something on a schedule. Ours checks every five seconds whether a document's chunks are done. is the only writer of job status. This is the Fan-inCollecting the results of those parallel tasks and deciding when the whole job is finished., and it runs every five seconds.
Because the checker releases a document on its first run after the last chunk finishes, the slowest chunk sets the finish, and the checker adds at most five seconds.
The slowest chunk sets the finish
One 100-page document, illustrative | each row is one five-page chunk
workers
Chunk run Cold start, about 6 s Fallback model call Checker, every 5 s Slowest chunk ends, then waits for the next check
Illustrative timings shaped by our medians: 2.9 s per model call, 7.7 s at p90, 35 s end to end for 51 to 100 pages
The checker applies three rules on every run.
- A job with every chunk done is marked successful and released to the indexer.
- A job with a failed chunk is marked failed.
- A job with no progress for 30 minutes is marked failed, so nothing waits forever.
Workers update only their own chunk, with one exception. If a message fails on its last allowed delivery, the worker fails the job at once, so a broken file doesn't sit until the 30-minute timeout. The indexer then lists the job's folder in S3, merges the pages in order (zero-padded file names keep them sorted) and indexes the text like any other document.
What we learned
More workers beat faster workers
We ran the same 100 documents twice with the same chunk size. In the second run the average Lambda call was slower, 12.5 seconds against 8.8, yet the batch finished three times faster, in 89 seconds against 265, because the peak number of busy workers rose from 52 to 195. Check how many workers are actually busy before making each one faster.
More workers beat faster workers
The same 100 documents and the same chunk size | first run | second run
Average time per call
Each call got slower
Peak workers busy
Almost four times as many
Time for the whole batch
Three times faster
First run Second run
Retries have to outlast real outages
One evening our model provider had a BrownoutA partial outage in which many requests fail for a while. Servers often reply with HTTP 503, “temporarily unavailable”. of about 25 minutes. Six in ten documents in that batch failed, and Lambda cost per page reached 4.9 times normal, nearly all of it spent on retries. Three things stacked up.
- The AWS SDK treats an HTTP 503 reply as a temporary error rather than as throttling, so its adaptive rate limiting never started.
- Errors came back in 0.3 to 0.5 seconds, so 500 workers retried in step with each other.
- The queue's whole retry budget was 60 seconds, against an outage of 25 minutes.
Retries gave up long before the outage ended
One evening | the model provider's brownout | the queue's whole retry budget
Add up the retries at every layer and compare the total with how long real outages last. Ours didn't come close. Waiting longer after each failure, with a random extra delay (Backoff with jitterWaiting longer after each failed try, plus a random extra delay, so many workers don't retry at the same instant.), is the standard remedy and worth reading about before your first provider outage.
Give each file type its own queue and capacity
PDF extraction reserves 600 workers (100 for splitting, 500 for chunks), conversion 100 and spreadsheets 100, which is 800 of the account's 1,000 and leaves room for everything else (Figure 9). Without Reserved concurrencyA share of an account's Lambda capacity set aside for one function, so no other function can use it up., one large upload of slide decks could starve every other function.
One function, many message types
Splitting and chunk extraction are the same Lambda function, and a type field in each message decides what to do. The spreadsheet function runs the same image too. There is one image to build and one place to fix bugs, with separate queues and capacity for each job type.
Not everything needs a model
Spreadsheets made it obvious. Plain parsing is faster and cheaper, and it keeps working when a model provider has a bad day.
What we measured
The GPU column covers 1,332 Ray jobs from the year before the OCR model upgrade, and only the extraction step, since splitting the PDF was a separate job then. The Lambda column covers more than 26,000 jobs. They aren't the same documents run twice, so read the comparison as a before and after rather than a benchmark.
Faster at every document size
Seconds from job created to job done, median | GPU servers | AWS Lambda
GPU servers AWS Lambda
In seconds
| Pages | GPU median | GPU p90 | Lambda median | Lambda p90 |
|---|---|---|---|---|
| 1 to 20 | 316 | 537 | 15 | 54 |
| 21 to 50 | 360 | 602 | 26 | 93 |
| 51 to 100 | 472 | 746 | 35 | 135 |
| 101 to 300 | 495 | 765 | 54 | 173 |
| Over 300 | 634 | 1,354 | 60 | 169 |
1,332 GPU jobs and more than 26,000 Lambda jobs
On Lambda, documents over 300 pages took only four times as long as documents under 20 pages. That is the fan-out doing its job. After the OCR model upgrade, the GPU median for documents over 300 pages rose to 1,888 seconds, about 31 minutes.
The largest runs
Cost
Over one full week of 4,389 documents and 151,053 pages, model calls cost $260.49 and Lambda compute $12.30. That is $1.81 per 1,000 pages, and 95% of it is the model.
The model is 95% of the cost
Cost per 1,000 pages | GPU counts busy seconds only | Lambda includes the model
GPU time at $0.80 an hour Model calls, $1.72 Lambda compute, $0.08
One week of 4,389 documents and 151,053 pages
The GPU path used 3.51 seconds of GPU time per page before the OCR upgrade and 10.78 seconds after. At the $0.80 an hour we paid for a g6.xlarge GPU server, that is $0.78 and $2.40 per 1,000 pages, counting only the seconds a GPU was busy and none of the start-up or idle time between bursts. Per page, a busy GPU was cheaper than Lambda plus the model before the upgrade, and more expensive after it. What serverless bought was speed, and no bill when idle.
Limits and trade-offs
- Account capacity is shared. Every Lambda function in an account and region draws from the same pool. Reserve capacity per function, or one workload will take the rest.
- The model provider becomes the real limit. Five hundred workers with five pages each means up to 2,500 model calls in flight. Provider Rate limitThe number of requests a provider accepts per second or per minute. and brownouts now decide throughput.
- Cold starts are real. About 9% of runs were Cold startThe few seconds a new Lambda copy needs to start up before it can take its first task., and start-up took 5.8 seconds at the median, or 6.6 seconds for the LibreOffice image. Keep heavy imports and model clients off the path that doesn't need them.
- It isn't right for every workload. If documents can't be sent to an external model provider, or the load is steady enough to keep GPUs busy all day, a GPU cluster still makes sense.
Is serverless extraction a fit for your workload?
Tick each statement that is true for you
Tick the statements that are true for your workload to see what each one means.
What this means for you
All four apply: serverless fits. Split the work into small units, reserve capacity for each function and make every step safe to run twice.
- Waiting uses almost no processor or memory, so many small workers waiting side by side cost less than a few large machines.
- Workers start with the burst and stop when it ends, so you pay for the burst and nothing in between.
- Each worker calls a hosted vision model, so there is no GPU to run, scale or keep busy.
- Short pieces fit Lambda’s time limits, and splitting is what lets hundreds of workers run at once.
Any statement that is not true for you points to a long-running server or a GPU cluster for that work.
Source: the trade-offs in this post
Conclusion
The lesson from the GPU work was to structure the work first and parallelise second, and it still holds. What changed is where the heavy lifting happens. Once a model reads the page, extraction stops being a computing problem and becomes a coordination problem: splitting the work, retrying safely and knowing when everything is done. Five ideas carried this design. Each one came with a cost, and the costs are what we would tell a team starting today.
| Idea | In this pipeline | What it costs |
|---|---|---|
| 1Break every file into small, independent units | Five-page chunks and one unit per sheet, so up to 500 run at once | More messages and more status rows to track |
| 2Let the number of workers drive speed | Hundreds of short Lambda runs instead of a few large servers | The model provider's rate limits become the ceiling |
| 3Save the expensive result first | Model, then S3, then the database, with an S3 check before each page | One extra storage check per page |
| 4Keep one writer for job state | A checker that releases finished documents every five seconds | Up to five seconds before a finished document is released |
| 5Use a model only where you need one | Spreadsheets read directly, with no model call | Search gets plain cell text; analysis reads the original file |
For teams on Needl.ai, the result is that newly added documents, long ones included, become searchable and citable sooner. If you run document extraction on large volumes and want to compare notes, get in touch.
Further reading. Using Lambda with Amazon SQS, on batching, retries and partial batch failures. Lambda reserved concurrency, on keeping one function from starving the others. python-calamine, the spreadsheet reader used for Excel files. Exponential Backoff and Jitter, on the AWS Architecture Blog. The Write Last, Read First Rule, the same ordering idea applied to keeping two databases in step. Scaling AI Document Processing on Kubernetes with Ray and KubeRay, the GPU side of this story.
Glossary
The ten core terms are defined in part one. These are the rest.
Reading documents
- OCR
- Optical character recognition: software that recognises the letters in a picture of a page.
- API
- A way for one program to send a request to another over the internet and get an answer back.
- Markdown
- A plain-text format that keeps headings, lists and tables. Each extracted page is saved as one Markdown file.
Where the work runs
- Cluster
- A group of servers managed together. Our GPU workers ran on a Kubernetes cluster, with Ray spreading the work.
- Reserved concurrency
- A share of an account's Lambda capacity set aside for one function, so no other function can use it up.
- Cold start
- The few seconds a new Lambda copy needs to start up before it can take its first task.
- Container image
- A packaged copy of a program with everything it needs to run, such as LibreOffice for Office files.
How work moves
- Amazon SQS
- Amazon's queue service. Each Lambda function in this pipeline reads from its own SQS queue.
- Message
- One task on a queue. Ours carry ids only, such as the job, the document and the page range.
- Fan-out
- Splitting one task into many that run at the same time, such as one document into 60 chunks.
- Fan-in
- Collecting the results of those parallel tasks and deciding when the whole job is finished.
- Poller
- A small program that checks something on a schedule. Ours checks every five seconds whether a document's chunks are done.
Where results are kept
- S3
- Amazon's file storage. The original files and the text of every page are kept there.
- Database
- Postgres, which tracks each document's job and the status of every chunk.
- Search index
- Elasticsearch, the engine that makes extracted text searchable. A document reaches it only when every page is ready.
- Job
- One run of extraction for one version of one document, with its own id.
When things go wrong
- Retry
- Running a failed task again, usually by putting its message back on the queue.
- Idempotent
- Safe to run twice. Running the task a second time gives the same result.
- At-least-once delivery
- A queue's promise that every message is delivered, sometimes more than once, so workers must expect repeats.
- Visibility timeout
- How long a message stays hidden from other workers while one works on it. Unfinished work reappears after it.
- Dead-letter queue
- Where a message goes after failing too many times, so a person can look at it.
- Timeout
- The longest a task may run before it is stopped.
- Throttling
- A service limiting how many requests it accepts and asking callers to slow down.
- Rate limit
- The number of requests a provider accepts per second or per minute.
- Brownout
- A partial outage in which many requests fail for a while. Servers often reply with HTTP 503, “temporarily unavailable”.
- Backoff with jitter
- Waiting longer after each failed try, plus a random extra delay, so many workers don't retry at the same instant.
Measuring
- Median
- The middle value: half the documents finished faster than this and half slower.
- 90th percentile (p90)
- 90% of documents finished within this time. It shows the slow tail that a median hides.
- Throughput
- Work finished per unit of time, such as pages per second.
FAQs
What is serverless document extraction?
Serverless document extraction turns PDFs, Office files and spreadsheets into searchable text with short-lived cloud functions instead of servers you keep running yourself. Each file is split into small pieces, each piece is read by its own function, and every function stops when its piece is done. Needl.ai runs this on AWS Lambda, where a document of more than 300 pages takes about a minute.
What is AWS Lambda?
AWS Lambda is Amazon's service for running code without managing servers. Lambda starts a copy of a function for each event, such as a message arriving on a queue, runs many copies at once and bills for each millisecond of running time. When nothing is happening, nothing runs and nothing is billed. Needl.ai's extraction pipeline peaked at 507 copies running at the same time.
Why use a vision model instead of OCR?
OCR recognises characters but often loses a page's structure: table rows, chart labels and multi-column layouts. A vision model reads the page as an image and returns its text with that structure intact. In Needl.ai's benchmark of 42 extraction tools, a small group of vision models led on every quality measure, while classic text extractors kept the words but lost the tables.
When do GPU servers still make sense?
GPU servers make sense when the load is steady enough to keep them busy all day, or when documents can't be sent to an external model provider. Bursty workloads suit serverless better: Needl.ai's extraction traffic arrives in large batches with quiet hours between them, so a pool capped at 14 GPU workers was too small at peak.
What happens if a worker fails halfway through a document?
The queue delivers the message again and another worker picks up the chunk. Because the text of each page is saved before the chunk is marked done, the retry skips pages already saved and pays only for the rest. In Needl.ai's pipeline, a checker marks the document failed if a chunk keeps failing or nothing has moved for 30 minutes.
How are spreadsheets handled?
Spreadsheets skip the vision model. A spreadsheet reader extracts the cell values as plain text, with one header per sheet and one line per row, and each sheet gets its own worker. On a 324 MB workbook with 14 sheets, splitting by sheet cut the time in Needl.ai's tests from 27.8 seconds to about 14 seconds: 0.7 seconds to list the sheets, then 13.4 seconds for the slowest sheet.
Does Needl.ai train models on my documents?
No. Needl.ai does not train models on customer data. Each page is read by our primary vision model through its provider's paid API, whose terms exclude prompts and responses from training. If that model times out, blocks a page or returns nothing, the page is retried on a fallback model hosted on AWS Bedrock, where AWS does not train on the data and the model's maker has no access to it. Needl.ai runs as SaaS, as a dedicated instance or inside the customer's own AWS or Azure account. See Trust and security for the details.
Why Needl.ai Doesn't Rely on Vector Databases
Context Is Becoming the Real Infrastructure Layer for Enterprise AI
How to Choose an AI Due Diligence Platform in 2026: Rogo, Hebbia and Alternatives Compared