Financial Document Extraction & Benchmarking Tool
A batch extraction pipeline pulling predefined financial fields out of annual reports - native PDFs and scanned documents alike - with a benchmarking harness to prove the accuracy claim before anyone relied on it.
1. The problem
A consulting team on a live engagement needed structured financial and operational data extracted from a large volume of annual reports, financial statements, and scanned documents. They had been doing it manually through a consumer chatbot interface, and the workflow broke down exactly where you’d expect: complex tables, scanned images with no text layer, and no way to process documents in batch.
The request that came in had five requirements: extract predefined fields at high accuracy; handle both native PDFs and scanned documents; output to a structured format for analysis; support recurring, repeatable extraction across many documents; and potentially be packaged for the client’s own team to run.
The honest framing of my job was feasibility assessment first, build second - establishing whether this could be done reliably enough to stake an engagement on, rather than demonstrating that it could be done once.
2. Approach
Document handling
A managed Document Intelligence OCR service for layout and table recognition, handling native and scanned documents through one path. Table structure is the hard part in financial reports - a figure’s meaning is defined by its row and column headers, and losing the layout means losing the semantics, however good the raw text extraction is.
Extraction and mapping
Field extraction routes through the firm’s internal LLM gateway, with a distinct mapping layer translating extracted content into the target field schema. Separating extraction from mapping is the design decision that matters: financial reports label the same concept differently across issuers and years, so mapping needs to evolve independently of extraction logic rather than being tangled into one prompt.
An alias cache handles the label-variation problem - resolving the many surface forms of a single financial concept without re-deriving the mapping on every document. This is both an accuracy and a cost mechanism: fewer repeated model calls for labels already resolved.
Benchmarking harness - and its honest limits
The component that made the assessment credible rather than anecdotal. The harness compares eight extraction methods against hand-typed ground truth: full-document OCR with regex, targeted-page OCR with regex, coordinate-based PDF parsing, managed-OCR layout output with regex, targeted-page managed OCR, the production LLM pipeline with automatic page-finding, the same pipeline with ground-truth pages supplied, and an offline document-conversion library.
The production pipeline scored 100% on both samples. But the finding I’d actually lead with is the opposite of a success claim:
The regex baselines also scored 100% - and that number is misleading, which the harness proved. Those methods were overfit to the two sample documents: hardcoded note numbers, string-matching on OCR artifacts, fixed row indices. The proof is that feeding the same regex logic a different markdown formatting of the same content collapsed it to 7.7%. A method that looks perfect and shatters on formatting variation is worse than useless in production, because it will pass a demo and fail on document three.
And the benchmark is n=2, single-issuer. I stated that explicitly in the findings rather than letting “100% accuracy” travel without its denominator. Two documents cannot establish reliability; what they can establish is relative method ranking and the overfitting failure above - which is genuinely decision-useful, and is what the requesting team needed to know.
The documentation for this project is deliberately more conservative than the code: it calls its own fastest baseline misleading and its own sample size tiny. That direction of error is the correct one when a team is about to commit engagement workflow to your conclusion. An earlier prototype’s higher-sounding accuracy figures were superseded and are not reproducible from the current pipeline - so they were retired rather than quoted.
This is the same instinct I applied on the strategy knowledge agent: for anything where a team is about to make a commitment based on my output, the measurement harness is the deliverable, and the extraction pipeline is what it measures.
Structured export
Results export to spreadsheet format for direct analytical use - the format the requesting team actually works in, rather than an API they’d have to build against.
3. What I’d highlight
- Assessment framing over demo framing. The team needed to know whether to build their engagement workflow on this, so I built the thing that answers that question - a benchmark - alongside the pipeline.
- Separating extraction from field mapping, so label variation across issuers and years is handled in a layer that can evolve independently.
- Alias caching as a combined accuracy and cost mechanism for repeated label resolution.
- Table-structure preservation as the core technical requirement in financial documents, not an afterthought to text extraction.
- Honest scope conclusions. Part of the deliverable was documented findings on where the approach is and isn’t reliable - including the packaging-and-handover question, which depends on a client technical conversation rather than on the tool.
4. Skills demonstrated
document extraction pipelines · OCR and layout analysis · table structure extraction · LLM structured output · field mapping and alias resolution · caching strategies · accuracy benchmarking · technical feasibility assessment · financial data modelling · Python · stakeholder-facing technical reporting