Messy RFQ and PDF documents to structured line items
Per-document accuracy looks fine. Per-line-item accuracy is where the actual work lives.
Problem
Quotation work starts with a document somebody sent you: a PDF, a spreadsheet, an email body, or a photograph of a printed enquiry. Before anything can be priced, the line items have to come out of that document correctly.
Extraction demos are almost always shown on clean, native PDFs with a tidy table. Real enquiries include scans, stamps over the quantity column, merged cells, multi-page tables that break across pages, handwritten annotations and free-text descriptions that do not match any catalogue entry.
The question: across a realistic mix of enquiry documents, how does line-item extraction accuracy vary by document type, and what specifically causes it to fail?
Input
- A corpus of genuine enquiry and purchase-order documents assembled from public tenders and openly published procurement documents, plus any documents shared directly with identifying details removed.
- Deliberately spread across categories: native digital PDFs, clean scans, poor scans, photographs, spreadsheets and plain email text.
- Ground truth established by transcribing every line item by hand before any extraction is run, so scoring is against a human reading of the document rather than against another model.
Approach
- Score at line-item level, not document level. A document where nine of ten lines are correct is not a 90% success — it is a quote that needs checking, which is the same operational cost as one that needs checking for any other reason.
- Score each field separately: description, quantity, unit, and any part or reference number. These fail independently and at very different rates.
- Report per document category rather than as a single figure, since the categories are not comparable and an average across them would be misleading.
- Catalogue every failure with the input that caused it, because the failure catalogue is more useful to a buyer than the headline number.
Architecture
- 01IngestionNormalizes the input into page images and any embedded text layer, keeping both — the text layer is more accurate where it exists and absent on scans.
- 02Table and region detectionLocates the line-item region, which is the step that fails hardest on documents where the table has no ruled borders.
- 03Field extractionPulls description, quantity, unit and reference per row, retaining the source text span for each value.
- 04ValidationApplies structural checks — quantities are numeric and positive, units come from a known set, row counts are consistent across pages.
- 05ScoringCompares against the hand-transcribed ground truth per field and per document category.
Output
Not yet measured
This experiment has a defined method and architecture, but the run has not been completed and scored. We publish results once they have been measured — an estimated number is not a result, and putting one here would undermine every other figure on this site.
Failure modes
- Tables without ruled borders: region detection has nothing structural to anchor to and rows merge or split.
- Multi-page line-item tables where the header does not repeat — later pages lose their column meanings.
- Stamps, signatures and handwriting placed over the quantity or unit column, which is common on documents that have been through an approval process.
- Free-text descriptions that combine several items in one line, where the correct output is arguably more than one row and there is no single right answer.
- Units expressed inconsistently within the same document — pieces, nos, sets and boxes used interchangeably for the same item.
Limitations
- Documents sourced from public tenders skew more formal and better structured than a typical trade enquiry arriving by email. Real inbound is likely to be messier than this corpus.
- Ground truth is one person's reading of each document. On genuinely ambiguous lines, another reader could reasonably transcribe differently.
- Extraction quality is only the first stage. Matching an extracted description to the right catalogue item is a separate and harder problem, and is not measured here.
Production considerations
- Confidence has to be reported per line and calibrated against this corpus, so that a review queue can be thresholded rather than requiring every line to be checked.
- Every extracted value needs to retain its source span, so a reviewer can see the text a number came from without reopening the original document.
- Corrections made during review should be captured and fed back, since customer-specific phrasing is exactly the signal that improves matching over time.
What we learned
Document-level accuracy is a vanity metric in this problem. The operational unit is the line item, and the operational cost is whether a human has to re-check the document at all.
The failure catalogue is more persuasive to a buyer than the accuracy figure, because it tells them whether their documents are in the hard category.
This is an engineering experiment on public documents. It is not a client engagement, it is not a case study, and nothing here is a performance guarantee for any particular business.
Related: RFQ Automation →