Pricing anomaly detection on synthetic distributor transaction data
If you generate the leakage yourself, you can measure your own false positive rate instead of guessing at it.
Problem
Anomaly detection on pricing data has an uncomfortable property: on real customer data you never learn whether a flagged line was actually wrong. The customer investigates, and either comes back weeks later or does not come back at all.
That makes it very easy for a vendor — including us — to claim detection quality that has never been tested. We wanted a setup where the correct answer is known in advance, so precision and recall can be stated as measurements rather than impressions.
The question: on a dataset where we know exactly which lines are leakage, how many does the detection actually catch, and how many clean lines does it wrongly flag?
Input
- A generated distributor dataset: customers with differing contract terms, a SKU catalogue with cost histories, quantity-break tiers, branch-level overrides and a realistic order distribution across 24 months.
- Known leakage injected at controlled rates across several distinct patterns — supplier cost increases never passed through, contract pricing not applied at order entry, discounts below an agreed floor, and quantity breaks applied at the wrong tier.
- Deliberate noise: legitimate negotiated exceptions and promotional pricing that look anomalous but are not errors. Without these the benchmark is too easy and the false positive rate is meaningless.
Approach
- Reconstruct the expected price for every line from contract terms, price files, cost records and quantity-break tiers — the same reconstruction step an audit performs on real data.
- Compare each invoiced line against its reconstructed expected price, and set aside the lines that reconcile.
- Group the remaining variances by probable cause rather than by size, so that one systemic pattern across 800 lines does not present as 800 unrelated findings.
- Score against the injected ground truth, reported per leakage pattern rather than as a single headline accuracy figure — the patterns behave very differently and an average hides that.
Architecture
- 01GeneratorProduces the customer, catalogue, cost-history and order tables, plus a ledger recording every line where leakage was injected and why.
- 02ReconstructionDerives the expected price per line from the contract, price file, cost record and quantity-break tier applicable on the transaction date.
- 03ComparisonMeasures invoiced price against expected price and classifies the variance direction and magnitude.
- 04GroupingClusters variances into candidate findings by probable cause, customer and item.
- 05ScoringJoins candidate findings back to the injection ledger to compute precision and recall per pattern.
Output
Not yet measured
This experiment has a defined method and architecture, but the run has not been completed and scored. We publish results once they have been measured — an estimated number is not a result, and putting one here would undermine every other figure on this site.
Failure modes
- Legitimate negotiated exceptions are the hardest case by design — they are, structurally, identical to the thing we are looking for. Separating them requires knowing the agreement existed, which is exactly the information that is usually not in the export.
- Cost-history gaps: when a cost record is missing for the transaction date, the expected price cannot be reconstructed and the line has to be excluded rather than flagged. How often that happens on real exports is a bigger practical problem than detection quality.
- Quantity-break boundaries produce variances that are real but trivially small, and will dominate a finding count if they are not grouped and sized before ranking.
Limitations
- Synthetic data is generated from assumptions about how distributor pricing behaves. If those assumptions are wrong, the measured detection quality will not transfer to a real export.
- The generator produces cleaner identifiers than any real ERP export. Real data brings duplicate customer records, inconsistent item codes and unit mismatches that this benchmark does not test.
- A measured result here is evidence that the method works on data shaped like this. It is not a forecast of what any particular business would find.
Production considerations
- Real exports need a normalization stage this benchmark skips entirely — reconciling customer and item identifiers, units and date formats before any reconstruction is possible.
- Findings need traceability all the way back to invoice line identifiers so a customer can look each one up in their own system and argue with it.
- A confidence or evidence-strength indicator matters more than raw detection volume: a ranked list a team stops trusting after three bad findings is worse than a shorter list.
What we learned
Being able to state a false positive rate is a stronger position than being able to state an accuracy figure, because it is the number that determines whether a team keeps using the output.
Grouping before ranking is not a presentation detail. It is the difference between a report someone acts on and a spreadsheet of 2,000 rows.
This is an engineering experiment on synthetic data. It is not a client engagement, it is not a case study, and nothing here is a performance guarantee for any particular business.
Related: Distributor Margin Audit →