A Vendor's Figure Measures Their Data, Not Yours
When Tom asked the vendor about the bounces, the answer was straightforward. The 97% came from a reference set of US mid-market companies, mostly established firms with a single main domain and clean registration records. Tom's list had a lot of small businesses, international companies, recent rebrands and companies acquired in the past 18 months.
Accuracy depends on the input. A tool can be excellent on one population and mediocre on another, so a published figure tells you the tool's ceiling, not what you'll get on your list.
For a domain finder, the useful definition is: the share of your records for which the tool returns the correct operational domain, measured against verified answers drawn from your own data.
"Correct" needs defining before you start. Tom used the domain that actually receives the company's email, confirmed by an MX record check and the company's own website — not the top search result.
The Four-Step Benchmarking Method
Build a Ground Truth Set
A ground truth set is a sample of your records for which you've independently verified the right domain. Tom built his the week after the bounce problem, from 150 records in the next planned batch. He deliberately included a mix: large US enterprises, mid-market companies, small businesses, international companies, recent rebrands and post-acquisition companies. He verified each domain by hand and noted how he'd confirmed it, and flagged anything ambiguous.
It took about four hours, and it can't be automated — because automated verification is the thing being tested.
- Match the mix to your real list, then oversample the hard cases. Fortune 500 records alone tell you nothing about rebrands.
- Aim for roughly 30 or more records per segment you care about. Smaller segments give directional results only. Tom's rebrand group had 20 records, so every conclusion about it comes with a caveat.
Measure Precision, Recall and Match Rate Separately
Tom's first instinct was a single accuracy percentage. The tool had returned a domain for 138 of the 150 records, and 126 of those were correct. Here's what that single number hides:
| Measure | What it asks | Tom's result |
|---|---|---|
| Match rate | For how many records did the tool return anything? | 138 / 150 = 92% |
| Precision | Of the domains it returned, how many were right? | 126 / 138 = 91.3% |
| Recall | Of all records with a correct answer, how many did it find correctly? | 126 / 150 = 84% |
If someone asked Tom for "the accuracy," 91% (precision) and 84% (recall) are both defensible answers, and they tell different stories. Precision describes how far to trust a returned result. Recall describes how much of your list the tool actually solves.
The 12 wrong results and 12 unmatched records need different fixes: wrong results need a quality gate before they reach a sequence, and unmatched records need manual research.
Breaking Down by Segment
Then he broke the results down by segment:
| Segment | Records | Returned | Correct | Precision |
|---|---|---|---|---|
| Large US enterprises | 30 | 30 | 30 | 100% |
| US mid-market | 30 | 30 | 29 | 96.7% |
| Small businesses (<20 staff) | 25 | 22 | 21 | 95.5% |
| International | 20 | 17 | 15 | 88.2% |
| Recently rebranded | 20 | 18 | 13 | 72.2% |
| Post-acquisition | 25 | 21 | 18 | 85.7% |
The segment view shows what the vendor's figure couldn't. The tool did very well on established US companies and noticeably worse on rebrands.
Two cautions apply. First, small samples are noisy: 13 correct out of 18 is 72%, but the true rate could plausibly be anywhere from about 49% to 88%. Treat the rebrand result as a warning sign, not a precise number. Second, because Tom oversampled hard cases, his overall 91.3% isn't what he'd see on a typical production batch. To estimate that, weight each segment's result by its share of your real list.
Check Whether Confidence Scores Are Calibrated
The tool also returned a confidence score between 0 and 1. A well-calibrated score should mean that higher-confidence results are right more often. Tom grouped his 138 results into two bands and checked each against his ground truth.
| Confidence band | Returned | Correct | Actual accuracy |
|---|---|---|---|
| 0.9 and above | 108 | 104 | 96.3% |
| Below 0.9 | 30 | 22 | 73.3% |
Tom had been letting results at 0.75 and above skip manual review, which meant part of a band that was right only about 73% of the time went straight into sequences. He raised the cutoff to 0.9. The review queue grew, and what bypassed it was much more reliable.
Even so, four of the 108 high-confidence results were wrong. That's why he also added an email verification step before sending — no confidence threshold removes every error. It also explains why six of his original bounces had high scores: high confidence reduces risk but doesn't eliminate it.
Re-Audit Regularly
Company data changes constantly. Businesses rebrand, get acquired and move domains, and enrichment databases update on their own schedule, often weeks or months behind. A benchmark from January may not describe your tool's performance in June.
A quarter later, Tom ran a fresh audit on 100 records. Precision came out at 93.1% (81 correct of 87 returned), up from 91.3%.
He was careful about what that meant. The sample was different and drawn from a narrower target list, so the improvement wasn't a clean comparison, and he couldn't tell how much came from list composition and how much from any change in the tool. What the audit did give him was a current figure and a view of which segments were weakest, so he could aim manual checks where they mattered.
What Tom Told Rachel
A few weeks after the bounce problem, Tom gave Rachel a one-page summary.
On established US companies, the tool performed about as well as the vendor said. Precision on his sample was 97% or higher.
On rebrands and international companies, precision was well below that, though the small samples meant he'd keep testing.
The bounce problem came from a Q1 list that leaned heavily on recently funded companies, many of which had renamed or moved infrastructure.
The fix wasn't replacing the tool. It was:
- A manual verification step for rebranded, international and post-acquisition records
- A 0.9 confidence cutoff
- Email verification before every send
Six hours of work in total — four for the first ground truth set, two for the re-audit — showed where to trust the tool and where to check it.
A Ground Truth Template You Can Copy
Set up a sheet with these columns, then fill in 100 to 150 rows:
Ground Truth Sheet Columns
- Company name and country
- Segment (enterprise, mid-market, small, international, rebrand, post-acquisition)
- Verified domain
- How you verified it (MX check, company site, other)
- Tool's returned domain
- Tool's confidence score
- Result: correct, wrong or no match
From there, calculate match rate, precision and recall overall and per segment, then split results at your confidence cutoff. The whole process takes six hours the first time and two hours on re-audit runs.