Why Most Tool Evaluations Miss What Actually Matters
The standard way to evaluate a domain matching tool is to ask three questions: what is the coverage, what is the accuracy rate, and what does it cost? These are not wrong questions. They are incomplete ones.
Coverage tells you how many records the tool can return something for. Accuracy tells you — in aggregate, on the vendor's reference dataset — what proportion of those returns are correct. Cost tells you the per-record price at scale. None of these three questions tell you how the tool behaves on the specific failure modes that will actually cost your team time.
A corporate domain matching engine is a system that resolves a natural language company name — which may be ambiguous, abbreviated, rebranded, or incomplete — to a single verified operational domain, with a stated confidence level indicating the reliability of the match.
The five dimensions that differentiate high-performing matching engines from mediocre ones are not coverage or headline accuracy, but failure mode transparency, segment-level performance, disambiguation capability, confidence score calibration, and operational freshness.
Lena's evaluation had measured coverage and headline accuracy. It had missed all five. Here is the checklist.
The Five Criteria
Failure Mode Transparency
The first thing Lena had not evaluated was what the tool did when it was wrong.
Every matching tool produces incorrect results. The question is not whether a tool fails — all of them do, on some records — but whether it fails visibly or invisibly. A tool that returns a wrong domain with the same high confidence score it assigns to correct domains is producing invisible failures: results that look indistinguishable from accurate ones, which pass through any confidence-based quality gate without triggering review.
A tool that flags its uncertainty — returning a lower confidence score on records where it is less certain, returning nothing rather than a low-confidence guess, or explicitly marking records as ambiguous — produces visible failures that can be routed to a review queue before they reach a sequence.
Take twenty records you know are hard — recently rebranded companies, post-acquisition entities, companies with common names that appear in multiple markets. Run them through the tool. Check whether the confidence scores on the hard records are lower than the confidence scores on the straightforward ones. If they are indistinguishable, the confidence score is not calibrated to difficulty. It is decorative.
Segment-Level Performance, Not Headline Accuracy
A headline accuracy figure is an average. Averages conceal distributions. A tool with 93% overall accuracy might perform at 98% on large, established US enterprises and 71% on recently-funded startups, international companies, or businesses that have undergone structural changes in the past two years.
This matters because B2B outreach lists are rarely uniformly distributed. A team targeting recently-funded companies — the exact segment most likely to have undergone domain changes in the preceding year — using a tool that performs well on average but poorly on that specific cohort will experience failure rates significantly higher than the headline accuracy suggests.
Build a stratified sample of 100 to 150 records from your own CRM, deliberately overweighting the segments your target list is concentrated in. Run the tool on that sample. Calculate accuracy separately by segment — enterprises vs small businesses, domestic vs international, companies with stable histories vs companies with acquisitions or rebrands in the past 24 months. The segment breakdown will tell you more about the tool's fit for your use case than any number the vendor provides.
Lena ran this test in month three — after the Monday correction runs had become a fixture of her week. The result:
96.1% precision on large US enterprises. 68.3% on companies that had been acquired or rebranded in the preceding 18 months. Her Q3 outreach had been concentrated in the second segment. The headline accuracy had been 91%. The segment accuracy had been the number that mattered.
Disambiguation Capability
Company names are frequently ambiguous. "Apex Solutions" refers to multiple real, active, commercially legitimate entities simultaneously. "Principal Financial" is shared by a major financial services firm and numerous smaller companies. "Atlas Group" returns dozens of entities across industries and geographies.
A domain matching engine that handles disambiguation well accepts additional context — industry vertical, geography, founding year, employee count — and uses that context to narrow the result set before returning a match. A tool that does not handle disambiguation will return the highest-confidence match based on name alone, which is frequently the most prominent entity in its database rather than the one the researcher intended.
Submit ten genuinely ambiguous company names — names shared by multiple entities. Submit each twice: once with only the name, once with name plus industry vertical and city. If the second submission returns a more specific result, the tool is using context to disambiguate. If both submissions return identical results, the additional fields are being discarded.
Lena's tool had industry and city fields in its API. She had assumed they were doing something. In month four she found the results were identical whether she passed those fields or not. The tool was accepting the input and discarding it. She had been paying for a disambiguation capability that did not function.
Confidence Score Calibration
A confidence score is only useful if it is calibrated — if the tool's stated confidence corresponds to its actual accuracy at that confidence level. A tool that says 0.9 confidence should be correct approximately 90% of the time on records at that score. A tool that says 0.9 confidence and is correct 74% of the time on those records is miscalibrated in the direction that costs you most: it is telling you to trust results you should be verifying.
Calibration is distinct from accuracy. A tool can have high overall accuracy and poor calibration if it assigns similar confidence scores to correct and incorrect results. Calibration is what makes a confidence score actionable as an operational threshold.
Using your ground truth set from criterion two, group the tool's results by confidence band — above 0.9, 0.7 to 0.9, below 0.7. Calculate the actual accuracy rate in each band. If the actual accuracy in the 0.7–0.9 band is substantially below 80%, the tool is overconfident in that range and your manual review threshold should be adjusted upward. The calibration check takes thirty minutes with a ground truth set already built. Without one, you have no basis for setting a confidence threshold at all.
Lena set her manual review threshold at 0.8 based on the vendor's recommendation. Her ground truth calibration data showed 74% actual accuracy in the 0.7–0.9 band. Every record she had released with a confidence score between 0.8 and 0.9 had been approved based on a threshold that her own data did not support.
Data Freshness and Update Cadence
A domain matching engine is only as accurate as its underlying data. Companies rebrand, get acquired, migrate domains, spin up subsidiaries. A tool whose database was last refreshed six months ago will produce systematically incorrect results on any company that changed its domain in the intervening period — with high confidence, because the old domain is still in the database and looks like the right answer.
Data freshness is particularly consequential for outbound-focused teams, because outreach programs tend to target recently-funded companies, companies in growth phases, and companies undergoing transitions — precisely the segment most likely to have changed their domain recently.
Ask the vendor directly: how frequently is the underlying domain database updated, and what is the typical lag between a real-world domain change and the database reflecting it? Then test independently: take ten companies from your ground truth set that you know underwent a domain change in the past six to twelve months and check what the tool returns for each. If the tool consistently returns the old domain for recently-changed companies, the database lag is material for your use case.
Lena's tool's database was updated quarterly. In her market — recently-funded B2B SaaS companies — the average company had a meaningful domain or infrastructure change event roughly once every 18 months. A quarterly refresh sounds frequent until you realise it means every company that changed their domain in the three months since the last update is a confident miss.
The Checklist in Use
Lena sent the checklist to her RevOps network with one annotation: run each test before the contract, not after. Total time: roughly six hours — two to four building the ground truth set, the rest running the five evaluations.
The Full Checklist
- Failure mode transparency: Does the tool assign lower confidence to genuinely hard records, or are all scores uniformly high?
- Segment-level performance: What is accuracy on your specific ICP segments — not on the vendor's reference dataset?
- Disambiguation capability: Does the tool actually use context fields (industry, geography) to return more specific results on ambiguous names?
- Confidence score calibration: Does the tool's stated confidence correspond to its actual accuracy rate in that confidence band?
- Data freshness: How frequently is the database updated, and what is the lag between a real-world domain change and the database reflecting it?
A tool that passes all five gives you a map of its gaps: you know exactly where to add manual review, where to tighten the confidence threshold, and where to layer in a supplementary source.
The evaluation does not guarantee the right choice. It guarantees the choice you make is based on how the tool performs on your data — not on the vendor's reference population, which is a different dataset from the one you will be running through it every Monday morning.