Ground-Truth-Aware Metric Terminology for Vector Retrieval: A Proposal for Disambiguating Evaluation in Embedding-Based Systems

Vector databases and embedding models have made semantic retrieval ubiquitous, yet evaluation practices suffer from a critical ambiguity: terms like "Recall@k" are routinely used to mean either (i) overlap with a baseline model's top-k results or (ii) performance against human- or business-defined ground truth. This conflation is comparable to mixing physical units (e.g., power vs. energy): the metrics are incommensurate, and optimizing the wrong one can reproduce baseline biases while producing misleading quality claims. This whitepaper proposes a ground-truth-aware terminology standard requiring every reported metric to explicitly encode its reference (ground truth type). Key contributions: Clear separation of customer-relevant metrics (e.g., nDCG@k[GT-W], SetOverlap@k[GT-H]) from diagnostic-only baseline metrics (e.g., BO@k[baseline=original-float32] or alias BRecall@k). Legacy-to-canonical mapping for common ambiguous terms ("Recall@k" → BO@k[baseline=B]). Minimal reproducible evaluation protocol, including candidate pool definition and uncertainty estimation. Machine-readable metric header schema (YAML/JSON) for linting and dashboard enforcement. Worked examples illustrating the "blind vs. one-eyed" effect: improvements in truth-based utility can decrease baseline overlap. The proposal addresses real-world patterns in domains like legal search and e-commerce, where general embedding semantics diverge from business relevance. This terminology is designed for immediate adoption in benchmarks (MTEB, BEIR), vector DB dashboards, and engineering workflows. We invite the community to integrate ground-truth-aware labels. Related identifier: Primary reference – AQEA Technical Report (DOI: 10.5281/zenodo.18138436)