Fifty Queries Is Not a Measurement: Sampling adequacy, stochastic stability, and the coverage-leverage of substrate-level remediation in AI recommendation measurement
Abstract
Working paper. In August 2026 the IAB published the first industry standard for measuring brand visibility in AI-powered discovery, ruling that a programme running fewer than 50 queries per category is exploratory and not fit for budget decisions. We agree the floor is real and argue it is set against the wrong quantity: the standard measures visibility (whether a brand is mentioned) and, by its own scope statement, excludes multi-turn conversations and recommendation at the point of decision. We show that (i) the first-prompt space is combinatorially large — of order 10^4 to 10^6 distinct prompt-contexts per category — so 50 single-shot queries sample well under one-tenth of one percent of it, once; (ii) large-language-model output is stochastic, so a single reading has no recoverable variance and no confidence interval (in blind founding data, one question produced six different terminal winners across twenty runs); and (iii) observational conversational-panel data, however large, cannot isolate the stochastic component, attribute change to the model, or support a counterfactual. We then advance a positive claim: because a model's recommendation for any phrasing is generated from a small, shared semantic substrate — the brand's legibility, compiled as brand.context — the object worth measuring and repairing is not the output space but the substrate that generates it. We estimate that this substrate comprises of order 8 to 15 context atoms per brand, and that repairing it covers the entire 10^4 to 10^6 prompt-context space of a category (coverage leverage of roughly 10^3 to 10^5 prompt-contexts per atom), given as a pre-registered hypothesis testable by counterfactual injection. Finally, we show the argument strengthens as AI becomes more agentic: research assistants and shopping agents collapse the phrasing-variation problem themselves, so what a brand must make legible is not more prompts but more substrate layers. Not peer-reviewed. The prompt-space model and coverage ratio are transparent order-of-magnitude estimates on stated assumptions, offered as a pre-registered hypothesis, not measured results.
Fields of Study & Subjects
Ethics and Social Impacts of AI
Explainable Artificial Intelligence (XAI)
Artificial Intelligence in Healthcare and Education
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 18 August 2026