Fifty Queries Is Not a Measurement: Sampling adequacy, stochastic stability, and the coverage-leverage of substrate-level remediation in AI recommendation measurement

Read full text
Authors: Paul Sheals, Tim de Rosen
Cited by: 6
References: 1

Abstract

Working paper. In August 2026 the IAB published the first industry standard for measuring brand visibility in AI-powered discovery, ruling that a programme running fewer than 50 queries per category is exploratory and not fit for budget decisions. We agree the floor is real and argue it is set against the wrong quantity: the standard measures visibility (whether a brand is mentioned) and, by its own scope statement, excludes multi-turn conversations and recommendation at the point of decision. We show that (i) the first-prompt space is combinatorially large — of order 10^4 to 10^6 distinct prompt-contexts per category — so 50 single-shot queries sample well under one-tenth of one percent of it, once; (ii) large-language-model output is stochastic, so a single reading has no recoverable variance and no confidence interval (in blind founding data, one question produced six different terminal winners across twenty runs); and (iii) observational conversational-panel data, however large, cannot isolate the stochastic component, attribute change to the model, or support a counterfactual. We then advance a positive claim: because a model's recommendation for any phrasing is generated from a small, shared semantic substrate — the brand's legibility, compiled as brand.context — the object worth measuring and repairing is not the output space but the substrate that generates it. We estimate that this substrate comprises of order 8 to 15 context atoms per brand, and that repairing it covers the entire 10^4 to 10^6 prompt-context space of a category (coverage leverage of roughly 10^3 to 10^5 prompt-contexts per atom), given as a pre-registered hypothesis testable by counterfactual injection. Finally, we show the argument strengthens as AI becomes more agentic: research assistants and shopping agents collapse the phrasing-variation problem themselves, so what a brand must make legible is not more prompts but more substrate layers. Not peer-reviewed. The prompt-space model and coverage ratio are transparent order-of-magnitude estimates on stated assumptions, offered as a pre-registered hypothesis, not measured results.

Fields of Study & Subjects

Ethics and Social Impacts of AI Explainable Artificial Intelligence (XAI) Artificial Intelligence in Healthcare and Education

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
18 August 2026