Who Watches the Watchmen?
Abstract
AI systems are increasingly evaluated on ethics benchmarks to ensure alignment with human values. However, a critical question remains unexamined: who evaluates the evaluators? Benchmarks themselves may encode hidden assumptions, cultural biases, and malformed framings that AI systems inherit during training or evaluation — a phenomenon termed drift in, drift out. We introduce the Transcendent Invariant Kernel (TIK) framework, a systematic methodology for meta-evaluating AI ethics benchmarks. We model AI as LLM ⊕ CogOS (Language Model ⊕ Cognitive Operating System) and show, via Gödel's incompleteness applied to the deductive closure of CogOS, that undecidable ethical statements exist, and that the resulting undecidability trap is resolvable only by external grounding. We formalize this as the Gödel-trolley: ethical dilemmas conditioned on undecidable statements have no honest internal resolution, but an external kernel Φ provides principled resolution via the "do not do wrong" principle. Through a 3+1+1 judge architecture combined with an external kernel grounded in three ethical traditions (Christ-Teachings, Kantian, Ubuntu), we detect ontological holes (hidden assumptions) and forbidden fruits (malformed questions). We develop a learned TIK predictor architecture and demonstrate adversarial robustness methodology via perturbation analysis. We evaluate 9 major benchmarks (Moral Machine, ETHICS, TruthfulQA, Social Chemistry, Moral Stories, CommonsenseQA, MMLU-Ethics, Scruples, GAIA). Upon completion of data collection, we plan to release BenchmarkMeta, a dataset of annotated questions with TIK scores. Note: This preprint presents the methodological framework with illustrative empirical results. Empirical findings are preliminary and should not be treated as definitive. Feedback welcome: artiomkovnatsky@pm.me
Fields of Study & Subjects
Ethics and Social Impacts of AI
Ethics in Business and Education
Philosophical Ethics and Theory
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 5 March 2026
- Language
- EN
- Document type
- Preprint