Relational Consistency Probing: Mapping Judgment Stability Across Domains in Deployed Language Models

Read full text Summarise with VIRAT
Authors: Declan Michaels
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

This experiment measures how deployed language models' similarity judgments shift under cultural framing perturbations, comparing drift magnitude across conceptual domains. We probe models with pairwise concept similarity ratings (1-7 scale) across three domains arranged by expected cultural loading: physical/causal (control, expected stable), institutional/social (intermediate), and moral/cultural (target, expected unstable). Each pair is rated under seven framings: neutral baseline, four cultural perspectives derived from Mary Douglas's Grid-Group Cultural Theory, an irrelevant control (weather context), and a nonsense control (geometric morality). The central hypothesis is that models trained with monocultural alignment signals will show stable judgment geometry in physical domains but rigid or noisy geometry in moral domains. Two follow-on experiments test generalization to software engineering and financial concept domains. Pilot data (N=1 deterministic run per model, temperature 0.0) has been collected for five models and is labeled as exploratory. This pre-registration covers the confirmatory phase: stochastic replication runs (5 repetitions at temperature 0.7) providing variance estimates, confidence intervals, and pre-specified permutation tests.

Fields of Study & Subjects

Topic Modeling Categorization perception and language Ethics and Social Impacts of AI

Publication details

Publication date
1 January 2026
Language
EN
Document type
Preprint