Generator–Judge Family Confound in LLM Judge Reliability Harnesses

I'm testing whether an LLM judge's measured reliability depends on whether the model that generated its test items belongs to the same family as the judge itself. The motivation is that reliability harnesses like RAND's Judge Reliability Harness build their perturbed test items (label flips, paraphrases, formatting and verbosity changes) using a single LLM generator, while the judges under test sometimes come from that same family, so a judge's reliability score could partly reflect who authored the perturbations rather than the judge's actual robustness. That would make part of the headline number an artifact of how the harness was built. To check this, I reproduce the harness and run a pre-registered 4×4 generator-family × judge-family factorial on HarmBench, using all 298 behaviours as the clustering unit, five perturbation types, and three repeats, with four open-weight families (Llama, Qwen, Mistral, Gemma) served on a single stack so that serving differences can't confound the comparison. The primary contrast is same-family (diagonal) versus cross-family (off-diagonal) measured reliability, analysed with item-clustered bootstrap and a within-judge permutation test, plus a TOST equivalence test against a 3-percentage-point smallest effect size of interest. I'm treating the direction as genuinely open, since same-family judging could raise or lower measured reliability and a precise null is a real and reportable outcome, so the hypotheses, analysis plan, exclusion rules, and decision gates are all fixed here before any confirmatory data is collected. As a contribution back to the harness, I add a family-aware reporting control that breaks the reliability metric down by generator-versus-judge family overlap.