A Psychometric Evaluation of LLM-Scored Communication Skills: Construct Validity and Bias Analysis

With the proliferation of AI-powered scoring systems in K–12 and higher education, questions of scoring validity and demographic equity have become urgent research priorities. This audit subjects an LLM-based essay scoring system to a modern, SOTA "LLM Psychometric" evaluation. Moving beyond classical correlational statistics (like Mantel-Haenszel DIF), we implement advanced Causal Fairness protocols. Using 800 real student essays drawn from the ASAP 2.0 corpus, stratified across intersectional demographics including Race and SES, we evaluate the AI scorer on: (1) Predictive Validity via Cleary's Rule (1968) to test calibration invariance; (2) Polytomous Non-Uniform Bias via Ordinal Logistic Regression (OLR) to capture "Variance Collapse"; and (3) Causal Linguistic Redlining via Counterfactual Lexical Perturbation, isolating syntactic dialect bias from cognitive reasoning. Collectively, our findings demonstrate that while LLM scorers achieve acceptable correlational agreement (QWK = 0.339), they systemically fail SOTA causal fairness tests. We conclude by proving that Multi-Agent Construct Decoupling successfully remediates the detected algorithmic prejudice (+1.10 points).