Psychometric Audit Pipeline: An Audit-Before-Modeling Framework for Reliability, Fairness, and Interpretable IRT Analysis of Mental-Health Self-Report Scales

Introduction This repository presents Psychometric Audit Pipeline, a research-grade, open-source framework for auditing the measurement validity, reliability, fairness, and uncertainty of mental-health self-report scale data prior to downstream modeling. The project is motivated by a core methodological concern in applied machine learning and psychological research: predictive or latent models are frequently fit to questionnaire data without first establishing whether the underlying measurement properties justify such modeling. The pipeline operationalizes an audit-before-modeling paradigm, in which data quality diagnostics, psychometric reliability checks, fairness screening, uncertainty quantification, and interpretable latent trait modeling are integrated into a single, reproducible workflow. The emphasis throughout is on methodological rigor, transparency, and ethical restraint, rather than predictive accuracy, automation, or deployment. While motivated by mental-health measurement, the pipeline is intentionally instrument-agnostic and can be adapted to other psychological, educational, or survey-based self-report scales. Conceptual Framework The pipeline is structured around four sequential methodological questions that reflect best practices in psychometrics and responsible modeling: Is the data suitable for modeling?(Data integrity, response behavior, and missingness) Is the scale internally consistent?(Reliability and item functioning) Is the scale comparable across groups?(Fairness and bias diagnostics) If so, what latent structure can be estimated transparently?(Interpretable AI via Item Response Theory) Each stage produces explicit outputs, diagnostics, and warnings. Later stages are designed to be interpretable only in light of earlier audit results, preventing inappropriate escalation from flawed measurement to latent modeling. Methods Data Quality Audit The pipeline begins with item-level diagnostics intended to surface structural issues that would invalidate reliability estimation or latent modeling if ignored. These include: Missingness analysis to quantify nonresponse patterns Response distribution summaries for exploratory inspection Floor and ceiling effect detection to identify restricted variability Explicit handling of sentinel missing codes (e.g., -1 → NA) These procedures are designed to make data pathologies visible before any psychometric or latent assumptions are imposed. Reliability and Internal Consistency Internal consistency is evaluated using multiple complementary diagnostics: Cronbach’s alpha for the full scale Corrected item–total correlations to identify poorly functioning items Group-stratified reliability estimates where applicable To avoid overinterpretation of single-run estimates, the pipeline optionally supports bootstrap-based confidence intervals for reliability metrics. Explicit minimum sample-size safeguards are enforced to prevent invalid subgroup estimates. Fairness and Bias Diagnostics Fairness is evaluated through Differential Item Functioning (DIF) screening, implemented using regression-based methods that test whether item responses differ across groups after conditioning on total score. Key features include: Support for multiple grouping variables (e.g., gender, age group, site/region) Explicit minimum sample-size thresholds per group Separation of overall scale differences from item-specific bias An optional bootstrap DIF stability procedure estimates how consistently items are flagged under resampling, emphasizing robustness and reproducibility rather than binary decision rules. Interpretable AI Modeling via Item Response Theory When audit conditions are satisfied, the pipeline supports interpretable latent trait modeling via a Bayesian Item Response Theory Graded Response Model (GRM) implemented in PyMC. The model estimates: Item discrimination parameters Ordered threshold parameters Latent severity scores (θ) for respondents in the fitted subset The modeling approach is explicitly black-box free: All learned parameters are saved as transparent tables Reusable scoring utilities enable evaluation of new data without retraining Item and test information functions quantify precision across the latent continuum Model fitting is performed on configurable subsets of complete data to balance statistical fidelity and computational feasibility. Computational Design and Reproducibility The pipeline is engineered for reproducibility and controlled execution: Fully configuration-driven via config.yaml Deterministic sampling with fixed random seeds Modular codebase enabling partial or staged execution Designed to scale to large datasets through controlled subsampling Explicit separation between audit procedures and modeling procedures Due to the depth and breadth of audit components, the audit framework is distributed as a compressed archive to preserve full directory structure and methodological intent. Scope and Limitations Scope Auditing of mental-health self-report scales prior to modeling Reliability assessment with uncertainty quantification Fairness evaluation via DIF screening and stability analysis Interpretable Bayesian IRT modeling Reproducible, configuration-driven research workflows Limitations No empirical or clinical datasets are included No pre-computed outputs are provided The full pipeline was not executed end-to-end prior to release Computational demands depend on configuration and hardware The pipeline is intended as a methodological reference and research scaffold, not as a turnkey modeling solution. Ethics Statement This project does not involve human participants, identifiable data, or clinical records. It is provided solely for research and educational purposes. The pipeline is explicitly non-diagnostic and must not be used for clinical inference, individual-level decision making, or operational deployment without independent validation and appropriate ethical oversight