A Blinded Multi-LLM Content-Coding Instrument: Protocol Specification and Inter-Coder Reliability Validation on an Organizational Corpus

Content coding is a reliability bottleneck of qualitative and mixed-methods organizational research: it is labor-intensive, difficult to reproduce, and vulnerable to a coder who can see the study’s hypotheses. Large language models now annotate text at or above crowd-worker quality, but naive single-model use reintroduces the failure modes intercoder-reliability protocols were built to remove – unauditable calls, hypothesis leakage, fabricated codes, and single-rater dependence. This article specifies a blinded multi-LLM content-coding instrument that imports human intercoder-reliability discipline into machine-assisted coding. Three heterogeneous models code each case independently from a neutral, source-cited evidence dossier while blinded to the hypotheses and the case’s study role; cells combine by majority-vote-or-flag with author adjudication; a model that cannot determine a cell returns uncertain rather than guess; every call is logged with an exact model pin; and the harness and the empty coding schema are committed before any datum exists. On a corpus of thirty documented organizational integration cases the instrument reached almost-perfect inter-coder agreement (Fleiss’ κ = .838), with a 2.7% flag rate concentrated entirely on auxiliary tokens. Reliability is distinguished from downstream inferential validity, which is left to future contrast studies. The harness, logs, and datasets are released as an auditable artifact. Includes zharnikov-2026bh-multi-llm-coding-instrument.yaml (Paper Spec v0.1.0) – a machine-readable specification of the paper’s claims, assumptions, and dependencies. The paper’s full machine-first bundle (the SPINE claim/dependency graph and the ONTOLOGY term module) lives in the public repository; see github.com/spectralbranding/paper-spec for the standard. This PDF is generated programmatically from that machine-first source under a research-as-repository model.