The Information Theory of AI Architectures: Effective Computational Capacity and Intelligence Decomposition

Version 2 (March 2026). Changes from v1:- Added Experiment 1b (teacher-student α-crossover, digits/MLP, 43/60 accuracy, p=5.33e-4)- Added Experiment 1c (GPT-2 held-out α-crossover, 7/7 accuracy, p=0.0078)- Revised experimental framing for statistical rigor- Supplementary experiment_package.zip with full reproducible code and raw outputs --- We define Effective Computational Capacity (ECC) as the generic Jacobian rank of the realization map from parameters to functions. This quantity measures the number of functionally independent parameter directions of a neural architecture, and we prove that it is the unique capacity measure satisfying three natural axioms: equivalence-invariance, direct-sum additivity, and normalization. We compute corrected ECC formulas for standard transformer components. For full multi-head attention with H heads, head dimension d_h, and value dimension d_v, ECC_Attn = H[d_h(2d − d_h) + d_v(2d − d_v)], which specializes under the standard partition d = Hd_h = Hd_v to ECC_Attn = 4d² − 2d²/H. For a width-m ReLU feed-forward network, ECC_FFN = m(2d − 1), so the standard choice m = 4d yields ECC_FFN = 8d² − 4d. Thus, for a standard transformer block, single-block ECC is close to raw parameter count: most parameters are functionally active at first order. The main bottleneck therefore is not static capacity but utilization. To formalize this point we define task intelligence by Intelligence(T) = ECC · τ_rep · τ_stat · τ_opt, where τ_rep is representational alignment, τ_stat statistical efficiency, and τ_opt optimization quality. The operational identity τ_rep = k_use / ECC shows that ECC · τ_rep is the usable task-aligned dimension. We then prove a conditional coding theorem linking excess risk to an aligned approximation dimension k, and we isolate the exact point at which a purely static bridge from ECC to performance fails. Applications include: a full ECC calculation for transformer attention, a function-space derivation of FFN capacity, a proof that standard Mamba blocks have ECC ≈ 6.25d², a geometric account of LoRA and catastrophic forgetting, a workload-aware compression and data-curation principle, a formal coverage-efficiency split for universal versus specialized models, and a no-universal-τ_rep theorem. Three experiments illustrate empirical contact: (1) α-crossover validation with teacher-student and GPT-2 held-out protocols, (2) GPT-2 subspace intervention across 8 domains with energy-gap correlation, and (3) local bridge bound verification confirming tightness of the spectral residual η_T(k).