Scaling, Information, and Innovation: A Theory of Maximum Feasible Innovation in Large Pretrained Models

We study the relationship between (i) the effective training information available to a learner, (ii) the learner's encoding/representational capacity, and (iii) the expected number and quality of feasible innovations (defined rigorously below) that the learner can produce. We provide an information-theoretic framework that formalizes "innovation" as a measurable information gain combined with a value predicate, derive upper bounds on the total achievable useful innovation as a function of dataset information and model encoding capacity, and prove sample-complexity lower bounds for reliably producing a target number of independent valuable innovations. We further analyze conditions under which non-linear/threshold phenomena ("emergent abilities") can occur, and characterize how training regimes (pretraining vs. post-training such as RLHF or self-play) affect the feasible-innovation frontier. Our results reconcile empirical scaling-law observations (power-law loss scaling) with information-theoretic limits, and identify clear hypotheses for empirical verification.