Corpus-Derived Structural Bias as an Inductive Prior in Single-Corpus Language Modeling

We examine whether static structural information derived directly from a training corpus canserve as a useful inductive prior for language models trained in a single-corpus, low-resource regime.Using the Sanskrit Mahābhārata as a controlled case study, we construct a token-level co-occurrencegraph from the corpus andextract unsupervised structural signals via community detection and graphcentrality analysis. These signals are injected as a fixed, additive bias at the embedding level duringmodel training, without modifying model architecture or learning dynamics.Weevaluate this approach across two architectures: a Standard Transformer and a stronger baseline, BDH(BabyDragonHatchling). Allmodelsaretrainedunderidenticalconditions, differingonlyin the presence or absence of the corpus-derived structural bias. Because standard automated metricssuch as perplexity are insufficiently discriminative in this setting, we employ a structured, comparative evaluation protocol using a fixed large-language-model judge, emphasizing narrative stability,stylistic coherence, and grammatical integrity. Explicit memorization analysis is conducted to ruleout trivial copying effects.Across architectures, models augmented with corpus-derived structural bias exhibit consistentlyimproved judged stability and coherence, as well as faster qualitative improvement over training,without increased memorization. These results suggest that simple, static structure extracted from acorpus itself can provide a meaningful inductive prior in single-corpus language modeling, offering alightweight alternative to external supervision or architectural modification in low-resource settings