Geometry-Grounded Representation Learning and Generative Modeling: Beyond Surface Metrics

Token-autoregressive language models achieve strong likelihoods but enforce sequential decoding and do not expose an explicit prior over long-range dynamics. We study an alternative in which all sequential structure is confined to a continuous latent trajectory. We consider a variational text model with (i) a Gaussian-process prior on an ordered time grid, made time-ordered by exact Gaussian conditioning, and (ii) a fully non-autoregressive decoder that generates tokens in parallel given the latent path. We compare this latent-autoregressive GP prior to a diagonal i.i.d. ablation with matched marginals. Across very long horizons, surface stability can remain similar while latent organization diverges sharply: correlated priors induce smooth, GP-aligned trajectories, whereas diagonal latents yield rough, high-energy paths under GP whitening. We argue that token-level autoregressive perplexity is a poor proxy for evaluating parallel generators driven by correlated latent dynamics, and that geometry-consistent latent diagnostics should be treated as a primary lens on model behavior.