Temporal Categorical Modeling Using Latent Gaussian Processes

Abstract Latent Gaussian processes are widely used in latent variable models because they offer a flexible nonlinear mapping from a low-dimensional latent space to a high-dimensional space via Gaussian processes. However, these models are computationally intensive and do not scale well. In this paper, we propose a temporal categorical model that utilizes latent Gaussian processes, inducing-input approximation, and a regularization framework to model multivariate categorical processes, with and without priors on hyperparameters. We analyze the underlying properties and introduce two variational inference approaches: one based on a Monte Carlo method and the other on the delta method. We also found that latent dynamics tend to collapse into a constant zero, which hinders the reflection of dynamic information. To address this, we propose two strategies to regularize the latent dynamics for better alignment with observation dynamics: introducing a regularization term based on inducing variables and incorporating a scale prior for latent Gaussian processes. Additionally, we propose an efficient and effective stochastic variational inference technique. Finally, we demonstrate our model and inference methods using both synthetic data and real financial data.