Distillation of a tractable model from the VQ-VAE

Deep generative models with a discrete latent space, such as the Vector-Quantized Variational Autoencoder (VQ-VAE), offer excellent data generation capabilities, but-due to the large size of their latent space-their probabilistic inference is deemed intractable.We demonstrate that the VQ-VAE can be distilled into a tractable model by selecting a subset of latent variables with high probability under the prior.We frame the distilled model as a probabilistic circuit, and show that it preserves the expressiveness of the VQ-VAE while providing tractable probabilistic inference.Experiments illustrate competitive performance in both density estimation and conditional generation tasks, challenging the view of the VQ-VAE as an inherently intractable model.