Information Theoretic Regularization for Learning Global Features by Sequential VAE

Sequential variational autoencoders (VAEs) with global latent variable $z$ have been studied for the purpose of disentangling the global features of data, which is useful in many downstream tasks. To assist the sequential VAEs further in obtaining meaningful $z$, an auxiliary loss that maximizes the mutual information (MI) between the observation and $z$ is often employed. However, by analyzing the sequential VAEs from the information theoretic perspective, we can claim that simply maximizing the MI encourages the latent variables to have redundant information and prevents the disentanglement of global and local features. Based on this analysis, we derive a novel regularization method that makes $z$ informative while encouraging the disentanglement. Specifically, the proposed method removes redundant information by minimizing the MI between $z$ and the local features by using adversarial training. In the experiments, we trained state-space and autoregressive model variants using speech and image datasets. The results indicate that the proposed method improves the performance of the downstream classification and data generation tasks, thereby supporting our information theoretic perspective in the learning of global representations.