Latent variable models have been an integral part of probabilistic machine learning, ranging from simple mixture models to variational autoencoders to powerful diffusion probabilistic models at the center of recent media attention. Perhaps less well-appreciated is the intimate connection between latent variable models and data compression, and the potential of these models for advancing natural science. This talk will explore these topics. I will begin by showcasing connections between variational methods and the theory and practice of neural data compression. On the applied side, variational methods lead to machine-learned compressors of data such as images and videos and offer principled techniques for enhancing their compression performance, as well as reducing their decoding complexity. On the theory side, variational methods also provide scalable bounds on the fundamental compressibility of real-world data, such as images and particle physics data. Lastly, I will also delve into applications, where I show how deep latent variable models allow solving challenging inverse problems in weather and climate modeling tasks.