Stochastics, Statistics, and Probabilistic in Image Compression
Abstract: The introduction of machine learning into image compression several years ago has led to a greatly improved trade-off between bit rate and fidelity, due to better adaptation of methods to then natural image distribution. Inspired by the success of so-called generative image models, recent work increasingly focuses on better and better probabilistic modeling of the image distribution, leading to hyper-realistic images even at very low data rates. This, however, comes at the expense of a computational complexity that is several orders of magnitude higher than today’s commercial image codecs, and thus prohibitive for most practical applications. In this talk, I’ll first review how stochastic optimization has improved image compression. Then, I’ll demonstrate that by further improving our models of human perception, we can achieve a better trade-off between visual quality and bit rate, similar to “generative” compression models, while staying competitive with commercial codecs in terms of computational complexity. We achieve this by optimizing a learned image compression model for Wasserstein Distortion, a measure of texture realism rooted in statistical models of visual perception.
Recent Advances in Topic Modeling
Abstract: Topic modeling is a widely used technique in text analysis, with classical models relying on an approximate low-rank factorization of the word count matrix. In the first part of this talk, we introduce Topic-SCORE, a spectral algorithm for estimating classical topic models. It is computationally faster than other popular algorithms for topic modeling and enjoys a theoretically optimal rate.
In the second part, we extend the classical topic model to capture the distribution of word embeddings from pre-trained large language models (LLMs), enabling the incorporation of word context. We propose a flexible algorithm that integrates traditional topic modeling with nonparametric estimation. We showcase the effectiveness of our methods using MADStat, a dataset comprising 83,000 paper abstracts from statistics-related journals.
The Emergence of Generalizability and Semantic Low-Dim Subspaces in Diffusion Models
Abstract: Recent empirical studies have shown that diffusion models possess a unique reproducibility property, transiting from memorization to generalization as the number of training samples increases. This demonstrates that diffusion models can effectively learn image distributions and generate new samples. Remarkably, these models achieve this even with a small number of training samples, despite the challenge of large image dimensions, effectively circumventing the curse of dimensionality. In this work, we provide theoretical insights into this phenomenon by leveraging two key empirical observations: (i) the low intrinsic dimensionality of image datasets and (ii) the low-rank property of the denoising autoencoder in trained diffusion models. With these setups, we rigorously demonstrate that optimizing the training loss of diffusion models is equivalent to solving the canonical subspace clustering problem across the training samples. This insight has practical implications for training and controlling diffusion models. Specifically, it enables us to precisely characterize the minimal number of samples necessary for accurately learning the low-rank data support, shedding light on the phase transition from memorization to generalization. Additionally, we empirically establish a correspondence between the subspaces and the semantic representations of image data, which enables one-step, transferrable, efficient image editing. Moreover, our results have profound practical implications for training efficiency and model safety, and they also open up numerous intriguing theoretical questions for future research.
