# Generative Modeling with Denoising Auto-Encoders and Langevin Sampling

@article{Block2020GenerativeMW, title={Generative Modeling with Denoising Auto-Encoders and Langevin Sampling}, author={Adam Block and Youssef Mroueh and Alexander Rakhlin}, journal={ArXiv}, year={2020}, volume={abs/2002.00107} }

We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes.
We overcome the challenge of relying only on $L^2$ bounds on the score estimation error and provide… Expand

#### Topics from this paper

#### 9 Citations

Fast Mixing of Multi-Scale Langevin Dynamics under the Manifold Hypothesis

- Computer Science, Mathematics
- ArXiv
- 2020

This work demonstrates how the manifold hypothesis allows for the considerable reduction of mixing time, from exponential in the ambient dimension to depending only on the (much smaller) intrinsic dimension of the data. Expand

Joint Intensity-Gradient Guided Generative Modeling for Colorization

- Computer Science
- ArXiv
- 2020

The joint intensity-gradient constraint in data-fidelity term is proposed to limit the degree of freedom within generative model at the iterative colorization stage, and it is conducive to edge-preserving. Expand

Iterative Reconstruction for Low-Dose CT using Deep Gradient Priors of Generative Model

- Engineering, Computer Science
- ArXiv
- 2020

This work integrates the data-consistency as a conditional term into the iterative generative model for low-dose CT, and uses the annealing Langevin dynamics to update the trained priors with conditional scheme. Expand

Dimension-free log-Sobolev inequalities for mixture distributions

- Mathematics
- Journal of Functional Analysis
- 2021

We prove that if (Px)x∈X is a family of probability measures which satisfy the log-Sobolev inequality and whose pairwise chi-squared divergences are uniformly bounded, and μ is any mixing… Expand

Deep Generative Learning via Schrödinger Bridge

- Computer Science
- ICML
- 2021

The theoretical results guarantee that the distribution learned by the approach converges to the target distribution and indicate that the generative model via Schrödinger Bridge is comparable with state-ofthe-art GANs, suggesting a new formulation of generative learning. Expand

High-dimensional Assisted Generative Model for Color Image Restoration

- Computer Science, Engineering
- ArXiv
- 2021

The proposed unsupervised learning and iterative restoration algorithm, which involves a pre-trained generative network to obtain prior, has transparent and clear interpretation compared to other data-driven approaches. Expand

Homotopic Gradients of Generative Density Priors for MR Image Reconstruction

- Computer Science, Medicine
- IEEE transactions on medical imaging
- 2021

This work trains a more powerful noise conditional score network by forming high-dimensional tensor as the network input at the training phase and estimates the target gradients in higher-dimensional space to tackle low-dimensional manifold and low data density region issues in generative density prior. Expand

Inference and De-Noising of Non-Gaussian Particle Distribution Functions: A Generative Modeling Approach

- Computer Science, Physics
- ArXiv
- 2021

The use of normalizing flows is demonstrated to learn a smooth, tractable approximation to the noisy particle distribution function and it is demonstrated that the resulting data driven likelihood conserves relevant physics and may be extended to encapsulate the temporal evolution of the distribution function. Expand

On Energy-Based Models with Overparametrized Shallow Neural Networks

- Computer Science, Mathematics
- ICML
- 2021

This work shows that models trained in the so-called ’active’ regime provide a statistical advantage over their associated ’lazy’ or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Expand

#### References

SHOWING 1-10 OF 38 REFERENCES

Generative Modeling by Estimating Gradients of the Data Distribution

- Computer Science, Mathematics
- NeurIPS
- 2019

A new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching, which allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Expand

What regularized auto-encoders learn from the data-generating distribution

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2014

It is shown that the auto-encoder captures the score (derivative of the log-density with respect to the input) and contradicts previous interpretations of reconstruction error as an energy function. Expand

Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis

- Mathematics, Computer Science
- COLT
- 2017

The present work provides a nonasymptotic analysis in the context of non-convex learning problems, giving finite-time guarantees for SGLD to find approximate minimizers of both empirical and population risks. Expand

A Connection Between Score Matching and Denoising Autoencoders

- Mathematics, Computer Science
- Neural Computation
- 2011

A proper probabilistic model for the denoising autoencoder technique is defined, which makes it in principle possible to sample from them or rank examples by their energy, and a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives is suggested. Expand

Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond

- Computer Science, Mathematics
- NeurIPS
- 2019

The convergence rate of sampling algorithms obtained by discretizing smooth Ito diffusions exhibiting fast Wasserstein-$2$ contraction is established, based on local deviation properties of the integration scheme. Expand

Generative Adversarial Nets

- Computer Science
- NIPS
- 2014

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a… Expand

Estimation of Wasserstein distances in the Spiked Transport Model

- Mathematics
- 2019

We propose a new statistical model, the spiked transport model, which formalizes the assumption that two probability distributions differ only on a low-dimensional subspace. We study the minimax rate… Expand

Sampling from a Log-Concave Distribution with Projected Langevin Monte Carlo

- Mathematics, Computer Science
- Discret. Comput. Geom.
- 2018

We extend the Langevin Monte Carlo (LMC) algorithm to compactly supported measures via a projection step, akin to projected stochastic gradient descent (SGD). We show that (projected) LMC allows to… Expand

Sparse Regression Learning by Aggregation and Langevin Monte-Carlo

- Computer Science, Mathematics
- COLT
- 2009

A sharp PAC-Bayesian type bound for the exponentially weighted aggregate (EWA) under the expected squared empirical loss is proved and it is valid even for unbounded regression functions and the choice of the temperature parameter depends exclusively on the noise level. Expand

Theoretical guarantees for approximate sampling from smooth and log‐concave densities

- Mathematics
- 2014

Sampling from various kinds of distributions is an issue of paramount importance in statistics since it is often the key ingredient for constructing estimators, test procedures or confidence… Expand