Search Results for author: Stefano Ermon

Found 257 papers, 148 papers with code

Bridging the Gap Between f-GANs and Wasserstein GANs

1 code implementation ICML 2020 Jiaming Song, Stefano Ermon

Generative adversarial networks (GANs) variants approximately minimize divergences between the model and the data distribution using a discriminator.

Density Ratio Estimation Image Generation +1

State-Free Inference of State-Space Models: The Transfer Function Approach

1 code implementation10 May 2024 Rom N. Parnichkun, Stefano Massaroli, Alessandro Moro, Jimmy T. H. Smith, Ramin Hasani, Mathias Lechner, Qi An, Christopher Ré, Hajime Asama, Stefano Ermon, Taiji Suzuki, Atsushi Yamashita, Michael Poli

We approach designing a state-space model for deep learning applications through its dual representation, the transfer function, and uncover a highly efficient sequence parallel inference algorithm that is state-free: unlike other proposed algorithms, state-free inference does not incur any significant memory or computational cost with an increase in state size.

Language Modelling

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

1 code implementation22 Apr 2024 Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, Aviral Kumar

Our main finding is that, in general, approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i. e., employ a "negative gradient") outperform offline and maximum likelihood objectives.

Contrastive Learning Reinforcement Learning (RL)

Disentangling Length from Quality in Direct Preference Optimization

1 code implementation28 Mar 2024 Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn

A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO).


Mechanistic Design and Scaling of Hybrid Architectures

no code implementations26 Mar 2024 Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, Stefano Massaroli

The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation.

Cross-Modal Contextualized Diffusion Models for Text-Guided Visual Generation and Editing

1 code implementation26 Feb 2024 Ling Yang, Zhilong Zhang, Zhaochen Yu, Jingwei Liu, Minkai Xu, Stefano Ermon, Bin Cui

To address this issue, we propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample into forward and reverse processes.

Text-to-Image Generation Text-to-Video Editing +1

Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution

2 code implementations13 Feb 2024 Tailin Wu, Willie Neiswanger, Hongtao Zheng, Stefano Ermon, Jure Leskovec

Deep learning-based surrogate models have demonstrated remarkable advantages over classical solvers in terms of speed, often achieving speedups of 10 to 1000 times over traditional partial differential equation (PDE) solvers.

Decision Making Uncertainty Quantification

Large Language Models are Geographically Biased

1 code implementation5 Feb 2024 Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, Stefano Ermon

Initially, we demonstrate that LLMs are capable of making accurate zero-shot geospatial predictions in the form of ratings that show strong monotonic correlation with ground truth (Spearman's $\rho$ of up to 0. 89).


Segment Any Change

no code implementations2 Feb 2024 Zhuo Zheng, Yanfei Zhong, Liangpei Zhang, Stefano Ermon

Visual foundation models have achieved remarkable results in zero-shot image classification and segmentation, but zero-shot change detection remains an open problem.

Change Detection Image Classification +1

Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

1 code implementation22 Jan 2024 Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, Bin Cui

In this paper, we propose a brand new training-free text-to-image generation/editing framework, namely Recaption, Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning ability of multimodal LLMs to enhance the compositionality of text-to-image diffusion models.

Diffusion Personalization Tuning Free Large Language Model

Equivariant Graph Neural Operator for Modeling 3D Dynamics

no code implementations19 Jan 2024 Minkai Xu, Jiaqi Han, Aaron Lou, Jean Kossaifi, Arvind Ramanathan, Kamyar Azizzadenesheli, Jure Leskovec, Stefano Ermon, Anima Anandkumar

Modeling the complex three-dimensional (3D) dynamics of relational systems is an important problem in the natural sciences, with applications ranging from molecular simulations to particle mechanics.

Operator learning

Equivariant Flow Matching with Hybrid Probability Transport

1 code implementation12 Dec 2023 Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, Wei-Ying Ma

The generation of 3D molecules requires simultaneously deciding the categorical features~(atom types) and continuous features~(atom coordinates).

DiffusionSat: A Generative Foundation Model for Satellite Imagery

no code implementations6 Dec 2023 Samar Khanna, Patrick Liu, Linqi Zhou, Chenlin Meng, Robin Rombach, Marshall Burke, David Lobell, Stefano Ermon

Our method outperforms previous state-of-the-art methods for satellite image generation and is the first large-scale generative foundation model for satellite imagery.

Crop Yield Prediction Image Generation

Manifold Preserving Guided Diffusion

no code implementations28 Nov 2023 Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, Stefano Ermon

Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training.

Conditional Image Generation

DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling

1 code implementation28 Nov 2023 Linqi Zhou, Andy Shih, Chenlin Meng, Stefano Ermon

Recent methods such as Score Distillation Sampling (SDS) and Variational Score Distillation (VSD) using 2D diffusion models for text-to-3D generation have demonstrated impressive generation quality.

3D Generation Text to 3D

Diffusion Model Alignment Using Direct Preference Optimization

no code implementations21 Nov 2023 Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, Nikhil Naik

Large language models (LLMs) are fine-tuned using human comparison data with Reinforcement Learning from Human Feedback (RLHF) methods to make them better aligned with users' preferences.

Calibration by Distribution Matching: Trainable Kernel Calibration Metrics

1 code implementation NeurIPS 2023 Charles Marx, Sofian Zalouk, Stefano Ermon

Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies.

Decision Making regression

Generative Fractional Diffusion Models

no code implementations26 Oct 2023 Gabriel Nobis, Marco Aversa, Maximilian Springenberg, Michael Detzel, Stefano Ermon, Shinichi Nakajima, Roderick Murray-Smith, Sebastian Lapuschkin, Christoph Knochenhauer, Luis Oala, Wojciech Samek

We generalize the continuous time framework for score-based generative models from an underlying Brownian motion (BM) to an approximation of fractional Brownian motion (FBM).

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

1 code implementation25 Oct 2023 Aaron Lou, Chenlin Meng, Stefano Ermon

Experimentally, we test our Score Entropy Discrete Diffusion models (SEDD) on standard language modeling tasks.

Denoising Language Modelling

GeoLLM: Extracting Geospatial Knowledge from Large Language Models

1 code implementation10 Oct 2023 Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, Stefano Ermon

With GeoLLM, we observe that GPT-3. 5 outperforms Llama 2 and RoBERTa by 19% and 51% respectively, suggesting that the performance of our method scales well with the size of the model and its pretraining dataset.

The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models

no code implementations4 Oct 2023 Chenwei Wu, Li Erran Li, Stefano Ermon, Patrick Haffner, Rong Ge, Zaiwei Zhang

Compositionality is a common property in many modalities including natural languages and images, but the compositional generalization of multi-modal models is not well-understood.

Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion

1 code implementation1 Oct 2023 Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, Stefano Ermon

Consistency Models (CM) (Song et al., 2023) accelerate score-based diffusion model sampling at the cost of sample quality but lack a natural way to trade-off quality for speed.

Denoising Image Generation

SSIF: Learning Continuous Image Representation for Spatial-Spectral Super-Resolution

no code implementations30 Sep 2023 Gengchen Mai, Ni Lao, Weiwei Sun, Yuchi Ma, Jiaming Song, Chenlin Meng, Hongxu Ma, Jinmeng Rao, Ziyuan Li, Stefano Ermon

Existing digital sensors capture images at fixed spatial and spectral resolutions (e. g., RGB, multispectral, and hyperspectral images), and each combination requires bespoke machine learning models.

Spectral Super-Resolution Super-Resolution

Denoising Diffusion Bridge Models

1 code implementation29 Sep 2023 Linqi Zhou, Aaron Lou, Samar Khanna, Stefano Ermon

However, for many applications such as image editing, the model input comes from a distribution that is not random noise.

Denoising Image Generation

Sphere2Vec: A General-Purpose Location Representation Learning over a Spherical Surface for Large-Scale Geospatial Predictions

no code implementations30 Jun 2023 Gengchen Mai, Yao Xuan, Wenyun Zuo, Yutong He, Jiaming Song, Stefano Ermon, Krzysztof Janowicz, Ni Lao

So when applied to large-scale real-world GPS coordinate datasets, which require distance metric learning on the spherical surface, both types of models can fail due to the map projection distortion problem (2D) and the spherical-to-Euclidean distance approximation error (3D).

Image Classification Metric Learning +2

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

2 code implementations NeurIPS 2023 Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, Chris Ré

Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models.

4k In-Context Learning +2

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking

no code implementations8 Jun 2023 Chris Cundy, Stefano Ermon

This allows us to minimize a variety of divergences between the distribution of sequences generated by an autoregressive model and sequences from a dataset, including divergences with weight on OOD generated sequences.

Imitation Learning Text Generation

GEO-Bench: Toward Foundation Models for Earth Monitoring

1 code implementation NeurIPS 2023 Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan David Sherwin, Hannah Kerner, Björn Lütjens, Jeremy Andrew Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, Mehmet Gunturkun, Gabriel Huang, David Vazquez, Dava Newman, Yoshua Bengio, Stefano Ermon, Xiao Xiang Zhu

Recent progress in self-supervision has shown that pre-training large neural networks on vast amounts of unsupervised data can lead to substantial increases in generalization to downstream tasks.

On the Equivalence of Consistency-Type Models: Consistency Models, Consistent Diffusion Models, and Fokker-Planck Regularization

no code implementations1 Jun 2023 Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Naoki Murata, Yuki Mitsufuji, Stefano Ermon

The emergence of various notions of ``consistency'' in diffusion models has garnered considerable attention and helped achieve improved sample quality, likelihood estimation, and accelerated sampling.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

15 code implementations NeurIPS 2023 Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF).

Language Modelling reinforcement-learning +1

MADiff: Offline Multi-agent Learning with Diffusion Models

1 code implementation27 May 2023 Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, Weinan Zhang

MADiff is realized with an attention-based diffusion model to model the complex coordination among behaviors of multiple agents.

Offline RL Trajectory Prediction

Geometric Latent Diffusion Models for 3D Molecule Generation

2 code implementations2 May 2023 Minkai Xu, Alexander Powers, Ron Dror, Stefano Ermon, Jure Leskovec

Generative models, especially diffusion models (DMs), have achieved promising results for generating feature-rich geometries and advancing foundational science problems such as molecule design.

3D Molecule Generation valid

CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations

no code implementations1 May 2023 Gengchen Mai, Ni Lao, Yutong He, Jiaming Song, Stefano Ermon

To directly leverage the abundant geospatial information associated with images in pre-training, fine-tuning, and inference stages, we present Contrastive Spatial Pre-Training (CSP), a self-supervised learning framework for geo-tagged images.

Contrastive Learning Image Classification +1

MUDiff: Unified Diffusion for Complete Molecule Generation

no code implementations28 Apr 2023 Chenqing Hua, Sitao Luan, Minkai Xu, Rex Ying, Jie Fu, Stefano Ermon, Doina Precup

Our model is a promising approach for designing stable and diverse molecules and can be applied to a wide range of tasks in molecular modeling.

Drug Discovery valid

MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning

no code implementations10 Apr 2023 Arundhati Banerjee, Soham Phade, Stefano Ermon, Stephan Zheng

We then show that our model-based meta-learning approach is cost-effective in intervening on bandit agents with unseen explore-exploit strategies.


Reflected Diffusion Models

1 code implementation10 Apr 2023 Aaron Lou, Stefano Ermon

To incorporate data constraints in a principled manner, we present Reflected Diffusion Models, which instead reverse a reflected stochastic differential equation evolving on the support of the data.

 Ranked #1 on Image Generation on CIFAR-10 (Inception score metric)

Image Generation

Ideal Abstractions for Decision-Focused Learning

no code implementations29 Mar 2023 Michael Poli, Stefano Massaroli, Stefano Ermon, Bryan Wilder, Eric Horvitz

We present a methodology for formulating simplifying abstractions in machine learning systems by identifying and harnessing the utility structure of decisions.

Decision Making Management

End-to-End Diffusion Latent Optimization Improves Classifier Guidance

1 code implementation ICCV 2023 Bram Wallace, Akash Gokul, Stefano Ermon, Nikhil Naik

Classifier guidance -- using the gradients of an image classifier to steer the generations of a diffusion model -- has the potential to dramatically expand the creative control over image generation and editing.

Denoising Image Generation

GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation

1 code implementation ICCV 2023 Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, ran Xu

Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language models such as XLM-Roberta can be aligned with existing T2I models, allowing for the generation of high-quality images from captions beyond English; 2) GlueNet can align multi-modal encoders such as AudioCLIP with the Stable Diffusion model, enabling sound-to-image generation; 3) it can also upgrade the current text encoder of the latent diffusion model for challenging case generation.

Decoder Image Generation

HIVE: Harnessing Human Feedback for Instructional Visual Editing

1 code implementation16 Mar 2023 Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, ran Xu

Incorporating human feedback has been shown to be crucial to align text generated by large language models to human preferences.

Text-based Image Editing

Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching

no code implementations5 Mar 2023 Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, Stefano Ermon

In this case, a well-known issue is the distribution shift between the learned policy and the behavior policy that collects the offline data.

Continuous Control Imitation Learning

Hyena Hierarchy: Towards Larger Convolutional Language Models

6 code implementations21 Feb 2023 Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré

Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale.

2k 8k +2

Long Horizon Temperature Scaling

1 code implementation7 Feb 2023 Andy Shih, Dorsa Sadigh, Stefano Ermon

LHTS is compatible with all likelihood-based models, and optimizes for the long horizon likelihood of samples.


GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration

1 code implementation30 Jan 2023 Naoki Murata, Koichi Saito, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji, Stefano Ermon

Pre-trained diffusion models have been successfully used as priors in a variety of linear inverse problems, where the goal is to reconstruct a signal from noisy linear measurements.

Blind Image Deblurring Denoising +1

Extreme Q-Learning: MaxEnt RL without Entropy

3 code implementations5 Jan 2023 Divyansh Garg, Joey Hejna, Matthieu Geist, Stefano Ermon

Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy.

D4RL Offline RL +2

Building Coverage Estimation with Low-resolution Remote Sensing Imagery

no code implementations4 Jan 2023 Enci Liu, Chenlin Meng, Matthew Kolodner, Eun Jee Sung, Sihang Chen, Marshall Burke, David Lobell, Stefano Ermon

In this paper, we propose a method for estimating building coverage using only publicly available low-resolution satellite imagery that is more frequently updated.

Deep Latent State Space Models for Time-Series Generation

1 code implementation24 Dec 2022 Linqi Zhou, Michael Poli, Winnie Xu, Stefano Massaroli, Stefano Ermon

Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series.

Time Series Time Series Analysis +1

Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models

1 code implementation3 Nov 2022 Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, Jun-Yan Zhu

With about $1\%$-area edits, SIGE accelerates DDPM by $3. 0\times$ on NVIDIA RTX 3090 and $4. 6\times$ on Apple M1 Pro GPU, Stable Diffusion by $7. 2\times$ on 3090, and GauGAN by $5. 6\times$ on 3090 and $5. 2\times$ on M1 Pro GPU.

Concrete Score Matching: Generalized Score Matching for Discrete Data

no code implementations2 Nov 2022 Chenlin Meng, Kristy Choi, Jiaming Song, Stefano Ermon

To this end, we propose an analogous score function called the "Concrete score", a generalization of the (Stein) score for discrete settings.

Density Estimation

LMPriors: Pre-Trained Language Models as Task-Specific Priors

no code implementations22 Oct 2022 Kristy Choi, Chris Cundy, Sanjari Srivastava, Stefano Ermon

Particularly in low-data regimes, an outstanding challenge in machine learning is developing principled techniques for augmenting our models with suitable priors.

Causal Inference Common Sense Reasoning +3

FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation

1 code implementation9 Oct 2022 Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, Stefano Ermon

Score-based generative models (SGMs) learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise.


Exploration via Planning for Information about the Optimal Trajectory

1 code implementation6 Oct 2022 Viraj Mehta, Ian Char, Joseph Abbate, Rory Conlin, Mark D. Boyer, Stefano Ermon, Jeff Schneider, Willie Neiswanger

In this work, we develop a method that allows us to plan for exploration while taking both the task and the current knowledge about the dynamics into account.

Reinforcement Learning (RL)

On Distillation of Guided Diffusion Models

2 code implementations CVPR 2023 Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, Tim Salimans

For standard diffusion models trained on the pixel-space, our approach is able to generate images visually comparable to that of the original model using as few as 4 sampling steps on ImageNet 64x64 and CIFAR-10, achieving FID/IS scores comparable to that of the original model while being up to 256 times faster to sample from.

Denoising Image Generation +1

Generalizing Bayesian Optimization with Decision-theoretic Entropies

no code implementations4 Oct 2022 Willie Neiswanger, Lantao Yu, Shengjia Zhao, Chenlin Meng, Stefano Ermon

Bayesian optimization (BO) is a popular method for efficiently inferring optima of an expensive black-box function via a sequence of queries.

Bayesian Optimization Decision Making

Towards General-Purpose Representation Learning of Polygonal Geometries

1 code implementation29 Sep 2022 Gengchen Mai, Chiyu Jiang, Weiwei Sun, Rui Zhu, Yao Xuan, Ling Cai, Krzysztof Janowicz, Stefano Ermon, Ni Lao

For the spatial domain approach, we propose ResNet1D, a 1D CNN-based polygon encoder, which uses circular padding to achieve loop origin invariance on simple polygons.

Representation Learning

ButterflyFlow: Building Invertible Layers with Butterfly Matrices

no code implementations28 Sep 2022 Chenlin Meng, Linqi Zhou, Kristy Choi, Tri Dao, Stefano Ermon

Normalizing flows model complex probability distributions using maps obtained by composing invertible layers.

Density Estimation

Multipoint-BAX: A New Approach for Efficiently Tuning Particle Accelerator Emittance via Virtual Objectives

no code implementations10 Sep 2022 Sara A. Miskovich, Willie Neiswanger, William Colocho, Claudio Emma, Jacqueline Garrahan, Timothy Maxwell, Christopher Mayes, Stefano Ermon, Auralee Edelen, Daniel Ratner

Traditional black-box optimizers such as Bayesian optimization are slow and inefficient when dealing with such objectives as they must acquire the full series of measurements, but return only the emittance, with each query.

Bayesian Optimization

A General Recipe for Likelihood-free Bayesian Optimization

1 code implementation27 Jun 2022 Jiaming Song, Lantao Yu, Willie Neiswanger, Stefano Ermon

To extend BO to a broader class of models and utilities, we propose likelihood-free BO (LFBO), an approach based on likelihood-free inference.

Bayesian Optimization

Modular Conformal Calibration

no code implementations23 Jun 2022 Charles Marx, Shengjia Zhao, Willie Neiswanger, Stefano Ermon

We introduce a versatile class of algorithms for recalibration in regression that we call Modular Conformal Calibration (MCC).


Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

3 code implementations9 Jun 2022 Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, ZiRui Wang, Ziyi Wu

BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.

Common Sense Reasoning Math +1

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

9 code implementations27 May 2022 Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré

We also extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method.

16k 4k +3

Training and Inference on Any-Order Autoregressive Models the Right Way

1 code implementation26 May 2022 Andy Shih, Dorsa Sadigh, Stefano Ermon

Conditional inference on arbitrary subsets of variables is a core problem in probabilistic inference with important applications such as masked language modeling and image inpainting.

Image Inpainting Language Modelling +1

Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations

no code implementations15 Apr 2022 Michael Poli, Winnie Xu, Stefano Massaroli, Chenlin Meng, Kuno Kim, Stefano Ermon

We investigate how to leverage the representations produced by Neural Collages in various tasks, including data compression and generation.

Data Compression

Tracking Urbanization in Developing Regions with Remote Sensing Spatial-Temporal Super-Resolution

no code implementations4 Apr 2022 Yutong He, William Zhang, Chenlin Meng, Marshall Burke, David B. Lobell, Stefano Ermon

Automated tracking of urban development in areas where construction information is not available became possible with recent advancements in machine learning and remote sensing.

Image Super-Resolution Object Tracking +2

Generative Modeling Helps Weak Supervision (and Vice Versa)

1 code implementation22 Mar 2022 Benedikt Boecking, Nicholas Roberts, Willie Neiswanger, Stefano Ermon, Frederic Sala, Artur Dubrawski

The model outperforms baseline weak supervision label models on a number of multiclass image classification datasets, improves the quality of generated images, and further improves end-model performance through data augmentation with synthetic samples.

Data Augmentation Image Classification

Dual Diffusion Implicit Bridges for Image-to-Image Translation

1 code implementation16 Mar 2022 Xuan Su, Jiaming Song, Chenlin Meng, Stefano Ermon

Image translation with DDIBs relies on two diffusion models trained independently on each domain, and is a two-step process: DDIBs first obtain latent encodings for source images with the source diffusion model, and then decode such encodings using the target model to construct target images.

Image-to-Image Translation Translation

GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation

2 code implementations ICLR 2022 Minkai Xu, Lantao Yu, Yang song, Chence Shi, Stefano Ermon, Jian Tang

GeoDiff treats each atom as a particle and learns to directly reverse the diffusion process (i. e., transforming from a noise distribution to stable conformations) as a Markov chain.

Drug Discovery

LISA: Learning Interpretable Skill Abstractions from Language

1 code implementation28 Feb 2022 Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, Stefano Ermon

Learning policies that effectively utilize language instructions in complex, multi-task environments is an important problem in sequential decision-making.

Imitation Learning Quantization

Imitation Learning by Estimating Expertise of Demonstrators

1 code implementation2 Feb 2022 Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, Ramtin Pedarsani

In this work, we show that unsupervised learning over demonstrator expertise can lead to a consistent boost in the performance of imitation learning algorithms.

Continuous Control Imitation Learning

Denoising Diffusion Restoration Models

1 code implementation27 Jan 2022 Bahjat Kawar, Michael Elad, Stefano Ermon, Jiaming Song

Many interesting tasks in image restoration can be cast as linear inverse problems.

Colorization Deblurring +4

Conditional Imitation Learning for Multi-Agent Games

no code implementations5 Jan 2022 Andy Shih, Stefano Ermon, Dorsa Sadigh

In this work, we study the problem of conditional multi-agent imitation learning, where we have access to joint trajectory demonstrations at training time, and we must interact with and adapt to new partners at test time.

Imitation Learning Tensor Decomposition

IS-COUNT: Large-scale Object Counting from Satellite Images with Covariate-based Importance Sampling

1 code implementation16 Dec 2021 Chenlin Meng, Enci Liu, Willie Neiswanger, Jiaming Song, Marshall Burke, David Lobell, Stefano Ermon

We show empirically that the proposed framework achieves strong performance on estimating the number of buildings in the United States and Africa, cars in Kenya, brick kilns in Bangladesh, and swimming pools in the U. S., while requiring as few as 0. 01% of satellite images compared to an exhaustive approach.

Object Object Counting +2

Quantifying and Understanding Adversarial Examples in Discrete Input Spaces

no code implementations12 Dec 2021 Volodymyr Kuleshov, Evgenii Nikishin, Shantanu Thakoor, Tingfung Lau, Stefano Ermon

In this work, we seek to understand and extend adversarial examples across domains in which inputs are discrete, particularly across new domains, such as computational biology.

Attribute Sentiment Analysis

An Experimental Design Perspective on Model-Based Reinforcement Learning

1 code implementation9 Dec 2021 Viraj Mehta, Biswajit Paria, Jeff Schneider, Stefano Ermon, Willie Neiswanger

In particular, we leverage ideas from Bayesian optimal experimental design to guide the selection of state-action queries for efficient learning.

Continuous Control Experimental Design +3

A Unified Framework for Multi-distribution Density Ratio Estimation

no code implementations7 Dec 2021 Lantao Yu, Yujia Jin, Stefano Ermon

Binary density ratio estimation (DRE), the problem of estimating the ratio $p_1/p_2$ given their empirical samples, provides the foundation for many state-of-the-art machine learning algorithms such as contrastive representation learning and covariate shift adaptation.

Density Ratio Estimation Representation Learning

BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery

1 code implementation NeurIPS 2021 Chris Cundy, Aditya Grover, Stefano Ermon

We propose Bayesian Causal Discovery Nets (BCD Nets), a variational inference framework for estimating a distribution over DAGs characterizing a linear-Gaussian SEM.

Causal Discovery Stochastic Optimization +1

HyperSPNs: Compact and Expressive Probabilistic Circuits

1 code implementation NeurIPS 2021 Andy Shih, Dorsa Sadigh, Stefano Ermon

Probabilistic circuits (PCs) are a family of generative models which allows for the computation of exact likelihoods and marginals of its probability distributions.

Density Estimation

Reliable Decisions with Threshold Calibration

no code implementations NeurIPS 2021 Roshni Sahoo, Shengjia Zhao, Alyssa Chen, Stefano Ermon

We propose a stronger notion of calibration called threshold calibration, which is exactly the condition required to ensure that decision loss is predicted accurately for threshold decisions.


D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation

1 code implementation NeurIPS 2021 Abhishek Sinha, Jiaming Song, Chenlin Meng, Stefano Ermon

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire.

Conditional Image Generation Image Manipulation +1

Density Ratio Estimation via Infinitesimal Classification

1 code implementation22 Nov 2021 Kristy Choi, Chenlin Meng, Yang song, Stefano Ermon

We then estimate the instantaneous rate of change of the bridge distributions indexed by time (the "time score") -- a quantity defined analogously to data (Stein) scores -- with a novel time score matching objective.

Classification Density Ratio Estimation +1

Solving Inverse Problems in Medical Imaging with Score-Based Generative Models

1 code implementation NeurIPS Workshop Deep_Invers 2021 Yang song, Liyue Shen, Lei Xing, Stefano Ermon

These measurements are typically synthesized from images using a fixed physical model of the measurement process, which hinders the generalization capability of models to unknown measurement processes.

Computed Tomography (CT)

Estimating High Order Gradients of the Data Distribution by Denoising

no code implementations NeurIPS 2021 Chenlin Meng, Yang song, Wenzhe Li, Stefano Ermon

By leveraging Tweedie's formula on higher order moments, we generalize denoising score matching to estimate higher order derivatives.

Audio Synthesis Denoising +2

SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning

1 code implementation8 Nov 2021 Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, David B. Lobell, Stefano Ermon

Our goals for SustainBench are to (1) lower the barriers to entry for the machine learning community to contribute to measuring and achieving the SDGs; (2) provide standard benchmarks for evaluating machine learning models on tasks across a variety of SDGs; and (3) encourage the development of novel machine learning methods where improved model performance facilitates progress towards the SDGs.

BIG-bench Machine Learning

Sphere2Vec: Self-Supervised Location Representation Learning on Spherical Surfaces

no code implementations29 Sep 2021 Gengchen Mai, Yao Xuan, Wenyun Zuo, Yutong He, Stefano Ermon, Jiaming Song, Krzysztof Janowicz, Ni Lao

Location encoding is valuable for a multitude of tasks where both the absolute positions and local contexts (image, text, and other types of metadata) of spatial objects are needed for accurate predictions.

Image Classification Representation Learning +1

Equivariant Neural Network for Factor Graphs

no code implementations29 Sep 2021 Fan-Yun Sun, Jonathan Kuck, Hao Tang, Stefano Ermon

Several indices used in a factor graph data structure can be permuted without changing the underlying probability distribution.

Inductive Bias

An Experimental Design Perspective on Exploration in Reinforcement Learning

no code implementations ICLR 2022 Viraj Mehta, Biswajit Paria, Jeff Schneider, Willie Neiswanger, Stefano Ermon

In particular, we leverage ideas from Bayesian optimal experimental design to guide the selection of state-action queries for efficient learning.

Continuous Control Experimental Design +2

H-Entropy Search: Generalizing Bayesian Optimization with a Decision-theoretic Uncertainty Measure

no code implementations29 Sep 2021 Willie Neiswanger, Lantao Yu, Shengjia Zhao, Chenlin Meng, Stefano Ermon

For special cases of the loss and design space, we develop gradient-based methods to efficiently optimize our proposed family of acquisition functions, and demonstrate that the resulting BO procedure shows strong empirical performance on a diverse set of optimization tasks.

Bayesian Optimization

Mind Your Bits and Errors: Prioritizing the Bits that Matter in Variational Autoencoders

no code implementations29 Sep 2021 Rui Shu, Stefano Ermon

In this work, we consider the task of image generative modeling with variational autoencoders and posit that the nature of high-dimensional image data distributions poses an intrinsic challenge.

Provably Calibrated Regression Under Distribution Drift

no code implementations29 Sep 2021 Shengjia Zhao, Yusuke Tashiro, Danny Tse, Stefano Ermon

Accurate uncertainty quantification is a key building block of trustworthy machine learning systems.

regression Time Series +2

On the Opportunities and Risks of Foundation Models

2 code implementations16 Aug 2021 Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, aditi raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, Percy Liang

AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.

Transfer Learning

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

1 code implementation ICLR 2022 Chenlin Meng, Yutong He, Yang song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon

The key challenge is balancing faithfulness to the user input (e. g., hand-drawn colored strokes) and realism of the synthesized image.

Denoising Image Generation

Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration

no code implementations NeurIPS 2021 Shengjia Zhao, Michael P. Kim, Roshni Sahoo, Tengyu Ma, Stefano Ermon

In this work, we introduce a new notion -- \emph{decision calibration} -- that requires the predicted distribution and true distribution to be ``indistinguishable'' to a set of downstream decision-makers.

Decision Making

Multi-Agent Imitation Learning with Copulas

no code implementations10 Jul 2021 Hongwei Wang, Lantao Yu, Zhangjie Cao, Stefano Ermon

Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions, which is essential for understanding physical, social, and team-play systems.

Imitation Learning

CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

4 code implementations NeurIPS 2021 Yusuke Tashiro, Jiaming Song, Yang song, Stefano Ermon

In this paper, we propose Conditional Score-based Diffusion models for Imputation (CSDI), a novel time series imputation method that utilizes score-based diffusion models conditioned on observed data.

Audio Synthesis Image Generation +3

Featurized Density Ratio Estimation

1 code implementation5 Jul 2021 Kristy Choi, Madeline Liao, Stefano Ermon

Density ratio estimation serves as an important technique in the unsupervised machine learning toolbox.

Data Augmentation Density Ratio Estimation +1

IQ-Learn: Inverse soft-Q Learning for Imitation

5 code implementations NeurIPS 2021 Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Matthieu Geist, Stefano Ermon

In many sequential decision-making problems (e. g., robotics control, game playing, sequential prediction), human or expert data is available containing useful information about the task.

Atari Games Continuous Control +3

Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis

1 code implementation NeurIPS 2021 Yutong He, Dingjie Wang, Nicholas Lai, William Zhang, Chenlin Meng, Marshall Burke, David B. Lobell, Stefano Ermon

High-resolution satellite imagery has proven useful for a broad range of tasks, including measurement of global human population, local economic livelihoods, and biodiversity, among many others.

Object Counting Super-Resolution

Temporal Predictive Coding For Model-Based Planning In Latent Space

3 code implementations14 Jun 2021 Tung Nguyen, Rui Shu, Tuan Pham, Hung Bui, Stefano Ermon

High-dimensional observations are a major challenge in the application of model-based reinforcement learning (MBRL) to real-world environments.

Model-based Reinforcement Learning Representation Learning

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

2 code implementations12 Jun 2021 Abhishek Sinha, Jiaming Song, Chenlin Meng, Stefano Ermon

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire.

Conditional Image Generation Denoising +2

Improving Compositionality of Neural Networks by Decoding Representations to Inputs

no code implementations NeurIPS 2021 Mike Wu, Noah Goodman, Stefano Ermon

In traditional software programs, it is easy to trace program logic from variables back to input, apply assertion statements to block erroneous behavior, and compose programs together.

Fairness Out-of-Distribution Detection

Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information

1 code implementation19 Apr 2021 Willie Neiswanger, Ke Alexander Wang, Stefano Ermon

Given such an $\mathcal{A}$, and a prior distribution over $f$, we refer to the problem of inferring the output of $\mathcal{A}$ using $T$ evaluations as Bayesian Algorithm Execution (BAX).

Bayesian Optimization Experimental Design +1

On the Critical Role of Conventions in Adaptive Human-AI Collaboration

1 code implementation ICLR 2021 Andy Shih, Arjun Sawhney, Jovana Kondic, Stefano Ermon, Dorsa Sadigh

Humans can quickly adapt to new partners in collaborative tasks (e. g. playing basketball), because they understand which fundamental skills of the task (e. g. how to dribble, how to shoot) carry over across new partners.

Hybrid Mutual Information Lower-bound Estimators for Representation Learning

no code implementations ICLR Workshop Neural_Compression 2021 Abhishek Sinha, Jiaming Song, Stefano Ermon

We illustrate that with one set of representations, the hybrid approach is able to achieve good performance on multiple downstream tasks such as classification, reconstruction, and generation.

Representation Learning

Anytime Sampling for Autoregressive Models via Ordered Autoencoding

1 code implementation ICLR 2021 Yilun Xu, Yang song, Sahaj Garg, Linyuan Gong, Rui Shu, Aditya Grover, Stefano Ermon

Experimentally, we demonstrate in several image and audio generation tasks that sample quality degrades gracefully as we reduce the computational budget for sampling.

Audio Generation Computational Efficiency

Local Calibration: Metrics and Recalibration

no code implementations22 Feb 2021 Rachel Luo, Aadyot Bhatnagar, Yu Bai, Shengjia Zhao, Huan Wang, Caiming Xiong, Silvio Savarese, Stefano Ermon, Edward Schmerling, Marco Pavone

In this work, we propose the local calibration error (LCE) to span the gap between average and individual reliability.

Decision Making Fairness

Neural Network Compression for Noisy Storage Devices

no code implementations15 Feb 2021 Berivan Isik, Kristy Choi, Xin Zheng, Tsachy Weissman, Stefano Ermon, H. -S. Philip Wong, Armin Alaghi

Compression and efficient storage of neural network (NN) parameters is critical for applications that run on resource-constrained devices.

Neural Network Compression

Negative Data Augmentation

2 code implementations ICLR 2021 Abhishek Sinha, Kumar Ayush, Jiaming Song, Burak Uzkent, Hongxia Jin, Stefano Ermon

Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities.

Action Recognition Anomaly Detection +9

Maximum Likelihood Training of Score-Based Diffusion Models

3 code implementations NeurIPS 2021 Yang song, Conor Durkan, Iain Murray, Stefano Ermon

Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses.

Ranked #6 on Image Generation on ImageNet 32x32 (bpd metric)

Data Augmentation Image Generation

Understanding Classifiers with Generative Models

no code implementations1 Jan 2021 Laëtitia Shao, Yang song, Stefano Ermon

Although deep neural networks are effective on supervised learning tasks, they have been shown to be brittle.

Two-sample testing

H-divergence: A Decision-Theoretic Discrepancy Measure for Two Sample Tests

no code implementations1 Jan 2021 Shengjia Zhao, Abhishek Sinha, Yutong He, Aidan Perreault, Jiaming Song, Stefano Ermon

Based on ideas from decision theory, we investigate a new class of discrepancies that are based on the optimal decision loss.

Vocal Bursts Valence Prediction

Non-Markovian Predictive Coding For Planning In Latent Space

no code implementations1 Jan 2021 Tung Nguyen, Rui Shu, Tuan Pham, Hung Bui, Stefano Ermon

High-dimensional observations are a major challenge in the application of model-based reinforcement learning (MBRL) to real-world environments.

Model-based Reinforcement Learning Representation Learning

Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients

no code implementations30 Dec 2020 Chris Cundy, Rishi Desai, Stefano Ermon

We consider the task of training a policy that maximizes reward while minimizing disclosure of certain sensitive state variables through the actions.

Decision Making

PiRank: Scalable Learning To Rank via Differentiable Sorting

1 code implementation NeurIPS 2021 Robin Swezey, Aditya Grover, Bruno Charron, Stefano Ermon

A key challenge with machine learning approaches for ranking is the gap between the performance metrics of interest and the surrogate loss functions that can be optimized with gradient-based methods.


Score-Based Generative Modeling through Stochastic Differential Equations

10 code implementations ICLR 2021 Yang song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole

Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9. 89 and FID of 2. 20, a competitive likelihood of 2. 99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.

Colorization Density Estimation +2

Efficient Conditional Pre-training for Transfer Learning

no code implementations20 Nov 2020 Shuvam Chakraborty, Burak Uzkent, Kumar Ayush, Kumar Tanmay, Evan Sheehan, Stefano Ermon

Finally, we improve standard ImageNet pre-training by 1-3% by tuning available models on our subsets and pre-training on a dataset filtered from a larger scale dataset.

Transfer Learning

Geography-Aware Self-Supervised Learning

1 code implementation ICCV 2021 Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, Stefano Ermon

Contrastive learning methods have significantly narrowed the gap between supervised and unsupervised learning on computer vision tasks.

Ranked #5 on Semantic Segmentation on SpaceNet 1 (using extra training data)

Contrastive Learning Image Classification +4

Autoregressive Score Matching

no code implementations NeurIPS 2020 Chenlin Meng, Lantao Yu, Yang song, Jiaming Song, Stefano Ermon

To increase flexibility, we propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariate log-conditionals (scores), which need not be normalized.

Density Estimation Image Denoising +1

Probabilistic Circuits for Variational Inference in Discrete Graphical Models

1 code implementation NeurIPS 2020 Andy Shih, Stefano Ermon

Inference in discrete graphical models with variational methods is difficult because of the inability to re-parameterize gradients of the Evidence Lower Bound (ELBO).

Variational Inference

Imitation with Neural Density Models

no code implementations NeurIPS 2021 Kuno Kim, Akshat Jindal, Yang song, Jiaming Song, Yanan Sui, Stefano Ermon

We propose a new framework for Imitation Learning (IL) via density estimation of the expert's occupancy measure followed by Maximum Occupancy Entropy Reinforcement Learning (RL) using the density as a reward.

Density Estimation Imitation Learning +2

Denoising Diffusion Implicit Models

23 code implementations ICLR 2021 Jiaming Song, Chenlin Meng, Stefano Ermon

Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample.

Denoising Image Generation

Understanding Classifier Mistakes with Generative Models

no code implementations5 Oct 2020 Laëtitia Shao, Yang song, Stefano Ermon

From this observation, we develop a detection criteria for samples on which a classifier is likely to fail at test time.

Two-sample testing

Privacy Preserving Recalibration under Domain Shift

no code implementations21 Aug 2020 Rachel Luo, Shengjia Zhao, Jiaming Song, Jonathan Kuck, Stefano Ermon, Silvio Savarese

In an extensive empirical study, we find that our algorithm improves calibration on domain-shift benchmarks under the constraints of differential privacy.

Privacy Preserving

Multi-label Contrastive Predictive Coding

no code implementations NeurIPS 2020 Jiaming Song, Stefano Ermon

We demonstrate that the proposed approach is able to lead to better mutual information estimation, gain empirical improvements in unsupervised representation learning, and beat a current state-of-the-art knowledge distillation method over 10 out of 13 tasks.