Search Results for author:
Found 70 papers, 34 papers with code
ProtComposer: Compositional Protein Structure Generation with 3D Ellipsoids
We develop ProtComposer to generate protein structures conditioned on spatial protein layouts that are specified via a set of 3D ellipsoids capturing substructure shapes and semantics.
Proteina: Scaling Flow-based Protein Structure Generative Models
Here, we develop Proteina, a new large-scale flow-based protein backbone generator that utilizes hierarchical fold class labels for conditioning and relies on a tailored scalable transformer architecture with up to 5x as many parameters as previous models.
One-step Diffusion Models with $f$-Divergence Distribution Matching
This weighting function naturally emphasizes samples with higher density in the teacher distribution, when using a less mode-seeking divergence.
A2SB: Audio-to-Audio Schrodinger Bridges
Audio in the real world may be perturbed due to numerous factors, causing the audio quality to be degraded.
BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations
In addition, we introduce a learnable module to interpolate text embeddings so that users can control semantics in specific frames and obtain smooth object transitions.
GenMol: A Drug Discovery Generalist with Discrete Diffusion
Drug discovery is a complex process that involves multiple scenarios and stages, such as fragment-constrained molecule generation, hit generation and lead optimization.
Molecule Generation with Fragment Retrieval Augmentation
Fragment-based drug discovery, in which molecular fragments are assembled into new molecules with desirable biochemical properties, has achieved great success.
Energy-Based Diffusion Language Models for Text Generation
Unfortunately, these models still underperform the autoregressive counterparts, with the performance gap increasing when reducing the number of sampling steps.
Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
This perspective allows us to train function space diffusion models only on images and utilize them to solve temporally correlated inverse problems.
Truncated Consistency Models
Consistency models have recently been introduced to accelerate sampling from diffusion models by directly predicting the solution (i. e., data) of the probability flow ODE (PF ODE) from initial noise.
Ranked #13 on
Image Generation
on ImageNet 64x64
Heavy-Tailed Diffusion Models
We address this by repurposing the diffusion framework for heavy-tail estimation using multivariate Student-t distributions.
Stochastic Flow Matching for Resolving Small-Scale Physics
Conditioning diffusion and flow models have proven effective for super-resolving small-scale details in natural images. However, in physical sciences such as weather, super-resolving small-scale details poses significant challenges due to: (i) misalignment between input and output distributions (i. e., solutions to distinct partial differential equations (PDEs) follow different trajectories), (ii) multi-scale dynamics, deterministic dynamics at large scales vs. stochastic at small scales, and (iii) limited data, increasing the risk of overfitting.
Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models
To address this, recent approaches have incorporated human preference datasets to fine-tune T2I models or to optimize reward functions that capture these preferences.
Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling
Storm-scale convection-allowing models (CAMs) are an important tool for predicting the evolution of thunderstorms and mesoscale convective systems that result in damaging extreme weather.
DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
The discrete latents significantly simplify learning the DM's complex noise-to-data mapping by reducing the curvature of the DM's generative ODE.
Ranked #3 on
Image Generation
on ImageNet 128x128
Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization
AliDiff shifts the target-conditioned chemical distribution towards regions with higher binding affinity and structural rationality, specified by user-defined reward functions, via the preference optimization approach.
CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Recently video diffusion models have emerged as expressive generative tools for high-quality video content creation readily available to general users.
DiffUHaul: A Training-Free Method for Object Dragging in Images
In this work, we propose a training-free method, dubbed DiffUHaul, that harnesses the spatial understanding of a localized text-to-image model, for the object dragging task.
Compositional Text-to-Image Generation with Dense Blob Representations
To leverage the compositionality of large language models (LLMs), we introduce a new in-context learning approach to generate blob representations from text prompts.
AGG: Amortized Generative 3D Gaussians for Single Image to 3D
To overcome these challenges, we introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image, eliminating the need for per-instance optimization.
DiffiT: Diffusion Vision Transformers for Image Generation
In this paper, we study the effectiveness of ViTs in diffusion-based generative learning and propose a new model denoted as Diffusion Vision Transformers (DiffiT).
Ranked #31 on
Image Generation
on ImageNet 256x256
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences.
Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling
The model is trained to predict 2km data from a regional weather model over Taiwan, conditioned on a 25km global reanalysis.
Fast Training of Diffusion Models with Masked Transformers
For masked training, we introduce an asymmetric encoder-decoder architecture consisting of a transformer encoder that operates only on unmasked patches and a lightweight transformer decoder on full patches.
A Variational Perspective on Solving Inverse Problems with Diffusion Models
To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution.
Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models
Our Landmark DEQ (LDEQ) achieves state-of-the-art performance on the challenging WFLW facial landmark dataset, reaching $3. 92$ NME with fewer parameters and a training memory cost of $\mathcal{O}(1)$ in the number of recurrent modules.
Ranked #2 on
Face Alignment
on WFLW
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks.
Ranked #2 on
Open-World Instance Segmentation
on UVO
(using extra training data)
Open Vocabulary Panoptic Segmentation
Open Vocabulary Semantic Segmentation
+4
Score-based Diffusion Models in Function Space
They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising.
I$^2$SB: Image-to-Image Schrödinger Bridge
We propose Image-to-Image Schr\"odinger Bridge (I$^2$SB), a new class of conditional diffusion models that directly learn the nonlinear diffusion processes between two given distributions.
PhysDiff: Physics-Guided Human Motion Diffusion Model
Specifically, we propose a physics-based motion projection module that uses motion imitation in a physics simulator to project the denoised motion of a diffusion step to a physically-plausible motion.
Fast Sampling of Diffusion Models via Operator Learning
Diffusion models have found widespread adoption in various areas.
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
Therefore, in contrast to existing works, we propose to train an ensemble of text-to-image diffusion models specialized for different synthesis stages.
Ranked #14 on
Text-to-Image Generation
on MS COCO
Differentially Private Diffusion Models
While modern machine learning models rely on increasingly large training datasets, data is often limited in privacy-sensitive domains.
LION: Latent Point Diffusion Models for 3D Shape Generation
To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes.
Ranked #1 on
Point Cloud Generation
on ShapeNet Airplane
GENIE: Higher-Order Denoising Diffusion Solvers
Synthesis amounts to solving a differential equation (DE) defined by the learnt model.
Ranked #5 on
Image Generation
on AFHQV2
State-specific protein-ligand complex structure prediction with a multi-scale deep generative model
The binding complexes formed by proteins and small molecule ligands are ubiquitous and critical to life.
Diffusion Models for Adversarial Purification
Adversarial purification refers to a class of defense methods that remove adversarial perturbations using a generative model.
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
To the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to an extent that allows them to be applied to real-world applications inexpensively.
Ranked #9 on
Image Generation
on CelebA-HQ 256x256
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion
SGMs rely on a diffusion process that gradually perturbs the data towards a tractable distribution, while the generative model learns to denoise.
AdaViT: Adaptive Tokens for Efficient Vision Transformer
A-ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
Ranked #34 on
Efficient ViTs
on ImageNet-1K (with DeiT-S)
Don’t Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
Generative models trained with privacy constraints on private data can sidestep this challenge, providing indirect access to private data instead.
Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
Generative models trained with privacy constraints on private data can sidestep this challenge, providing indirect access to private data instead.
Controllable and Compositional Generation with Latent-Space Energy-Based Models
In compositional generation, our method excels at zero-shot generation of unseen attribute combinations.
Hardware-Aware Network Transformation
In the second phase, it solves the combinatorial selection of efficient operations using a novel constrained integer linear optimization approach.
LANA: Latency Aware Network Acceleration
We analyze three popular network architectures: EfficientNetV1, EfficientNetV2 and ResNeST, and achieve accuracy improvement for all models (up to $3. 0\%$) when compressing larger models to the latency level of smaller models.
Score-based Generative Modeling in Latent Space
Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling.
Ranked #4 on
Image Generation
on CelebA 256x256
Shifting Transformation Learning for Out-of-Distribution Detection
In this paper, we propose a simple framework that leverages a shifting transformation learning setting for learning multiple shifted representations of the training set for improved OOD detection.
Ranked #9 on
Anomaly Detection
on Unlabeled CIFAR-10 vs CIFAR-100
See through Gradients: Image Batch Recovery via GradInversion
In this work, we introduce GradInversion, using which input images from a larger batch (8 - 48 images) can also be recovered for large networks such as ResNets (50 layers), on complex datasets such as ImageNet (1000 classes, 224x224 px).
A Contrastive Learning Approach for Training Variational Autoencoder Priors
To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior.
Ranked #6 on
Image Generation
on CelebA 256x256
(FID metric)
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples.
Ranked #1 on
Image Generation
on Stacked MNIST
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors
To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior.
NVAE: A Deep Hierarchical Variational Autoencoder
For example, on CIFAR-10, NVAE pushes the state-of-the-art from 2. 98 to 2. 91 bits per dimension, and it produces high-quality images on CelebA HQ.
Ranked #3 on
Image Generation
on FFHQ 256 x 256
(bits/dimension metric)
Contrastive Learning for Weakly Supervised Phrase Grounding
Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions.
On the distance between two neural networks and the stability of learning
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions.
UNAS: Differentiable Architecture Search Meets Reinforcement Learning
Our framework brings the best of both worlds, and it enables us to search for architectures with both differentiable and non-differentiable criteria in one unified framework while maintaining a low search cost.
A Robust Learning Approach to Domain Adaptive Object Detection
To adapt to the domain shift, the model is trained on the target domain using a set of noisy object bounding boxes that are obtained by a detection model trained only in the source domain.
Undirected Graphical Models as Approximate Posteriors
We extend the class of posterior models that may be learned by using undirected graphical models.
Semi-Supervised Semantic Image Segmentation with Self-correcting Networks
Building a large image dataset with high-quality object masks for semantic segmentation is costly and time consuming.
Improved Gradient-Based Optimization Over Discrete Distributions
In many applications we seek to maximize an expectation with respect to a distribution over discrete variables.
Improved Gradient Estimators for Stochastic Discrete Variables
In many applications we seek to optimize an expectation with respect to a distribution over discrete variables.
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors
Experiments on the MNIST and OMNIGLOT datasets show that these relaxations outperform previous discrete VAEs with Boltzmann priors.
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult.
Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks
Collecting large training datasets, annotated with high-quality labels, is costly and time-consuming.
Hierarchical Deep Temporal Models for Group Activity Recognition
In order to model both person-level and group-level dynamics, we present a 2-stage deep temporal model for the group activity recognition problem.
A Hierarchical Deep Temporal Model for Group Activity Recognition
In group activity recognition, the temporal dynamics of the whole activity can be inferred based on the dynamics of the individual people representing the activity.
Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition
As a concrete example, group activity recognition involves the interactions and relative spatial relations of a set of people in a scene.
Ranked #6 on
Group Activity Recognition
on Collective Activity
Discovering Human Interactions in Videos with Limited Data Labeling
We present a novel approach for discovering human interactions in videos.
Visual Recognition by Counting Instances: A Multi-Instance Cardinality Potential Kernel
Many visual recognition problems can be approached by counting instances.
Latent Maximum Margin Clustering
We present a maximum margin framework that clusters data using latent variables.
Kernel Latent SVM for Visual Recognition
Latent SVMs (LSVMs) are a class of powerful tools that have been successfully applied to many applications in computer vision.
