Search Results for author:
Found 82 papers, 59 papers with code
PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction
Large Language Models (LLMs) are widely used in real-time voice chat applications, typically in combination with text-to-speech (TTS) systems to generate audio responses.
OpenThoughts: Data Recipes for Reasoning Models
Yet, there are still many open questions about the best training recipes for reasoning since state-of-the-art models often rely on proprietary datasets with little to no public information available.
Accelerating Diffusion LLMs via Adaptive Parallel Decoding
The generation speed of LLMs are bottlenecked by autoregressive decoding, where tokens are predicted sequentially one by one.
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following.
d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
Recent large language models (LLMs) have demonstrated strong reasoning capabilities that benefits from online reinforcement learning (RL).
WaterFlow: Learning Fast & Robust Watermarks using Stable Diffusion
In this work, we propose WaterFlow (WF), a fast and extremely robust approach for high fidelity visual watermarking based on a learned latent-dependent watermark.
Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflection
Currently, inference-time scaling for text-to-image diffusion models is largely limited to best-of-N sampling, where multiple images are generated per prompt and a selection model chooses the best output.
VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation
We perform human evaluation that assesses semantic adherence, physical commonsense, and grounding of physical rules in the generated videos.
Enabling Autoregressive Models to Fill In Masked Tokens
Historically, LLMs have been trained using either autoregressive (AR) or masked language modeling (MLM) objectives, with AR models gaining dominance in recent years.
MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants
To address these gaps, we present MedMax, the first large-scale multimodal biomedical instruction-tuning dataset for mixed-modal foundation models.
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
We introduce OmniFlow, a novel generative model designed for any-to-any generation tasks such as text-to-image, text-to-audio, and audio-to-image synthesis.
Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback
Our approach annotates the agent's collected experience via an asynchronous LLM server, which is then distilled into an intrinsic reward model.
PopAlign: Population-Level Alignment for Fair Text-to-Image Generation
To address this limitation, we introduce PopAlign, a novel approach for population-level preference optimization, while standard optimization would prefer entire sets of samples over others.
LICO: Large Language Models for In-Context Molecular Optimization
However, directly prompting a pretrained language model to produce predictions is not feasible in many scientific domains due to the scarcity of domain-specific data in the pretraining corpora and the challenges of articulating complex problems in natural language.
Probing the Decision Boundaries of In-context Learning in Large Language Models
In-context learning is a key paradigm in large language models (LLMs) that enables them to generalize to new tasks and domains by simply prompting these models with a few exemplars without explicit parameter updates.
VideoPhy: Evaluating Physical Commonsense for Video Generation
Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects.
TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generation
For instance, we condition the visual features of the earlier and later scenes of the generated video with the representations of the first scene description (e. g., 'a red panda climbing a tree') and second scene description (e. g., 'the red panda sleeps on the top of the tree'), respectively.
Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models
In this work, we highlight the following pitfall of prefilling: for batches containing high-varying prompt lengths, significant computation is wasted by the standard practice of padding sequences to the maximum length.
Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context.
Scaling Vision-and-Language Navigation With Offline RL
The study of vision-and-language navigation (VLN) has typically relied on expert trajectories, which may not always be available in real-world situations due to the significant effort required to collect them.
Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data
A recent architecture, Mamba, based on state space models has been shown to achieve comparable performance for modeling text sequences, while scaling linearly with the sequence length.
ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction
Thus, we propose ChaosBench, a challenging benchmark to extend the predictability range of data-driven weather emulators to S2S timescale.
InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following
We demonstrate that our system can perform a series of novel instruction-guided editing tasks.
Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
At the core of Stormer is a randomized forecasting objective that trains the model to forecast the weather dynamics over varying time intervals.
Guided Flows for Generative Modeling and Decision Making
Classifier-free guidance is a key component for enhancing the performance of conditional generative models across diverse tasks.
VideoCon: Robust Video-Language Alignment via Contrast Captions
Despite being (pre)trained on a massive amount of data, state-of-the-art video-language alignment models are not robust to semantically-plausible contrastive changes in the video captions.
ExPT: Synthetic Pretraining for Few-Shot Experimental Design
In this work, we address the more challenging yet realistic setting of few-shot experimental design, where only a few labeled data points of input designs and their corresponding values are available.
Group Preference Optimization: Few-Shot Alignment of Large Language Models
We introduce Group Preference Optimization (GPO), an alignment framework that steers language models to preferences of individual groups in a few-shot manner.
High Dimensional Causal Inference with Variational Backdoor Adjustment
For example, in medical settings, backdoor adjustment can be used to control for confounding and estimate the effectiveness of a treatment.
Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
In particular, we find that LLMs that leverage rankings data for alignment (say model X) are preferred over those that leverage ratings data (say model Y), with a rank-based evaluation protocol (is X/Y's response better than reference response?)
ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling
Modeling weather and climate is an essential endeavor to understand the near- and long-term impacts of climate change, as well as inform technology and policymaking for adaptation and mitigation efforts.
Diffusion Models for Black-Box Optimization
The goal of offline black-box optimization (BBO) is to optimize an expensive black-box function using a fixed dataset of function evaluations.
Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL
In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences.
CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning
Multimodal contrastive pretraining has been used to train multimodal representation models, such as CLIP, on large amounts of paired image-text data.
Leaving Reality to Imagination: Robust Classification via Generated Datasets
Recent research on robustness has revealed significant performance gaps between neural image classifiers trained on datasets that are similar to the test set, and those that are from a naturally shifted distribution, such as sketches, paintings, and animations of the object categories observed during training.
ClimaX: A foundation model for weather and climate
We develop and demonstrate ClimaX, a flexible and generalizable deep learning model for weather and climate science that can be trained using heterogeneous datasets spanning different variables, spatio-temporal coverage, and physical groundings.
Masked Autoencoding for Scalable and Generalizable Decision Making
We are interested in learning scalable agents for reinforcement learning that can learn from large-scale, diverse sequential data similar to current large vision and language models.
Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories
For this setting, we develop and study a simple meta-algorithmic pipeline that learns an inverse dynamics model on the labelled data to obtain proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy-labelled trajectories.
Reliable Conditioning of Behavioral Cloning for Offline Reinforcement Learning
We study CWBC in the context of RvS (Emmons et al., 2021) and Decision Transformers (Chen et al., 2021), and show that CWBC significantly boosts their performance on various benchmarks.
Matching Normalizing Flows and Probability Paths on Manifolds
Continuous Normalizing Flows (CNFs) are a class of generative models that transform a prior distribution to a model distribution by solving an ordinary differential equation (ODE).
Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling
We propose Transformer Neural Processes (TNPs), a new member of the NP family that casts uncertainty-aware meta learning as a sequence modeling problem.
Short-range forecasts of global precipitation using deep learning-augmented numerical weather prediction
We augment the output of the well-known NWP model CFSv2 with deep learning to create a hybrid model that improves short-range global precipitation at 1-, 2-, and 3-day lead times.
Generative Pretraining for Black-Box Optimization
Many problems in science and engineering involve optimizing an expensive black-box function over a high-dimensional space.
CyCLIP: Cyclic Contrastive Language-Image Pretraining
Recent advances in contrastive representation learning over paired image-text data have led to models such as CLIP that achieve state-of-the-art performance for zero-shot classification and distributional robustness.
Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning
The goal of imitation learning is to mimic expert behavior from demonstrations, without access to an explicit reward signal.
It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation
Training such agents efficiently requires automatic generation of a goal curriculum.
Online Decision Transformer
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling.
BARACK: Partially Supervised Group Robustness With Guarantees
Theoretically, we provide generalization bounds for our approach in terms of the worst-group performance, which scale with respect to both the total number of training points and the number of training points with group labels.
BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery
We propose Bayesian Causal Discovery Nets (BCD Nets), a variational inference framework for estimating a distribution over DAGs characterizing a linear-Gaussian SEM.
Frame Averaging for Invariant and Equivariant Network Design
For example, Euclidean motion invariant/equivariant graph or point cloud neural networks.
CAGE: Probing Causal Relationships in Deep Generative Models
Deep generative models excel at generating complex, high-dimensional data, often exhibiting impressive generalization beyond the training distribution.
It Takes Four to Tango: Multiagent Self Play for Automatic Curriculum Generation
We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals.
Moser Flow: Divergence-based Generative Modeling on Manifolds
MF also produces a CNF via a solution to the change-of-variable formula, however differently from other CNF methods, its model (learned) density is parameterized as the source (prior) density minus the divergence of a neural network (NN).
Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits
We introduce the "inverse bandit" problem of estimating the rewards of a multi-armed bandit instance from observing the learning process of a low-regret demonstrator.
Rotation Invariant Graph Neural Networks using Spin Convolutions
We introduce a novel approach to modeling angular information between sets of neighboring atoms in a graph neural network.
Ranked #3 on
Initial Structure to Relaxed Energy (IS2RE)
on OC20
Graph Neural Network
Initial Structure to Relaxed Energy (IS2RE)
Decision Transformer: Reinforcement Learning via Sequence Modeling
In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling.
Ranked #3 on
Offline RL
on D4RL
JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data
Such a decomposition can dynamically control the reliability of information derived from the online and offline data and the use of pretrained neural networks permits scalability to large offline datasets.
Pretrained Transformers as Universal Computation Engines
We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning -- in particular, without finetuning of the self-attention and feedforward layers of the residual blocks.
Anytime Sampling for Autoregressive Models via Ordered Autoencoding
Experimentally, we demonstrate in several image and audio generation tasks that sample quality degrades gracefully as we reduce the computational budget for sampling.
Robust Imitation via Decision-Time Planning
The goal of imitation learning is to mimic expert behavior from demonstrations, without access to an explicit reward signal.
PiRank: Scalable Learning To Rank via Differentiable Sorting
A key challenge with machine learning approaches for ranking is the gap between the performance metrics of interest and the surrogate loss functions that can be optimized with gradient-based methods.
Reset-Free Lifelong Learning with Skill-Space Planning
We propose Lifelong Skill Planning (LiSP), an algorithmic framework for non-episodic lifelong RL based on planning in an abstract space of higher-order skills.
Permutation Invariant Graph Generation via Score-Based Generative Modeling
In particular, we design a permutation equivariant, multi-channel graph neural network to model the gradient of the data distribution at the input graph (a. k. a., the score function).
Fair Generative Modeling via Weak Supervision
Real-world datasets are often biased with respect to key demographic factors such as race and gender.
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
A standard technique to correct this bias is importance sampling, where samples from the model are weighted by the likelihood ratio under model and true distributions.
AlignFlow: Cycle Consistent Learning from Multiple Domains via Normalizing Flows
Given datasets from multiple domains, a key challenge is to efficiently exploit these data sources for modeling a target domain.
Bias Correction of Learned Generative Models via Likelihood-free Importance Weighting
A standard technique to correct this bias is by importance weighting samples from the model by the likelihood ratio under the model and true distributions.
Stochastic Optimization of Sorting Networks via Continuous Relaxations
Sorting input objects is an important step in many machine learning pipelines.
Uncertainty Autoencoders: Learning Compressed Representations via Variational Information Maximization
We treat the low-dimensional projections as noisy latent representations of an autoencoder and directly learn both the acquisition (i. e., encoding) and amortized recovery (i. e., decoding) procedures.
Learning Controllable Fair Representations
Learning data representations that are transferable and are fair with respect to certain protected attributes is crucial to reducing unfair decisions while preserving the utility of the data.
Streamlining Variational Inference for Constraint Satisfaction Problems
Several algorithms for solving constraint satisfaction problems are based on survey propagation, a variational inference scheme used to obtain approximate marginal probability estimates for variable assignments.
Neural Joint Source-Channel Coding
For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes.
Modeling Sparse Deviations for Compressed Sensing using Generative Models
In compressed sensing, a small number of linear measurements can be used to reconstruct an unknown signal.
Learning Policy Representations in Multiagent Systems
Modeling agent behavior is central to understanding the emergence of complex phenomena in multiagent systems.
Variational Rejection Sampling
Learning latent variable models with stochastic variational inference is challenging when the approximate posterior is far from the true posterior, due to high variance in the gradient estimates.
Best arm identification in multi-armed bandits with delayed feedback
We propose a generalization of the best arm identification problem in stochastic multi-armed bandits (MAB) to the setting where every pull of an arm is associated with delayed feedback.
Graphite: Iterative Generative Modeling of Graphs
Graphs are a fundamental abstraction for modeling relational data.
Ranked #8 on
Link Prediction
on Citeseer
Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
Implicit models such as generative adversarial networks (GAN) often generate better samples compared to explicit models trained by maximum likelihood.
Boosted Generative Models
We propose a novel approach for using unsupervised boosting to create an ensemble of generative models, where models are trained in sequence to correct earlier mistakes.
Variational Bayes on Monte Carlo Steroids
We provide a new approach for learning latent variable models based on optimizing our new bounds on the log-likelihood.
node2vec: Scalable Feature Learning for Networks
Taken together, our work represents a new way for efficiently learning state-of-the-art task-independent representations in complex networks.
Ranked #1 on
Node Classification
on PPI
(Macro-F1 metric)
Contextual Symmetries in Probabilistic Graphical Models
We extend previous work on exploiting symmetries in the MCMC framework to the case of contextual symmetries.
