Search Results for author:
Found 97 papers, 34 papers with code
Synthesize Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
The results show that 27% of reviewers who received feedback updated their reviews, and over 12, 000 feedback suggestions from the agent were incorporated by those reviewers.
Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models
To evaluate whether contemporary LLMs are able to do so, we use the Bayesian inference framework from probability theory, which lays out the optimal way to update an agent's beliefs as it receives new information.
Graders should cheat: privileged information enables expert-level automated evaluations
But this presents a paradox: how can we trust the grader LM, which is presumably weaker than the candidate LM, to assess problems that are beyond the frontier of the capabilities of either model or both?
Statistical Downscaling via High-Dimensional Distribution Matching with Generative Models
We introduce Generative Bias Correction and Super-Resolution (GenBCSR), a two-stage probabilistic framework for statistical downscaling that overcomes the limitations of previous methods.
Diff4Steer: Steerable Diffusion Prior for Generative Music Retrieval with Semantic Guidance
Modern music retrieval systems often rely on fixed representations of user preferences, limiting their ability to capture users' diverse and uncertain retrieval needs.
Dynamical-generative downscaling of climate model ensembles
Regional high-resolution climate projections are crucial for many applications, such as agriculture, hydrology, and natural hazard risk assessment.
Generative AI for fast and accurate statistical computation of fluids
Through extensive numerical experimentation with a set of challenging fluid flows, we demonstrate that GenCFD provides an accurate approximation of relevant statistical quantities of interest while also efficiently generating high-quality realistic samples of turbulent fluid flows and ensuring excellent spectral resolution.
Rational-WENO: A lightweight, physically-consistent three-point weighted essentially non-oscillatory scheme
Conventional WENO3 methods are known to be highly dissipative at lower resolutions, introducing significant errors in the pre-asymptotic regime.
A probabilistic framework for learning non-intrusive corrections to long-time climate simulations from short-time training data
In this work we present a general strategy for training neural network models to non-intrusively correct under-resolved long-time simulations of chaotic systems.
DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systems
Learning dynamics from dissipative chaotic systems is notoriously difficult due to their inherent instability, as formalized by their positive Lyapunov exponents, which exponentially amplify errors in the learned dynamics.
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises.
The Impact of Depth on Compositional Generalization in Transformer Language Models
Because model latency is approximately linear in the number of layers, these results lead us to the recommendation that, with a given total parameter budget, transformers can be made shallower than is typical without sacrificing performance.
WeatherBench 2: A benchmark for the next generation of data-driven global weather models
WeatherBench 2 is an update to the global, medium-range (1-14 day) weather forecasting benchmark proposed by Rasp et al. (2020), designed with the aim to accelerate progress in data-driven weather modeling.
SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models
The dominant approach to representing uncertainty in weather forecasting is to generate an ensemble of forecasts.
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Empirically, we show that our dataset poses a hard challenge for large vision+language models as they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA [37], yet it only achieves 13. 0% accuracy on our dataset.
User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems
In this work, we develop a probabilistic approximation scheme for the conditional score function which provably converges to the true distribution as the noise level decreases.
Debias Coarsely, Sample Conditionally: Statistical Downscaling through Optimal Transport and Probabilistic Diffusion Models
Moreover, our procedure correctly matches the statistics of physical quantities, even when the low-frequency content of the inputs and outputs do not match, a crucial but difficult-to-satisfy assumption needed by current state-of-the-art alternatives.
V2Meow: Meowing to the Visual Beat via Video-to-Music Generation
Video-to-music generation demands both a temporally localized high-quality listening experience and globally aligned video-acoustic signatures.
Policy-Induced Self-Supervision Improves Representation Finetuning in Visual RL
Together with freezing the bottom layers, this objective results in significantly better representation than frozen, finetuned, and self-supervised alternatives on a wide range of benchmarks.
Evolve Smoothly, Fit Consistently: Learning Smooth Latent Dynamics For Advection-Dominated Systems
We present a data-driven, space-time continuous framework to learn surrogate models for complex physical systems described by advection-dominated partial differential equations.
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute
Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks.
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks.
Ranked #7 on
Question Answering
on WebQuestions
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
Then, it retrieves exemplars with outputs similar to the preliminary prediction which are used to generate a final prediction.
ALMA: Hierarchical Learning for Composite Multi-Agent Tasks
Work in MARL often focuses on solving tasks where agents interact with all other agents and entities in the environment; however, we observe that real-world tasks are often composed of several isolated instances of local agent interactions (subtasks), and each agent can meaningfully focus on one subtask to the exclusion of all else in the environment.
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing
Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling.
Possibility Before Utility: Learning And Using Hierarchical Affordances
Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our definition of hierarchical affordances in the present state, our approach is more flexible than the multitude of approaches that ground their subtask dependencies in a symbolic history.
Policy Learning and Evaluation with Randomized Quasi-Monte Carlo
Reinforcement learning constantly deals with hard integrals, for example when computing expectations in policy evaluation and policy iteration.
Co-training Transformer with Videos and Images Improves Action Recognition
We term this approach as Co-training Videos and Images for Action Recognition (CoVeR).
Ranked #6 on
Action Classification
on MiT
(using extra training data)
Improving Compositional Generalization with Latent Structure and Data Augmentation
Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization.
Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
Neural network models often generalize poorly to mismatched domains or distributions.
HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks
Many types of physics-informed neural network models have been proposed in recent years as approaches for learning solutions to differential equations.
Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
We propose to address this problem by integrating a semi-parametric representation of a large text corpus into a Transformer model as a source of factual knowledge.
Ranked #1 on
Passage Retrieval
on EntityQuestions
Visually Grounded Concept Composition
We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images.
Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?
We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding.
Ranked #4 on
Compositional Generalization (AVG)
on ReaSCAN
Compositional Generalization (AVG)
Systematic Generalization
ReadTwice: Reading Very Large Documents with Memories
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections.
Embedding Adaptation is Still Needed for Few-Shot Learning
Constructing new and more challenging tasksets is a fruitful methodology to analyse and understand few-shot classification methods.
DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections
This enables a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision.
A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus
Identifying a short segment in a long video that semantically matches a text query is a challenging task that has important application potentials in language-based video search, browsing, and navigation.
AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization
Summarization is the task of compressing source document(s) into coherent and succinct passages.
Learning to Represent Image and Text with Denotation Graph
Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images.
Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.
Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation
Many methods have been proposed to quantify the predictive uncertainty associated with the outputs of deep neural networks.
Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities; however, common patterns of behavior often emerge among these agents/entities.
BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially.
Visual Storytelling via Predicting Anchor Word Embeddings in the Stories
To narrate a sequence of images, we use the predicted anchor word embeddings and the image features as the joint input to a seq2seq model.
Ranked #15 on
Visual Storytelling
on VIST
When MAML Can Adapt Fast and How to Assist When It Cannot
On the other hand, we have just started to understand and analyze how they are able to adapt fast to new tasks.
Decoupling Adaptation from Modeling with Meta-Optimizers for Meta Learning
Meta-learning methods, most notably Model-Agnostic Meta-Learning (Finn et al, 2017) or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks.
Topic Augmented Generator for Abstractive Summarization
In this paper, we propose a new decoder where the output summary is generated by conditioning on both the input text and the latent topics of the document.
Neural Theorem Provers Do Not Learn Rules Without Exploration
Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks.
Amortized Inference of Variational Bounds for Learning Noisy-OR
Classical approaches for approximate inference depend on cleverly designed variational distributions and bounds.
Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
Solving tasks with sparse rewards is one of the most important challenges in reinforcement learning.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Synthesized Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
Hyper-parameter Tuning under a Budget Constraint
We study a budgeted hyper-parameter tuning problem, where we optimize the tuning result under a hard resource constraint.
Classifier and Exemplar Synthesis for Zero-Shot Learning
Zero-shot learning (ZSL) enables solving a task without the need to see its examples.
Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions
Many few-shot learning methods address this challenge by learning an instance embedding function from seen classes and apply the function to instances from unseen classes with limited labels.
Cross-Modal and Hierarchical Modeling of Video and Text
Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story.
Actor-Attention-Critic for Multi-Agent Reinforcement Learning
Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images
Our approach learns textual and visual representations jointly: latent visual factors couple together a skip-gram model for co-occurrence in linguistic data and a generative latent variable model for visual data.
Retrospective Encoders for Video Summarization
The key idea is to complement the discriminative losses with another loss which measures if the predicted summary preserves the same information as in the original video.
Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partner
There have been several attempts to define a plausible motivation for a chit-chat dialogue agent that can lead to engaging conversations.
Multi-Task Learning for Sequence Tagging: An Empirical Study
We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks.
Cross-Dataset Adaptation for Visual Question Answering
Analogous to domain adaptation for visual recognition, this setting is appealing when the target dataset does not have a sufficient amount of labeled data to learn an "in-domain" model.
Learning Answer Embeddings for Visual Question Answering
These properties make the approach particularly appealing for transfer learning for open-ended Visual QA, where the source dataset on which the model is learned has limited overlapping with the target dataset in the space of answers.
An Empirical Study on The Properties of Random Bases for Kernel Methods
Kernel machines as well as neural networks possess universal function approximation properties.
Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
We apply the procedures to re-construct decoy answers for two popular Visual QA datasets as well as to create a new Visual QA dataset from the Visual Genome project, resulting in the largest dataset for this task.
LabelBank: Revisiting Global Perspectives for Semantic Segmentation
We advocate that holistic inference of image concepts provides valuable information for detailed pixel labeling.
Kernel Approximation Methods for Speech Recognition
First, in order to reduce the number of random features required by kernel models, we propose a simple but effective method for feature selection.
FastMask: Segment Multi-scale Object Candidates in One Shot
Objects appear to scale differently in natural images.
Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options.
Supervised Word Mover's Distance
Accurately measuring the similarity between text documents lies at the core of many real world applications of machine learning.
Recalling Holistic Information for Semantic Segmentation
We advocate that high-recall holistic inference of image concepts provides valuable information for detailed pixel labeling.
Attention Correctness in Neural Image Captioning
Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision.
Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning
Leveraging class semantic descriptions and examples of known objects, zero-shot learning makes it possible to train a recognition model for an object class whose examples are not available.
Video Summarization with Long Short-term Memory
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots.
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild
Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only.
A Comparison between Deep Neural Nets and Kernel Acoustic Models for Speech Recognition
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition.
Summary Transfer: Exemplar-based Subset Selection for Video Summarization
Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections.
Synthesized Classifiers for Zero-Shot Learning
Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided.
Ranked #1 on
Few-Shot Image Classification
on AWA - 0-Shot
Aligning where to see and what to tell: image caption with region-based attention and scene factorization
In this paper, we propose an image caption system that exploits the parallel structures between images and sentences.
Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential.
How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets
The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Similarity Learning for High-Dimensional Sparse Data
A good measure of similarity between data points is crucial to many tasks in machine learning.
Large-Margin Determinantal Point Processes
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
Decorrelating Semantic Visual Attributes by Resisting the Urge to Share
Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.
Two-Stage Metric Learning
This induces in the input data space a new family of distance metric with unique properties.
Sparse Compositional Metric Learning
We propose a new approach for metric learning by framing it as learning a sparse combination of locally discriminative metrics that are inexpensive to generate from the training data.
A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning
We further show that the communication cost of dFW is optimal by deriving a lower-bound on the communication cost required to construct an $\epsilon$-approximate solution.
Similarity Component Analysis
Moreover, we show how SCA can be instrumental in exploratory analysis of data, where we gain insights about the data by examining patterns hidden in its latent components' local similarity values.
Reshaping Visual Datasets for Domain Adaptation
By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.
Demystifying Information-Theoretic Clustering
We propose a novel method for clustering data which is grounded in information-theoretic principles and requires no parametric assumptions.
Deformable Spatial Pyramid Matching for Fast Dense Correspondences
We introduce a fast deformable spatial pyramid (DSP) matching algorithm for computing dense pixel correspondences.
Non-linear Metric Learning
On various benchmark data sets, we demonstrate these methods not only match the current state-of-the-art in terms of kNN classification error, but in the case of χ2-LMNN, obtain best results in 19 out of 20 learning settings.
Semantic Kernel Forests from Multiple Taxonomies
When learning features for complex visual recognition problems, labeled image exemplars alone can be insufficient.
Learning a Tree of Metrics with Disjoint Visual Features
Given a hierarchical taxonomy that captures semantic similarity between the objects, we learn a corresponding tree of metrics (ToM).
Unsupervised Kernel Dimension Reduction
In this framework, kernel-based measures of independence are used to derive low-dimensional representations that maximally capture information in covariates in order to predict responses.
DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification
By using the transformed topic mixture proportions as a new representation of documents, we obtain a supervised dimensionality reduction algorithm that uncovers the latent structure in a document collection while preserving predictive power for the task of classification.
