Search Results for author:
Found 86 papers, 29 papers with code
Synthesize Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises.
The Impact of Depth and Width on Transformer Language Model Generalization
To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways.
WeatherBench 2: A benchmark for the next generation of data-driven global weather models
WeatherBench 2 is an update to the global, medium-range (1-14 day) weather forecasting benchmark proposed by Rasp et al. (2020), designed with the aim to accelerate progress in data-driven weather modeling.
SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models
The dominant approach to representing uncertainty in weather forecasting is to generate an ensemble of forecasts.
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Empirically, we show that our dataset poses a hard challenge for large vision+language models as they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA [37], yet it only achieves 13. 0% accuracy on our dataset.
User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems
In this work, we develop a probabilistic approximation scheme for the conditional score function which provably converges to the true distribution as the noise level decreases.
V2Meow: Meowing to the Visual Beat via Music Generation
Generating high quality music that complements the visual content of a video is a challenging task.
Policy-Induced Self-Supervision Improves Representation Finetuning in Visual RL
Together with freezing the bottom layers, this objective results in significantly better representation than frozen, finetuned, and self-supervised alternatives on a wide range of benchmarks.
Evolve Smoothly, Fit Consistently: Learning Smooth Latent Dynamics For Advection-Dominated Systems
We present a data-driven, space-time continuous framework to learn surrogate models for complex physical systems described by advection-dominated partial differential equations.
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute
Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks.
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks.
Ranked #3 on
Question Answering
on WebQuestions
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
Then, it retrieves exemplars with outputs similar to the preliminary prediction which are used to generate a final prediction.
ALMA: Hierarchical Learning for Composite Multi-Agent Tasks
Work in MARL often focuses on solving tasks where agents interact with all other agents and entities in the environment; however, we observe that real-world tasks are often composed of several isolated instances of local agent interactions (subtasks), and each agent can meaningfully focus on one subtask to the exclusion of all else in the environment.
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing
Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling.
Possibility Before Utility: Learning And Using Hierarchical Affordances
Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our definition of hierarchical affordances in the present state, our approach is more flexible than the multitude of approaches that ground their subtask dependencies in a symbolic history.
Policy Learning and Evaluation with Randomized Quasi-Monte Carlo
Reinforcement learning constantly deals with hard integrals, for example when computing expectations in policy evaluation and policy iteration.
Improving Compositional Generalization with Latent Structure and Data Augmentation
Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization.
Co-training Transformer with Videos and Images Improves Action Recognition
We term this approach as Co-training Videos and Images for Action Recognition (CoVeR).
Ranked #6 on
Action Classification
on Moments in Time
(using extra training data)
Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
Neural network models often generalize poorly to mismatched domains or distributions.
HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks
Many types of physics-informed neural network models have been proposed in recent years as approaches for learning solutions to differential equations.
Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
We propose to address this problem by integrating a semi-parametric representation of a large text corpus into a Transformer model as a source of factual knowledge.
Ranked #1 on
Passage Retrieval
on EntityQuestions
Visually Grounded Concept Composition
We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images.
Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?
We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding.
ReadTwice: Reading Very Large Documents with Memories
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections.
Embedding Adaptation is Still Needed for Few-Shot Learning
Constructing new and more challenging tasksets is a fruitful methodology to analyse and understand few-shot classification methods.
DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections
This enables a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision.
A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus
Identifying a short segment in a long video that semantically matches a text query is a challenging task that has important application potentials in language-based video search, browsing, and navigation.
AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization
Summarization is the task of compressing source document(s) into coherent and succinct passages.
Learning to Represent Image and Text with Denotation Graph
Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images.
Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.
Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation
Many methods have been proposed to quantify the predictive uncertainty associated with the outputs of deep neural networks.
Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities; however, common patterns of behavior often emerge among these agents/entities.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially.
Visual Storytelling via Predicting Anchor Word Embeddings in the Stories
To narrate a sequence of images, we use the predicted anchor word embeddings and the image features as the joint input to a seq2seq model.
Ranked #15 on
Visual Storytelling
on VIST
When MAML Can Adapt Fast and How to Assist When It Cannot
On the other hand, we have just started to understand and analyze how they are able to adapt fast to new tasks.
Decoupling Adaptation from Modeling with Meta-Optimizers for Meta Learning
Meta-learning methods, most notably Model-Agnostic Meta-Learning (Finn et al, 2017) or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks.
Topic Augmented Generator for Abstractive Summarization
In this paper, we propose a new decoder where the output summary is generated by conditioning on both the input text and the latent topics of the document.
Neural Theorem Provers Do Not Learn Rules Without Exploration
Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks.
Amortized Inference of Variational Bounds for Learning Noisy-OR
Classical approaches for approximate inference depend on cleverly designed variational distributions and bounds.
Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
Solving tasks with sparse rewards is one of the most important challenges in reinforcement learning.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Synthesized Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
Hyper-parameter Tuning under a Budget Constraint
We study a budgeted hyper-parameter tuning problem, where we optimize the tuning result under a hard resource constraint.
Classifier and Exemplar Synthesis for Zero-Shot Learning
Zero-shot learning (ZSL) enables solving a task without the need to see its examples.
Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions
Many few-shot learning methods address this challenge by learning an instance embedding function from seen classes and apply the function to instances from unseen classes with limited labels.
Cross-Modal and Hierarchical Modeling of Video and Text
Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story.
Actor-Attention-Critic for Multi-Agent Reinforcement Learning
Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images
Our approach learns textual and visual representations jointly: latent visual factors couple together a skip-gram model for co-occurrence in linguistic data and a generative latent variable model for visual data.
Retrospective Encoders for Video Summarization
The key idea is to complement the discriminative losses with another loss which measures if the predicted summary preserves the same information as in the original video.
Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partner
There have been several attempts to define a plausible motivation for a chit-chat dialogue agent that can lead to engaging conversations.
Multi-Task Learning for Sequence Tagging: An Empirical Study
We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks.
Learning Answer Embeddings for Visual Question Answering
These properties make the approach particularly appealing for transfer learning for open-ended Visual QA, where the source dataset on which the model is learned has limited overlapping with the target dataset in the space of answers.
Cross-Dataset Adaptation for Visual Question Answering
Analogous to domain adaptation for visual recognition, this setting is appealing when the target dataset does not have a sufficient amount of labeled data to learn an "in-domain" model.
An Empirical Study on The Properties of Random Bases for Kernel Methods
Kernel machines as well as neural networks possess universal function approximation properties.
Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
We apply the procedures to re-construct decoy answers for two popular Visual QA datasets as well as to create a new Visual QA dataset from the Visual Genome project, resulting in the largest dataset for this task.
LabelBank: Revisiting Global Perspectives for Semantic Segmentation
We advocate that holistic inference of image concepts provides valuable information for detailed pixel labeling.
Kernel Approximation Methods for Speech Recognition
First, in order to reduce the number of random features required by kernel models, we propose a simple but effective method for feature selection.
FastMask: Segment Multi-scale Object Candidates in One Shot
Objects appear to scale differently in natural images.
Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options.
Supervised Word Mover's Distance
Accurately measuring the similarity between text documents lies at the core of many real world applications of machine learning.
Recalling Holistic Information for Semantic Segmentation
We advocate that high-recall holistic inference of image concepts provides valuable information for detailed pixel labeling.
Attention Correctness in Neural Image Captioning
Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision.
Video Summarization with Long Short-term Memory
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots.
Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning
Leveraging class semantic descriptions and examples of known objects, zero-shot learning makes it possible to train a recognition model for an object class whose examples are not available.
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild
Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only.
A Comparison between Deep Neural Nets and Kernel Acoustic Models for Speech Recognition
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition.
Summary Transfer: Exemplar-based Subset Selection for Video Summarization
Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections.
Synthesized Classifiers for Zero-Shot Learning
Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided.
Ranked #1 on
Few-Shot Image Classification
on AWA - 0-Shot
Aligning where to see and what to tell: image caption with region-based attention and scene factorization
In this paper, we propose an image caption system that exploits the parallel structures between images and sentences.
Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential.
How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets
The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Similarity Learning for High-Dimensional Sparse Data
A good measure of similarity between data points is crucial to many tasks in machine learning.
Large-Margin Determinantal Point Processes
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
Decorrelating Semantic Visual Attributes by Resisting the Urge to Share
Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.
Two-Stage Metric Learning
This induces in the input data space a new family of distance metric with unique properties.
Sparse Compositional Metric Learning
We propose a new approach for metric learning by framing it as learning a sparse combination of locally discriminative metrics that are inexpensive to generate from the training data.
A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning
We further show that the communication cost of dFW is optimal by deriving a lower-bound on the communication cost required to construct an $\epsilon$-approximate solution.
Reshaping Visual Datasets for Domain Adaptation
By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.
Similarity Component Analysis
Moreover, we show how SCA can be instrumental in exploratory analysis of data, where we gain insights about the data by examining patterns hidden in its latent components' local similarity values.
Demystifying Information-Theoretic Clustering
We propose a novel method for clustering data which is grounded in information-theoretic principles and requires no parametric assumptions.
Deformable Spatial Pyramid Matching for Fast Dense Correspondences
We introduce a fast deformable spatial pyramid (DSP) matching algorithm for computing dense pixel correspondences.
Semantic Kernel Forests from Multiple Taxonomies
When learning features for complex visual recognition problems, labeled image exemplars alone can be insufficient.
Non-linear Metric Learning
On various benchmark data sets, we demonstrate these methods not only match the current state-of-the-art in terms of kNN classification error, but in the case of χ2-LMNN, obtain best results in 19 out of 20 learning settings.
Learning a Tree of Metrics with Disjoint Visual Features
Given a hierarchical taxonomy that captures semantic similarity between the objects, we learn a corresponding tree of metrics (ToM).
Unsupervised Kernel Dimension Reduction
In this framework, kernel-based measures of independence are used to derive low-dimensional representations that maximally capture information in covariates in order to predict responses.
DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification
By using the transformed topic mixture proportions as a new representation of documents, we obtain a supervised dimensionality reduction algorithm that uncovers the latent structure in a document collection while preserving predictive power for the task of classification.
