Search Results for author:
Found 227 papers, 120 papers with code
Answering Ambiguous Questions with a Database of Questions, Answers, and Revisions
Many open-domain questions are under-specified and thus have multiple possible answers, each of which is correct under a different interpretation of the question.
Contrastive Example-Based Control
In this paper, we propose a method for offline, example-based control that learns an implicit model of multi-step transitions, rather than a reward function.
A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
One-step methods perform regularization by doing just a single step of policy improvement, while critic regularization methods do many steps of policy improvement with a regularized objective.
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data.
Localized Text-to-Image Generation for Free via Cross Attention Control
Despite the tremendous success in text-to-image generative models, localized text-to-image generation (that is, generating objects or features at specific locations in an image while maintaining a consistent overall generation) still requires either explicit training or substantial additional inference time.
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks?
Multimodal Fusion Interactions: A Study of Human and Automatic Quantification
Multimodal fusion of multiple heterogeneous and interconnected signals is a fundamental challenge in almost all multimodal problems and applications.
Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
We propose two lower bounds based on the amount of shared information between modalities and the disagreement between separately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings.
Stabilizing Contrastive RL: Techniques for Offline Goal Reaching
In the same way that the computer vision (CV) and natural language processing (NLP) communities have developed self-supervised methods, reinforcement learning (RL) can be cast as a self-supervised problem: learning to reach any goal, without requiring human-specified rewards or labels.
Generating Images with Multimodal Language Models
This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs.
Imitating Task and Motion Planning with Visuomotor Transformers
In this work, we show that the combination of large-scale datasets generated by TAMP supervisors and flexible Transformer models to fit them is a powerful paradigm for robot manipulation.
SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning
Finally, we show the potential of games as a test bed for LLMs.
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
We propose the Plan, Eliminate, and Track (PET) framework.
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities.
Effective Data Augmentation With Diffusion Models
Data augmentation is one of the most prevalent tools in deep learning, underpinning many recent advances, including those from classification, generative models, and representation learning.
Grounding Language Models to Images for Multimodal Inputs and Outputs
We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images.
Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
To tackle this problem, we show that relation alignment can be enforced by encouraging the directed language attention from 'mug' to 'grass' (capturing the semantic relation 'in') to match the directed visual attention from the mug to the grass.
Ranked #5 on
Visual Reasoning
on Winoground
Self-Supervised Object Goal Navigation with In-Situ Finetuning
In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions.
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more.
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue
We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress.
Paraphrasing Is All You Need for Novel Object Captioning
Since no ground truth captions are available for novel object images during training, our P2C leverages cross-modality (image-text) association modules to ensure the above caption characteristics can be properly preserved.
Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective
In this work, we propose a single objective which jointly optimizes a latent-space model and policy to achieve high returns while remaining self-consistent.
Graph Generative Model for Benchmarking Graph Neural Networks
As the field of Graph Neural Networks (GNN) continues to grow, it experiences a corresponding increase in the need for large, real-world datasets to train and test new GNN models on challenging, realistic problems.
MultiViz: Towards Visualizing and Understanding Multimodal Models
How can we visualize the internal modeling of multimodal interactions in these models?
A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Search
Physically rearranging objects is an important capability for embodied agents.
Contrastive Learning as Goal-Conditioned Reinforcement Learning
While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable and instead equip RL algorithms with additional representation learning parts (e. g., auxiliary losses, data augmentation).
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Imitating Past Successes can be Very Suboptimal
Prior work has proposed a simple strategy for reinforcement learning (RL): label experience with the outcomes achieved in that experience, and then imitate the relabeled experience.
Reasoning over Logically Interacted Conditions for Question Answering
Even more challenging, we only provide evidences for a subset of the conditions, so some questions may not have deterministic answers.
PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
Our paper takes a step towards real-world physical commonsense reasoning by contributing PACS: the first audiovisual benchmark annotated for physical commonsense attributes.
Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
We propose a zero-shot transfer learning module for HGNNs called a Knowledge Transfer Network (KTN) that transfers knowledge from label-abundant node types to zero-labeled node types through rich relational information given in the HG.
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making.
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots.
Learning Weakly-Supervised Contrastive Representations
The first stage is to cluster data according to its auxiliary information.
Conditional Contrastive Learning with Kernel
Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on specific variables.
SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency
The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model.
Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
An alternate but important component to consider improving is the interface of the RL algorithm with the robot.
C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the field.
ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers
In addition to conditional answers, the dataset also features: (1) long context documents with information that is related in logically complex ways; (2) multi-hop questions that require compositional logical reasoning; (3) a combination of extractive questions, yes/no questions, questions with multiple answers, and not-answerable questions; (4) questions asked without knowing the answers.
FILM: Following Instructions in Language with Modular Methods
In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal.
Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs
However, prior work has found that such recurrent model-free RL methods tend to perform worse than more specialized algorithms that are designed for specific types of POMDPs.
The Information Geometry of Unsupervised Reinforcement Learning
In this work, we show that unsupervised skill discovery algorithms based on mutual information maximization do not learn skills that are optimal for every possible reward function.
Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
Many model-based reinforcement learning (RL) methods follow a similar template: fit a model to previously observed data, and then use data from that model for RL or planning.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Learning Visual-Linguistic Adequacy, Fidelity, and Fluency for Novel Object Captioning
Novel object captioning (NOC) learns image captioning models for describing objects or visual concepts which are unseen (i. e., novel) in the training captions.
FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding
The few-shot natural language understanding (NLU) task has attracted much recent attention.
Robust Predictable Control
Many of the challenges facing today's reinforcement learning (RL) algorithms, such as robustness, generalization, transfer, and computational efficiency are closely related to compression.
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
Towards Understanding and Mitigating Social Biases in Language Models
As machine learning methods are deployed in real-world settings such as healthcare, legal systems, and social science, it is crucial to recognize how they shape social biases and stereotypes in these sensitive decision-making processes.
Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Using computational models, we find that language and multimodal representations of mobile typed text (spanning typed characters, words, keystroke timings, and app usage) are predictive of daily mood.
Online Sub-Sampling for Reinforcement Learning with General Function Approximation
For a value-based method with complexity-bounded function class, we show that the policy only needs to be updated for $\propto\operatorname{poly}\log(K)$ times for running the RL algorithm for $K$ episodes while still achieving a small near-optimal regret bound.
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation.
Ranked #3 on
Speech Recognition
on LibriSpeech test-other
(using extra training data)
Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning
In this paper, we propose a Conditional Contrastive Learning (CCL) approach to improve the fairness of contrastive SSL methods.
Integrating Auxiliary Information in Self-supervised Learning
Our approach contributes as follows: 1) Comparing to conventional self-supervised representations, the auxiliary-information-infused self-supervised representations bring the performance closer to the supervised representations; 2) The presented Cl-InfoNCE can also work with unsupervised constructed clusters (e. g., k-means clusters) and outperform strong clustering-based self-supervised learning approaches, such as the Prototypical Contrastive Learning (PCL) method; 3) We show that Cl-InfoNCE may be a better approach to leverage the data clustering information, by comparing it to the baseline approach - learning to predict the clustering assignments with cross-entropy loss.
Iterative Hierarchical Attention for Answering Complex Questions over Long Documents
We propose a new model, DocHopper, that iteratively attends to different parts of long, hierarchically structured documents to answer complex questions.
Ranked #2 on
Question Answering
on ConditionalQA
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration.
A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
In this report, we relate the algorithmic design of Barlow Twins' method to the Hilbert-Schmidt Independence Criterion (HSIC), thus establishing it as a contrastive learning approach that is free of negative samples.
Focused Attention Improves Document-Grounded Generation
Document grounded generation is the task of using the information provided in a document to improve text generation.
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e. g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence.
Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation
Many real-world applications such as robotics provide hard constraints on power and compute that limit the viable model complexity of Reinforcement Learning (RL) agents.
Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes?
Self-supervised Representation Learning with Relative Predictive Coding
This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance.
Instabilities of Offline RL with Pre-Trained Neural Representation
In offline reinforcement learning (RL), we seek to utilize offline data to evaluate (or learn) policies in scenarios where the data are collected from a distribution that substantially differs from that of the target policy to be evaluated.
On Proximal Policy Optimization's Heavy-tailed Gradients
In this paper, we present a detailed empirical study to characterize the heavy-tailed nature of the gradients of the PPO surrogate reward function.
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
We present the Open Predicate Query Language (OPQL); a method for constructing a virtual KB (VKB) trained entirely from text.
The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors
Although deep reinforcement learning has led to breakthroughs in many difficult domains, these successes have required an ever-increasing number of samples, affording only a shrinking segment of the AI community access to their development.
Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private.
Uncertainty Weighted Offline Reinforcement Learning
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration.
Feature-Robust Optimal Transport for High-Dimensional Data
To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence.
Learning To Hallucinate Examples From Extrinsic and Intrinsic Supervision
Learning to hallucinate additional examples has recently been shown as a promising direction to address few-shot learning tasks.
Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
In this work, we propose algorithms for cross-modal generalization: a learning paradigm to train a model that can (1) quickly perform new tasks in a target modality (i. e. meta-learning) and (2) doing so while being trained on a different source modality.
Probing Predictions on OOD Images via Nearest Categories
Overall, adversarially robust networks resemble a nearest neighbor classifier when it comes to OOD data.
C-Learning: Learning to Achieve Goals via Recursive Classification
This problem, which can be viewed as a reframing of goal-conditioned reinforcement learning (RL), is centered around learning a conditional probability density function over future states.
Unsupervised Domain Adaptation for Visual Navigation
We further show that our method can be used to transfer the navigation policies learned in simulation to the real world.
Planning with Submodular Objective Functions
We study planning with submodular objective functions, where instead of maximizing the cumulative reward, the goal is to maximize the objective value induced by a submodular function.
Case Study: Deontological Ethics in NLP
Recent work in natural language processing (NLP) has focused on ethical challenges such as understanding and mitigating bias in data and algorithms; identifying objectionable content like hate speech, stereotypes and offensive language; and building frameworks for better system design and data handling practices.
Information Obfuscation of Graph Neural Networks
While the advent of Graph Neural Networks (GNNs) has greatly improved node and graph representation learning in many applications, the neighborhood aggregation scheme exposes additional vulnerabilities to adversaries seeking to extract node-level information about sensitive attributes.
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function
In this paper, we study bidirectional LSTM network for the task of text classification using both supervised and semi-supervised approaches.
Ranked #3 on
Text Classification
on AG News
Few-Shot Learning with Intra-Class Knowledge Transfer
Recent works have proposed to solve this task by augmenting the training data of the few-shot classes using generative models with the few-shot training samples as the seeds.
Towards Debiasing Sentence Representations
As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes.
Object Goal Navigation using Goal-Oriented Semantic Exploration
We propose a modular system called, `Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category.
Ranked #4 on
Robot Navigation
on Habitat 2020 Object Nav test-std
Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function.
On Reward-Free Reinforcement Learning with Linear Function Approximation
The sample complexity of our algorithm is polynomial in the feature dimension and the planning horizon, and is completely independent of the number of states and actions.
Self-supervised Learning from a Multi-view Perspective
In particular, we propose a composite objective that bridges the gap between prior contrastive and predictive learning objectives, and introduce an additional objective term to discard task-irrelevant information.
Neural Methods for Point-wise Dependency Estimation
Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables.
Neural Topological SLAM for Visual Navigation
This paper studies the problem of image-goal navigation which involves navigating to the location indicated by a goal image in a novel previously unseen environment.
Feature Robust Optimal Transport for High-dimensional Data
To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence.
Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
Value function approximation has demonstrated phenomenal empirical success in reinforcement learning (RL).
Guaranteeing Reproducibility in Deep Learning Competitions
To encourage the development of methods with reproducible and robust training behavior, we propose a challenge paradigm where competitors are evaluated directly on the performance of their learning procedures rather than pre-trained agents.
Exploring Controllable Text Generation Techniques
In this work, we provide a new schema of the pipeline of the generation process by classifying it into five modules.
Topological Sort for Sentence Ordering
Sentence ordering is the task of arranging the sentences of a given text in the correct order.
Politeness Transfer: A Tag and Generate Approach
This paper introduces a new task of politeness transfer which involves converting non-polite sentences to polite sentences while preserving the meaning.
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language.
Learning to Explore using Active Neural SLAM
The use of learning provides flexibility with respect to input modalities (in the SLAM module), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies).
Weakly-Supervised Reinforcement Learning for Controllable Behavior
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks.
A Closer Look at Accuracy vs. Robustness
Current methods for training robust networks lead to a drop in test accuracy, which has led prior works to posit that a robustness-accuracy tradeoff may be inevitable in deep learning.
On Emergent Communication in Competitive Multi-Agent Teams
Several recent works have found the emergence of grounded compositional language in the communication protocols developed by mostly cooperative multi-agent systems when learned end-to-end to maximize performance on a downstream task.
Differentiable Reasoning over a Virtual Knowledge Base
In particular, we describe a neural module, DrKIT, that traverses textual data like a KB, softly following paths of relations between mentions of entities in the corpus.
Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
In this paper, we show that hindsight relabeling is inverse RL, an observation that suggests that we can use inverse RL in tandem for RL algorithms to efficiently solve many tasks.
Learning Not to Learn in the Presence of Noisy Labels
Learning in the presence of label noise is a challenging yet important task: it is crucial to design models that are robust in the presence of mislabeled datasets.
Capsules with Inverted Dot-Product Attention Routing
We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote.
Think Locally, Act Globally: Federated Learning with Local and Global Representations
To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices.
Geometric Capsule Autoencoders for 3D Point Clouds
The pose encodes where the entity is, while the feature encodes what it is.
Worst Cases Policy Gradients
Recent advances in deep reinforcement learning have demonstrated the capability of learning complex control policies from many types of environments.
Multiple Futures Prediction
Towards these goals, we introduce a probabilistic framework that efficiently learns latent variables to jointly model the multi-step future motions of agents in a scene.
Enhanced Convolutional Neural Tangent Kernels
An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel.
Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel
This new formulation gives us a better way to understand individual components of the Transformer{'}s attention, such as the better way to integrate the positional embedding.
Learning Data Manipulation for Augmentation and Weighting
In this work, we propose a new method that supports learning different manipulation schemes with the same gradient-based algorithm.
Complex Transformer: A Framework for Modeling Complex-Valued Sequence
While deep learning has received a surge of interest in a variety of fields in recent years, major deep learning models barely use complex numbers.
Ranked #2 on
Music Transcription
on MusicNet
On Universal Approximation by Neural Networks with Uniform Guarantees on Approximation of Infinite Dimensional Maps
Additionally, we provide the first lower-bound on the minimal number of input and output units required by a finite approximation to an infinite neural network to guarantee that it can uniformly approximate any nonlinear operator using samples from its inputs and outputs.
Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance.
A Simple Approach to the Noisy Label Problem Through the Gambler's Loss
Learning in the presence of label noise is a challenging yet important task.
LSMI-Sinkhorn: Semi-supervised Mutual Information Estimation with Optimal Transport
To estimate the mutual information from data, a common practice is preparing a set of paired samples $\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i=1}^n \stackrel{\mathrm{i. i. d.
Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel
This new formulation gives us a better way to understand individual components of the Transformer's attention, such as the better way to integrate the positional embedding.
``My Way of Telling a Story'': Persona based Grounded Story Generation
To this end, we propose five models which are incremental extensions to the baseline model to perform the task at hand.
MineRL: A Large-Scale Dataset of Minecraft Demonstrations
Therefore, we introduce a comprehensive, large-scale, simulator-paired dataset of human demonstrations: MineRL.
Learning Neural Networks with Adaptive Regularization
Feed-forward neural networks can be understood as a combination of an intermediate representation and a linear hypothesis.
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Our method is based on the observation that high-dimensional multimodal time series data often exhibit correlations across time and modalities which leads to low-rank tensor representations.
Deep Gamblers: Learning to Abstain with Portfolio Theory
We deal with the \textit{selective classification} problem (supervised-learning problem with a rejection option), where we want to achieve the best performance at a certain level of coverage of the data.
XLNet: Generalized Autoregressive Pretraining for Language Understanding
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling.
"My Way of Telling a Story": Persona based Grounded Story Generation
To this end, we propose five models which are incremental extensions to the baseline model to perform the task at hand.
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
We introduce a general control algorithm that combines the strengths of planning and reinforcement learning to effectively solve these tasks.
Efficient Exploration via State Marginal Matching
The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors.
Ranked #5 on
Multimodal Sentiment Analysis
on MOSI
Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
While graph kernels (GKs) are easy to train and enjoy provable theoretical guarantees, their practical performances are limited by their expressive power, as the kernel function often depends on hand-crafted combinatorial features of graphs.
Strong and Simple Baselines for Multimodal Utterance Embeddings
Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations.
Cross-Task Knowledge Transfer for Visually-Grounded Navigation
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for two different tasks: learning to follow navigational instructions and embodied question answering.
Learning From the Experience of Others: Approximate Empirical Bayes in Neural Networks
Learning deep neural networks could be understood as the combination of representation learning and learning halfspaces.
Transformer-XL: Language Modeling with Longer-Term Dependency
Moreover, Transformer-XL is up to 1, 800+ times faster than vanilla Transformer during evaluation.
On Exact Computation with an Infinitely Wide Neural Net
An attraction of such ideas is that a pure kernel-based method is used to capture the power of a fully-trained deep net of infinite width.
The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
To that end, we introduce: (1) the Minecraft ObtainDiamond task, a sequential decision making environment requiring long-term planning, hierarchical control, and efficient exploration methods; and (2) the MineRL-v0 dataset, a large-scale collection of over 60 million state-action pairs of human demonstrations that can be resimulated into embodied trajectories with arbitrary modifications to game state and visuals.
Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph
Visual relationship reasoning is a crucial yet challenging task for understanding rich interactions across visual concepts.
Concurrent Meta Reinforcement Learning
In this multi-agent setting, a set of parallel agents are executed in the same environment and each of these "rollout" agents are given the means to communicate with each other.
Ranked #1 on
Meta Reinforcement Learning
on 3-Reacher
The Omniglot challenge: a 3-year progress report
Three years ago, we released the Omniglot dataset for one-shot learning, along with five challenge tasks and a computational model that addresses these tasks.
Embodied Multimodal Multitask Learning
In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks.
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.
Ranked #3 on
Language Modelling
on One Billion Word
Connecting the Dots Between MLE and RL for Sequence Prediction
Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency.
Stackelberg GAN: Towards Provable Minimax Equilibrium via Multi-Generator Architectures
In this work, we give new results on the benefits of multi-generator architecture of GANs.
On the Complexity of Exploration in Goal-Driven Navigation
Next, we propose to measure the complexity of each environment by constructing dependency graphs between the goals and analytically computing \emph{hitting times} of a random walk in the graph.
Point Cloud GAN
In this paper, we first show a straightforward extension of existing GAN algorithm is not applicable to point clouds, because the constraint required for discriminators is undefined for set data.
AutoLoss: Learning Discrete Schedules for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters.
AutoLoss: Learning Discrete Schedule for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters.
Connecting the Dots Between MLE and RL for Sequence Generation
We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters.
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers.