Search Results for author:
Found 70 papers, 36 papers with code
diff History for Neural Language Agents
On NetHack, an unsolved video game that requires long-horizon reasoning for decision-making, LMs tuned with diff history match state-of-the-art performance for neural agents while needing 1800x fewer training examples compared to prior work.
Hierarchical reinforcement learning with natural language subgoals
Hierarchical reinforcement learning has been a compelling approach for achieving goal directed behavior over long sequences of actions.
Accelerating exploration and representation learning with offline pre-training
In this work, we follow the hypothesis that exploration and representation learning can be improved by separately learning two different models from a single offline dataset.
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance.
Collaborating with language models for embodied reasoning
On the other hand, Large Scale Language Models (LSLMs) have exhibited strong reasoning ability and the ability to to adapt to new tasks through in-context learning.
Distilling Internet-Scale Vision-Language Models into Embodied Agents
Instruction-following agents must ground language into their observation and action spaces.
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR).
Learning to Navigate Wikipedia by Taking Random Walks
A fundamental ability of an intelligent web-based agent is seeking out and acquiring new information.
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data.
Automatic Data Augmentation for Generalization in Reinforcement Learning
Deep reinforcement learning (RL) agents often fail to generalize beyond their training environments.
Match Prediction Using Learned History Embeddings
Contemporary ranking systems that are based on win/loss history, such as Elo or TrueSkill represent each player using a scalar estimate of ability (plus variance, in the latter case).
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control.
Imitation by Predicting Observations
Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior.
Offline Reinforcement Learning with Fisher Divergence Critic Regularization
Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline data.
Reinforcement Learning with Prototypical Representations
Unfortunately, in RL, representation learning is confounded with the exploratory experience of the agent -- learning a useful representation requires diverse data, while effective exploration is only possible with coherent representations.
Decoupling Value and Policy for Generalization in Reinforcement Learning
Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images.
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation.
Fast Adaptation via Policy-Dynamics Value Functions
An ensemble of conventional RL policies is used to gather experience on training environments, from which embeddings of both policies and environments can be learned.
Empirically Verifying Hypotheses Using Reinforcement Learning
This paper formulates hypothesis verification as an RL problem.
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
Our agent outperforms other baselines specifically designed to improve generalization in RL.
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training.
Ranked #1 on
Continuous Control
on DeepMind Walker Walk (Images)
Energy-based models for atomic-resolution protein conformations
We propose an energy-based model (EBM) of protein conformations that operates at atomic scale.
Finding Generalizable Evidence by Learning to Convince Q\&A Models
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
A promising approach is to learn a latent representation together with the control policy.
Agent as Scientist: Learning to Verify Hypotheses
In order to train the agents, we exploit the underlying structure in the majority of hypotheses -- they can be formulated as triplets (pre-condition, action sequence, post-condition).
Finding Generalizable Evidence by Learning to Convince Q&A Models
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.
Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies
The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time.
Disentangling Video with Independent Prediction
We propose an unsupervised variational model for disentangling video into independent factors, i. e. each factor's future can be predicted from its past without considering the others.
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
In hierarchical reinforcement learning a major challenge is determining appropriate low-level policies.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Understanding the Asymptotic Performance of Model-Based RL Methods
In complex simulated environments, model-based reinforcement learning methods typically lag the asymptotic performance of model-free approaches.
IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
In order to reach human performance on complexvisual tasks, artificial systems need to incorporate a sig-nificant amount of understanding of the world in termsof macroscopic objects, movements, forces, etc.
Composable Planning with Attributes
The tasks that an agent will need to solve often are not known during training.
Modeling Others using Oneself in Multi-Agent Reinforcement Learning
We consider the multi-agent reinforcement learning setting with imperfect information in which each agent is trying to maximize its own utility.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Stochastic Video Generation with a Learned Prior
Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.
Ranked #3 on
Video Prediction
on KTH
Learning by Asking Questions
We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.
Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
We introduce a simple semi-supervised learning approach for images based on in-painting using an adversarial loss.
Learning Multiagent Communication with Backpropagation
Many tasks in AI require the collaboration of multiple agents.
Learning Physical Intuition of Block Towers by Example
Wooden blocks are a common toy for infants, allowing them to develop motor skills and gain intuition about the physical behavior of the world.
Simple Baseline for Visual Question Answering
We describe a very simple bag-of-words baseline for visual question answering.
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
In this paper we introduce a generative model capable of producing high quality samples of natural images.
Learning Simple Algorithms from Examples
We present an approach for learning simple algorithms such as copying, multi-digit addition and single digit multiplication directly from examples.
MazeBase: A Sandbox for Learning from Games
This paper introduces MazeBase: an environment for simple 2D games, designed as a sandbox for machine learning approaches to reasoning and planning.
Deep End2End Voxel2Voxel Prediction
Over the last few years deep learning methods have emerged as one of the most prominent approaches for video analysis.
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
In this paper we introduce a generative parametric model capable of producing high quality samples of natural images.
End-to-End Integration of a Convolution Network, Deformable Parts Model and Non-Maximum Suppression
In this paper, we propose a new model that combines these two approaches, obtaining the advantages of each.
Web Scale Photo Hash Clustering on A Single Machine
In addition, we propose an online clustering method based on binary k-means that is capable of clustering large photo stream on a single machine, and show applications to spam detection and trending photo discovery.
Improving Image Classification with Location Context
With the widespread availability of cellphones and cameras that have GPS capabilities, it is common for images being uploaded to the Internet today to have GPS coordinates associated with them.
End-To-End Memory Networks
For the former our approach is competitive with Memory Networks, but with less supervision.
Ranked #6 on
Question Answering
on bAbi
Beyond Frontal Faces: Improving Person Recognition Using Multiple Cues
We explore the task of recognizing peoples' identities in photo albums in an unconstrained setting.
Learning Spatiotemporal Features with 3D Convolutional Networks
We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset.
Ranked #8 on
Action Recognition
on Sports-1M
Action Recognition In Videos
Dynamic Facial Expression Recognition
End-to-End Integration of a Convolutional Network, Deformable Parts Model and Non-Maximum Suppression
In this paper, we propose a new model that combines these two approaches, obtaining the advantages of each.
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
In this paper we address three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling.
Ranked #71 on
Monocular Depth Estimation
on NYU-Depth V2
Deep Poselets for Human Detection
We train the poselet model on top of PDF features and combine them with object-level CNNs for detection and bounding box prediction.
Training Convolutional Networks with Noisy Labels
The availability of large labeled datasets has allowed Convolutional Network models to achieve impressive recognition results.
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Predicting depth is an essential component in understanding the 3D geometry of a scene.
Learning to Discover Efficient Mathematical Identities
In this paper we explore how machine learning techniques can be applied to the discovery of efficient mathematical identities.
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks.
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
This integrated framework is the winner of the localization task of the ImageNet Large Scale Visual Recognition Challenge 2013 (ILSVRC2013) and obtained very competitive results for the detection and classifications tasks.
Intriguing properties of neural networks
Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks.
Understanding Deep Architectures using a Recursive Convolutional Network
A key challenge in designing convolutional network models is sizing them appropriately.
Blind Deconvolution with Non-local Sparsity Reweighting
Blind deconvolution has made significant progress in the past decade.
Visualizing and Understanding Convolutional Networks
Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark.
Regularization of Neural Networks using DropConnect
When training with Dropout, a randomly selected subset of activations are set to zero within each layer.
Ranked #6 on
Image Classification
on MNIST
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
We introduce a simple and effective method for regularizing large convolutional neural networks.
Ranked #39 on
Image Classification
on SVHN
Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines
We present a type of Temporal Restricted Boltzmann Machine that defines a probability distribution over an output sequence conditional on an input sequence.
Pose-Sensitive Embedding by Nonlinear NCA Regression
We apply our method to challenging real-world data and show that it can generalize beyond hand localization to infer a more general notion of body pose.
Semi-Supervised Learning in Gigantic Image Collections
With the advent of the Internet it is now possible to collect hundreds of millions of images.
Fast Image Deconvolution using Hyper-Laplacian Priors
In this paper we describe a deconvolution approach that is several orders of magnitude faster than existing techniques that use hyper-Laplacian priors.
Spectral Hashing
Semantic hashing seeks compact binary codes of datapoints so that the Hamming distance between codewords correlates with semantic similarity.
