Search Results for author: Rob Fergus

Found 70 papers, 35 papers with code

Hierarchical reinforcement learning with natural language subgoals

no code implementations20 Sep 2023 Arun Ahuja, Kavya Kopparapu, Rob Fergus, Ishita Dasgupta

Hierarchical reinforcement learning has been a compelling approach for achieving goal directed behavior over long sequences of actions.

Hierarchical Reinforcement Learning reinforcement-learning

NetHack is Hard to Hack

no code implementations30 May 2023 Ulyana Piterbarg, Lerrel Pinto, Rob Fergus

Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion.

Atari Games Imitation Learning +1

Accelerating exploration and representation learning with offline pre-training

no code implementations31 Mar 2023 Bogdan Mazoure, Jake Bruce, Doina Precup, Rob Fergus, Ankit Anand

In this work, we follow the hypothesis that exploration and representation learning can be improved by separately learning two different models from a single offline dataset.

Decision Making NetHack +2

Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC

2 code implementations22 Feb 2023 Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, Will Grathwohl

In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance.

Collaborating with language models for embodied reasoning

no code implementations1 Feb 2023 Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, Rob Fergus

On the other hand, Large Scale Language Models (LSLMs) have exhibited strong reasoning ability and the ability to to adapt to new tasks through in-context learning.

Language Modelling reinforcement-learning +1

Teacher Guided Training: An Efficient Framework for Knowledge Transfer

no code implementations14 Aug 2022 Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim, Chong You, Himanshu Jain, Andreas Veit, Rob Fergus, Sanjiv Kumar

In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data.

Generalization Bounds Image Classification +4

Match Prediction Using Learned History Embeddings

no code implementations29 Sep 2021 Maxwell Goldstein, Leon Bottou, Rob Fergus

Contemporary ranking systems that are based on win/loss history, such as Elo or TrueSkill represent each player using a scalar estimate of ability (plus variance, in the latter case).

Imitation by Predicting Observations

no code implementations8 Jul 2021 Andrew Jaegle, Yury Sulsky, Arun Ahuja, Jake Bruce, Rob Fergus, Greg Wayne

Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior.

Continuous Control Imitation Learning

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

2 code implementations14 Mar 2021 Ilya Kostrikov, Jonathan Tompson, Rob Fergus, Ofir Nachum

Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline data.

Offline RL reinforcement-learning +1

Reinforcement Learning with Prototypical Representations

1 code implementation22 Feb 2021 Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto

Unfortunately, in RL, representation learning is confounded with the exploratory experience of the agent -- learning a useful representation requires diverse data, while effective exploration is only possible with coherent representations.

Continuous Control reinforcement-learning +3

Decoupling Value and Policy for Generalization in Reinforcement Learning

1 code implementation20 Feb 2021 Roberta Raileanu, Rob Fergus

Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images.

reinforcement-learning Reinforcement Learning (RL)

Fast Adaptation via Policy-Dynamics Value Functions

1 code implementation6 Jul 2020 Roberta Raileanu, Max Goldstein, Arthur Szlam, Rob Fergus

An ensemble of conventional RL policies is used to gather experience on training environments, from which embeddings of both policies and environments can be learned.

Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

4 code implementations ICLR 2021 Ilya Kostrikov, Denis Yarats, Rob Fergus

We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training.

Atari Games 100k Continuous Control +4

Finding Generalizable Evidence by Learning to Convince Q\&A Models

no code implementations IJCNLP 2019 Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho

We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.

Question Answering

Agent as Scientist: Learning to Verify Hypotheses

no code implementations25 Sep 2019 Kenneth Marino, Rob Fergus, Arthur Szlam, Abhinav Gupta

In order to train the agents, we exploit the underlying structure in the majority of hypotheses -- they can be formulated as triplets (pre-condition, action sequence, post-condition).

Finding Generalizable Evidence by Learning to Convince Q&A Models

1 code implementation12 Sep 2019 Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho

We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.

Question Answering

Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies

no code implementations ICLR 2019 Kenneth Marino, Abhinav Gupta, Rob Fergus, Arthur Szlam

The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time.

Disentangling Video with Independent Prediction

no code implementations17 Jan 2019 William F. Whitney, Rob Fergus

We propose an unsupervised variational model for disentangling video into independent factors, i. e. each factor's future can be predicted from its past without considering the others.

Understanding the Asymptotic Performance of Model-Based RL Methods

no code implementations27 Sep 2018 William Whitney, Rob Fergus

In complex simulated environments, model-based reinforcement learning methods typically lag the asymptotic performance of model-free approaches.

Model-based Reinforcement Learning

IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning

1 code implementation20 Mar 2018 Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, Emmanuel Dupoux

In order to reach human performance on complexvisual tasks, artificial systems need to incorporate a sig-nificant amount of understanding of the world in termsof macroscopic objects, movements, forces, etc.

Modeling Others using Oneself in Multi-Agent Reinforcement Learning

1 code implementation ICML 2018 Roberta Raileanu, Emily Denton, Arthur Szlam, Rob Fergus

We consider the multi-agent reinforcement learning setting with imperfect information in which each agent is trying to maximize its own utility.

Multi-agent Reinforcement Learning reinforcement-learning +1

Stochastic Video Generation with a Learned Prior

3 code implementations ICML 2018 Emily Denton, Rob Fergus

Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.

Video Generation Video Prediction

Learning by Asking Questions

no code implementations CVPR 2018 Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, Laurens van der Maaten

We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.

Question Answering Visual Question Answering

Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play

3 code implementations ICLR 2018 Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, Rob Fergus

When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.

Learning Physical Intuition of Block Towers by Example

2 code implementations3 Mar 2016 Adam Lerer, Sam Gross, Rob Fergus

Wooden blocks are a common toy for infants, allowing them to develop motor skills and gain intuition about the physical behavior of the world.

Physical Intuition

Learning Simple Algorithms from Examples

1 code implementation23 Nov 2015 Wojciech Zaremba, Tomas Mikolov, Armand Joulin, Rob Fergus

We present an approach for learning simple algorithms such as copying, multi-digit addition and single digit multiplication directly from examples.


MazeBase: A Sandbox for Learning from Games

2 code implementations23 Nov 2015 Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve, Soumith Chintala, Rob Fergus

This paper introduces MazeBase: an environment for simple 2D games, designed as a sandbox for machine learning approaches to reasoning and planning.

Reinforcement Learning (RL) Starcraft

Deep End2End Voxel2Voxel Prediction

no code implementations20 Nov 2015 Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri

Over the last few years deep learning methods have emerged as one of the most prominent approaches for video analysis.

Neural Architecture Search Optical Flow Estimation +3

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

1 code implementation18 Jun 2015 Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus

In this paper we introduce a generative parametric model capable of producing high quality samples of natural images.

Web Scale Photo Hash Clustering on A Single Machine

no code implementations CVPR 2015 Yunchao Gong, Marcin Pawlowski, Fei Yang, Louis Brandy, Lubomir Bourdev, Rob Fergus

In addition, we propose an online clustering method based on binary k-means that is capable of clustering large photo stream on a single machine, and show applications to spam detection and trending photo discovery.

Clustering Online Clustering +1

Improving Image Classification with Location Context

no code implementations ICCV 2015 Kevin Tang, Manohar Paluri, Li Fei-Fei, Rob Fergus, Lubomir Bourdev

With the widespread availability of cellphones and cameras that have GPS capabilities, it is common for images being uploaded to the Internet today to have GPS coordinates associated with them.

Classification General Classification +1

Learning Spatiotemporal Features with 3D Convolutional Networks

24 code implementations ICCV 2015 Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri

We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset.

Action Recognition In Videos

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture

4 code implementations ICCV 2015 David Eigen, Rob Fergus

In this paper we address three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling.

Depth Prediction Monocular Depth Estimation +2

Deep Poselets for Human Detection

no code implementations2 Jul 2014 Lubomir Bourdev, Fei Yang, Rob Fergus

We train the poselet model on top of PDF features and combine them with object-level CNNs for detection and bounding box prediction.

Human Detection

Training Convolutional Networks with Noisy Labels

no code implementations9 Jun 2014 Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, Rob Fergus

The availability of large labeled datasets has allowed Convolutional Network models to achieve impressive recognition results.

General Classification

Learning to Discover Efficient Mathematical Identities

1 code implementation NeurIPS 2014 Wojciech Zaremba, Karol Kurach, Rob Fergus

In this paper we explore how machine learning techniques can be applied to the discovery of efficient mathematical identities.

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

no code implementations NeurIPS 2014 Emily Denton, Wojciech Zaremba, Joan Bruna, Yann Lecun, Rob Fergus

We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks.

Object Recognition

OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks

4 code implementations21 Dec 2013 Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, Yann Lecun

This integrated framework is the winner of the localization task of the ImageNet Large Scale Visual Recognition Challenge 2013 (ILSVRC2013) and obtained very competitive results for the detection and classifications tasks.

General Classification Image Classification +2

Intriguing properties of neural networks

11 code implementations21 Dec 2013 Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus

Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks.

Understanding Deep Architectures using a Recursive Convolutional Network

no code implementations6 Dec 2013 David Eigen, Jason Rolfe, Rob Fergus, Yann Lecun

A key challenge in designing convolutional network models is sizing them appropriately.

Blind Deconvolution with Non-local Sparsity Reweighting

no code implementations16 Nov 2013 Dilip Krishnan, Joan Bruna, Rob Fergus

Blind deconvolution has made significant progress in the past decade.

Visualizing and Understanding Convolutional Networks

18 code implementations12 Nov 2013 Matthew D. Zeiler, Rob Fergus

Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark.

General Classification Image Classification +1

Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines

no code implementations NeurIPS 2011 Matthew D. Zeiler, Graham W. Taylor, Leonid Sigal, Iain Matthews, Rob Fergus

We present a type of Temporal Restricted Boltzmann Machine that defines a probability distribution over an output sequence conditional on an input sequence.

Pose-Sensitive Embedding by Nonlinear NCA Regression

no code implementations NeurIPS 2010 Graham W. Taylor, Rob Fergus, George Williams, Ian Spiro, Christoph Bregler

We apply our method to challenging real-world data and show that it can generalize beyond hand localization to infer a more general notion of body pose.


Fast Image Deconvolution using Hyper-Laplacian Priors

no code implementations NeurIPS 2009 Dilip Krishnan, Rob Fergus

In this paper we describe a deconvolution approach that is several orders of magnitude faster than existing techniques that use hyper-Laplacian priors.

Deblurring Denoising +2

Semi-Supervised Learning in Gigantic Image Collections

no code implementations NeurIPS 2009 Rob Fergus, Yair Weiss, Antonio Torralba

With the advent of the Internet it is now possible to collect hundreds of millions of images.

Spectral Hashing

no code implementations NeurIPS 2008 Yair Weiss, Antonio Torralba, Rob Fergus

Semantic hashing seeks compact binary codes of datapoints so that the Hamming distance between codewords correlates with semantic similarity.

graph partitioning Semantic Similarity +1

Cannot find the paper you are looking for? You can Submit a new open access paper.