Search Results for author: Lisa Lee

Found 21 papers, 10 papers with code

Guide Your Agent with Adaptive Multimodal Rewards

1 code implementation19 Sep 2023 Changyeon Kim, Younggyo Seo, Hao liu, Lisa Lee, Jinwoo Shin, Honglak Lee, Kimin Lee

Developing an agent capable of adapting to unseen environments remains a difficult challenge in imitation learning.

Imitation Learning

Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation

no code implementations7 Jul 2023 Annie Xie, Lisa Lee, Ted Xiao, Chelsea Finn

Towards an answer to this question, we study imitation learning policies in simulation and on a real robot language-conditioned manipulation task to quantify the difficulty of generalization to different (sets of) factors.

Imitation Learning

Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models

no code implementations24 Oct 2022 Hao liu, Xinyang Geng, Lisa Lee, Igor Mordatch, Sergey Levine, Sharan Narang, Pieter Abbeel

Large language models (LLM) trained using the next-token-prediction objective, such as GPT3 and PaLM, have revolutionized natural language processing in recent years by showing impressive zero-shot and few-shot capabilities across a wide range of tasks.

Language Modelling Natural Language Inference +1

Instruction-Following Agents with Multimodal Transformer

1 code implementation24 Oct 2022 Hao liu, Lisa Lee, Kimin Lee, Pieter Abbeel

Our \ours method consists of a multimodal transformer that encodes visual observations and language instructions, and a transformer-based policy that predicts actions based on encoded representations.

Instruction Following Visual Grounding

Multi-Game Decision Transformers

1 code implementation30 May 2022 Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, Igor Mordatch

Specifically, we show that a single transformer-based model - with a single set of weights - trained purely offline can play a suite of up to 46 Atari games simultaneously at close-to-human performance.

Atari Games Offline RL

Multimodal Masked Autoencoders Learn Transferable Representations

1 code implementation27 May 2022 Xinyang Geng, Hao liu, Lisa Lee, Dale Schuurmans, Sergey Levine, Pieter Abbeel

We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks.

Contrastive Learning

f-IRL: Inverse Reinforcement Learning via State Marginal Matching

1 code implementation9 Nov 2020 Tianwei Ni, Harshit Sikchi, YuFei Wang, Tejus Gupta, Lisa Lee, Benjamin Eysenbach

Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories on IRL benchmarks.

Imitation Learning reinforcement-learning +1

Efficient Exploration via State Marginal Matching

1 code implementation12 Jun 2019 Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, Ruslan Salakhutdinov

The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective.

Efficient Exploration Unsupervised Reinforcement Learning

Cross-Task Knowledge Transfer for Visually-Grounded Navigation

no code implementations ICLR 2019 Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, Dhruv Batra

Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for two different tasks: learning to follow navigational instructions and embodied question answering.

Disentanglement Embodied Question Answering +3

Embodied Multimodal Multitask Learning

no code implementations4 Feb 2019 Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, Dhruv Batra

In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks.

Disentanglement Embodied Question Answering +3

On the Complexity of Exploration in Goal-Driven Navigation

no code implementations16 Nov 2018 Maruan Al-Shedivat, Lisa Lee, Ruslan Salakhutdinov, Eric Xing

Next, we propose to measure the complexity of each environment by constructing dependency graphs between the goals and analytically computing \emph{hitting times} of a random walk in the graph.


Gated Path Planning Networks

3 code implementations ICML 2018 Lisa Lee, Emilio Parisotto, Devendra Singh Chaplot, Eric Xing, Ruslan Salakhutdinov

Value Iteration Networks (VINs) are effective differentiable path planning modules that can be used by agents to perform navigation while still maintaining end-to-end differentiability of the entire architecture.

Combining LSTM and Latent Topic Modeling for Mortality Prediction

no code implementations8 Sep 2017 Yohan Jo, Lisa Lee, Shruti Palaskar

There is a great need for technologies that can predict the mortality of patients in intensive care units with both high accuracy and accountability.

Mortality Prediction

Dual Motion GAN for Future-Flow Embedded Video Prediction

no code implementations ICCV 2017 Xiaodan Liang, Lisa Lee, Wei Dai, Eric P. Xing

To make both synthesized future frames and flows indistinguishable from reality, a dual adversarial training method is proposed to ensure that the future-flow prediction is able to help infer realistic future-frames, while the future-frame prediction in turn leads to realistic optical flows.

Representation Learning Video Prediction

Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection

1 code implementation CVPR 2017 Xiaodan Liang, Lisa Lee, Eric P. Xing

To capture such global interdependency, we propose a deep Variation-structured Reinforcement Learning (VRL) framework to sequentially discover object relationships and attributes in the whole image.

Image Classification reinforcement-learning +3

Robotic Search & Rescue via Online Multi-task Reinforcement Learning

no code implementations29 Nov 2015 Lisa Lee

Reinforcement learning (RL) is a general and well-known method that a robot can use to learn an optimal control policy to solve a particular task.

Q-Learning reinforcement-learning +1

Cannot find the paper you are looking for? You can Submit a new open access paper.