Search Results for author: Andrew Szot

Found 17 papers, 7 papers with code

From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons

no code implementations11 Dec 2024 Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, Alexander Toshev

We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typically trained on.

Reinforcement Learning via Auxiliary Task Distillation

1 code implementation24 Jun 2024 Abhinav Narayan Harish, Larry Heck, Josiah P. Hanna, Zsolt Kira, Andrew Szot

We present Reinforcement Learning via Auxiliary Task Distillation (AuxDistill), a new method that enables reinforcement learning (RL) to perform long-horizon robot control problems by distilling behaviors from auxiliary RL tasks.

Object Rearrangement reinforcement-learning +2

Grounding Multimodal Large Language Models in Actions

no code implementations12 Jun 2024 Andrew Szot, Bogdan Mazoure, Harsh Agrawal, Devon Hjelm, Zsolt Kira, Alexander Toshev

For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance.

World Knowledge

Skill Transformer: A Monolithic Policy for Mobile Manipulation

no code implementations ICCV 2023 Xiaoyu Huang, Dhruv Batra, Akshara Rai, Andrew Szot

We present Skill Transformer, an approach for solving long-horizon robotic tasks by combining conditional sequence modeling and skill modularity.

Task Planning

Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second

1 code implementation CVPR 2023 Vincent-Pierre Berges, Andrew Szot, Devendra Singh Chaplot, Aaron Gokaslan, Roozbeh Mottaghi, Dhruv Batra, Eric Undersander

Specifically, a Fetch robot (equipped with a mobile base, 7DoF arm, RGBD camera, egomotion, and onboard sensing) is spawned in a home environment and asked to rearrange objects - by navigating to an object, picking it up, navigating to a target location, and then placing the object at the target location.

Reinforcement Learning (RL)

Adaptive Coordination in Social Embodied Rearrangement

no code implementations31 May 2023 Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, Akshara Rai

We present the task of "Social Rearrangement", consisting of cooperative everyday tasks like setting up the dinner table, tidying a house or unpacking groceries in a simulated multi-agent environment.

Diversity

Goal-Driven Imitation Learning from Observation by Inferring Goal Proximity

no code implementations1 Jan 2021 Andrew Szot, Youngwoon Lee, Shao-Hua Sun, Joseph J Lim

Humans can effectively learn to estimate how close they are to completing a desired task simply by watching others fulfill the task.

Imitation Learning

Generalization to New Actions in Reinforcement Learning

2 code implementations ICML 2020 Ayush Jain, Andrew Szot, Joseph J. Lim

A fundamental trait of intelligence is the ability to achieve goals in the face of novel circumstances, such as making decisions from new action choices.

reinforcement-learning Reinforcement Learning +2

Cannot find the paper you are looking for? You can Submit a new open access paper.