Search Results for author: Tianmin Shu

Found 29 papers, 10 papers with code

Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning

no code implementations8 Dec 2023 Zhiting Hu, Tianmin Shu

Despite their tremendous success in many applications, large language models often fall short of consistent reasoning and planning in various (language, embodied, and social) scenarios, due to inherent limitations in their inference, learning, and modeling capabilities.

Neural Amortized Inference for Nested Multi-agent Reasoning

1 code implementation21 Aug 2023 Kunal Jha, Tuan Anh Le, Chuanyang Jin, Yen-Ling Kuo, Joshua B. Tenenbaum, Tianmin Shu

Multi-agent interactions, such as communication, teaching, and bluffing, often rely on higher-order social inference, i. e., understanding how others infer oneself.

Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation

no code implementations12 Jul 2023 Andi Peng, Aviv Netanyahu, Mark Ho, Tianmin Shu, Andreea Bobu, Julie Shah, Pulkit Agrawal

Policies often fail due to distribution shift -- changes in the state and reward that occur when a policy is deployed in new environments.

Continuous Control counterfactual +2

Building Cooperative Embodied Agents Modularly with Large Language Models

1 code implementation5 Jul 2023 Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan

In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments.

Text Generation

Language Models Meet World Models: Embodied Experiences Enhance Language Models

1 code implementation NeurIPS 2023 Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, ZiRui Wang, Zichao Yang, Zhiting Hu

While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household activities.

NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants

no code implementations12 Jan 2023 Xavier Puig, Tianmin Shu, Joshua B. Tenenbaum, Antonio Torralba

Experiments show that our helper agent robustly updates its goal inference and adapts its helping plans to the changing level of uncertainty.

Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning

no code implementations24 Nov 2022 Aviv Netanyahu, Tianmin Shu, Joshua Tenenbaum, Pulkit Agrawal

To address this, we propose a reward learning approach, Graph-based Equivalence Mappings (GEM), that can discover spatial goal representations that are aligned with the intended goal specification, enabling successful generalization in unseen environments.

Imitation Learning

Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning

2 code implementations4 Oct 2022 Dianbo Liu, Vedant Shah, Oussama Boussif, Cristian Meo, Anirudh Goyal, Tianmin Shu, Michael Mozer, Nicolas Heess, Yoshua Bengio

We formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment.

Multi-agent Reinforcement Learning reinforcement-learning +1

Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration

no code implementations6 Mar 2021 Xiaofeng Gao, Luyao Yuan, Tianmin Shu, Hongjing Lu, Song-Chun Zhu

Our experiments with human participants demonstrate that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground truth.

Motion Planning

PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception

no code implementations NeurIPS Workshop SVRHM 2020 Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, Joshua B. Tenenbaum

The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation.

AGENT: A Benchmark for Core Psychological Reasoning

no code implementations24 Feb 2021 Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin A. Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua B. Tenenbaum, Tomer D. Ullman

For machine agents to successfully interact with humans in real-world settings, they will need to develop an understanding of human mental life.

Core Psychological Reasoning

Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks

no code implementations24 Jul 2020 Xiaofeng Gao, Ran Gong, Yizhou Zhao, Shu Wang, Tianmin Shu, Song-Chun Zhu

Thus, in this paper, we propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations, where the robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications based on its online Bayesian inference of the user's mental state.

Bayesian Inference Explainable Artificial Intelligence (XAI) +1

Active Visual Information Gathering for Vision-Language Navigation

1 code implementation ECCV 2020 Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, Jianbing Shen

Vision-language navigation (VLN) is the task of entailing an agent to carry out navigational instructions inside photo-realistic environments.

Vision-Language Navigation

M^3RL: Mind-aware Multi-agent Management Reinforcement Learning

no code implementations ICLR 2019 Tianmin Shu, Yuandong Tian

Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward.

Management Multi-agent Reinforcement Learning +2

VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning

1 code implementation13 Mar 2019 Xiaofeng Gao, Ran Gong, Tianmin Shu, Xu Xie, Shu Wang, Song-Chun Zhu

One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for training and testing AI agents.

reinforcement-learning Reinforcement Learning (RL)

Interactive Agent Modeling by Learning to Probe

no code implementations1 Oct 2018 Tianmin Shu, Caiming Xiong, Ying Nian Wu, Song-Chun Zhu

In particular, the probing agent (i. e. a learner) learns to interact with the environment and with a target agent (i. e., a demonstrator) to maximize the change in the observed behaviors of that agent.

Imitation Learning

M$^3$RL: Mind-aware Multi-agent Management Reinforcement Learning

1 code implementation ICLR 2019 Tianmin Shu, Yuandong Tian

Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward.

Management Multi-agent Reinforcement Learning +2

Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning

no code implementations ICLR 2018 Tianmin Shu, Caiming Xiong, Richard Socher

In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills.

reinforcement-learning Reinforcement Learning (RL) +1

CERN: Confidence-Energy Recurrent Network for Group Activity Recognition

no code implementations CVPR 2017 Tianmin Shu, Sinisa Todorovic, Song-Chun Zhu

This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities.

Group Activity Recognition

Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions

no code implementations1 Mar 2017 Tianmin Shu, Xiaofeng Gao, Michael S. Ryoo, Song-Chun Zhu

In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI).


Modeling and Inferring Human Intents and Latent Functional Objects for Trajectory Prediction

no code implementations24 Jun 2016 Dan Xie, Tianmin Shu, Sinisa Todorovic, Song-Chun Zhu

This paper is about detecting functional objects and inferring human intentions in surveillance videos of public spaces.

Clustering Trajectory Prediction

Learning Social Affordance for Human-Robot Interaction

no code implementations13 Apr 2016 Tianmin Shu, M. S. Ryoo, Song-Chun Zhu

In this paper, we present an approach for robot learning of social affordance from human activity videos.

Weakly-supervised Learning

Joint Inference of Groups, Events and Human Roles in Aerial Videos

no code implementations CVPR 2015 Tianmin Shu, Dan Xie, Brandon Rothrock, Sinisa Todorovic, Song-Chun Zhu

This paper addresses a new problem of parsing low-resolution aerial videos of large spatial areas, in terms of 1) grouping, 2) recognizing events and 3) assigning roles to people engaged in events.

Cannot find the paper you are looking for? You can Submit a new open access paper.