Search Results for author:
Found 31 papers, 10 papers with code
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
In this paper, we investigate the problem of embodied multi-agent cooperation, where decentralized agents must cooperate given only partial egocentric views of the world.
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment
GOMA formulates verbal communication as a planning problem that minimizes the misalignment between the parts of agents' mental states that are relevant to the goals.
MMToM-QA: Multimodal Theory of Mind Question Answering
Human ToM, on the other hand, is more than video or text understanding.
Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning
Despite their tremendous success in many applications, large language models often fall short of consistent reasoning and planning in various (language, embodied, and social) scenarios, due to inherent limitations in their inference, learning, and modeling capabilities.
The Cultural Psychology of Large Language Models: Is ChatGPT a Holistic or Analytic Thinker?
In our research, we probed the cultural cognitive traits of ChatGPT.
Neural Amortized Inference for Nested Multi-agent Reasoning
Multi-agent interactions, such as communication, teaching, and bluffing, often rely on higher-order social inference, i. e., understanding how others infer oneself.
Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation
Policies often fail due to distribution shift -- changes in the state and reward that occur when a policy is deployed in new environments.
Building Cooperative Embodied Agents Modularly with Large Language Models
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments.
Language Models Meet World Models: Embodied Experiences Enhance Language Models
While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household activities.
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Experiments show that our helper agent robustly updates its goal inference and adapts its helping plans to the changing level of uncertainty.
Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning
To address this, we propose a reward learning approach, Graph-based Equivalence Mappings (GEM), that can discover spatial goal representations that are aligned with the intended goal specification, enabling successful generalization in unseen environments.
Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
We formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
RLPrompt formulates a parameter-efficient policy network that generates the desired discrete prompt after training with reward.
Coordinating Policies Among Multiple Agents via an Intelligent Communication Channel
In Multi-Agent Reinforcement Learning (MARL), specialized channels are often introduced that allow agents to communicate directly with one another.
Intelligent Communication
Multi-agent Reinforcement Learning
+2
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Our experiments with human participants demonstrate that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground truth.
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation.
AGENT: A Benchmark for Core Psychological Reasoning
For machine agents to successfully interact with humans in real-world settings, they will need to develop an understanding of human mental life.
Ranked #1 on
Core Psychological Reasoning
on AGENT
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents.
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Thus, in this paper, we propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations, where the robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications based on its online Bayesian inference of the user's mental state.
Bayesian Inference
Explainable Artificial Intelligence (XAI)
+1
Active Visual Information Gathering for Vision-Language Navigation
Vision-language navigation (VLN) is the task of entailing an agent to carry out navigational instructions inside photo-realistic environments.
M^3RL: Mind-aware Multi-agent Management Reinforcement Learning
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward.
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for training and testing AI agents.
Interactive Agent Modeling by Learning to Probe
In particular, the probing agent (i. e. a learner) learns to interact with the environment and with a target agent (i. e., a demonstrator) to maximize the change in the observed behaviors of that agent.
M$^3$RL: Mind-aware Multi-agent Management Reinforcement Learning
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward.
Where and Why Are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks
We built a new video dataset of tasks, intentions, and attention.
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills.
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition
This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities.
Ranked #11 on
Group Activity Recognition
on Volleyball
Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI).
Modeling and Inferring Human Intents and Latent Functional Objects for Trajectory Prediction
This paper is about detecting functional objects and inferring human intentions in surveillance videos of public spaces.
Learning Social Affordance for Human-Robot Interaction
In this paper, we present an approach for robot learning of social affordance from human activity videos.
Joint Inference of Groups, Events and Human Roles in Aerial Videos
This paper addresses a new problem of parsing low-resolution aerial videos of large spatial areas, in terms of 1) grouping, 2) recognizing events and 3) assigning roles to people engaged in events.
