Search Results for author:
Found 40 papers, 13 papers with code
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users
For instance, real-world human instructions can be ambiguous, require different levels of AI assistance, and may evolve over time, reflecting changes in the user's mental state.
AutoToM: Automated Bayesian Inverse Planning and Model Discovery for Open-ended Theory of Mind
Based on the uncertainty of the inference, it iteratively refines the model, by introducing additional mental variables and/or incorporating more timesteps in the context.
GenEx: Generating an Explorable World
In this work, we take a step toward this goal by introducing GenEx, a system capable of planning complex embodied world exploration, guided by its generative imagination that forms priors (expectations) about the surrounding environments.
Generative World Explorer
Our experimental results demonstrate that (1) $\textit{Genex}$ can generate high-quality and consistent observations during long-horizon exploration of a large virtual physical world and (2) the beliefs updated with the generated observations can inform an existing decision-making model (e. g., an LLM agent) to make better plans.
Few-Shot Task Learning through Inverse Generative Modeling
Then, given a few demonstrations of a new concept (such as a new goal or a new action), our method learns the underlying concepts through backpropagation without updating the model weights, thanks to the invertibility of the generative model.
Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
In CHAIC, the goal is for an embodied agent equipped with egocentric observations to assist a human who may be operating under physical constraints -- e. g., unable to reach high places or confined to a wheelchair -- in performing common household or outdoor tasks as efficiently as possible.
SIFToM: Robust Spoken Instruction Following through Theory of Mind
We present a cognitively inspired model, Speech Instruction Following through Theory of Mind (SIFToM), to enable robots to pragmatically follow human instructions under diverse speech conditions by inferring the human's goal and joint plan as prior for speech perception and understanding.
MuMA-ToM: Multi-modal Multi-Agent Theory of Mind
To truly understand how and why people interact with one another, we must infer the underlying mental states that give rise to the social interactions, i. e., Theory of Mind reasoning in multi-agent interactions.
Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
We derive an approach for learning from these feature-level preferences, both for cases where users specify which features are reward-relevant, and when users do not.
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
To effectively plan in this setting, in contrast to learning world dynamics in a single-agent scenario, we must simulate world dynamics conditioned on an arbitrary number of agents' actions given only partial egocentric visual observations of the world.
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment
GOMA formulates verbal communication as a planning problem that minimizes the misalignment between the parts of agents' mental states that are relevant to the goals.
MMToM-QA: Multimodal Theory of Mind Question Answering
To engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models).
Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning
Despite their tremendous success in many applications, large language models often fall short of consistent reasoning and planning in various (language, embodied, and social) scenarios, due to inherent limitations in their inference, learning, and modeling capabilities.
The Cultural Psychology of Large Language Models: Is ChatGPT a Holistic or Analytic Thinker?
In our research, we probed the cultural cognitive traits of ChatGPT.
Neural Amortized Inference for Nested Multi-agent Reasoning
Multi-agent interactions, such as communication, teaching, and bluffing, often rely on higher-order social inference, i. e., understanding how others infer oneself.
Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation
Policies often fail due to distribution shift -- changes in the state and reward that occur when a policy is deployed in new environments.
Building Cooperative Embodied Agents Modularly with Large Language Models
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments.
Language Models Meet World Models: Embodied Experiences Enhance Language Models
While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household activities.
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Experiments show that our helper agent robustly updates its goal inference and adapts its helping plans to the changing level of uncertainty.
Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning
To address this, we propose a reward learning approach, Graph-based Equivalence Mappings (GEM), that can discover spatial goal representations that are aligned with the intended goal specification, enabling successful generalization in unseen environments.
Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
We formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
RLPrompt formulates a parameter-efficient policy network that generates the desired discrete prompt after training with reward.
Coordinating Policies Among Multiple Agents via an Intelligent Communication Channel
In Multi-Agent Reinforcement Learning (MARL), specialized channels are often introduced that allow agents to communicate directly with one another.
Intelligent Communication
Multi-agent Reinforcement Learning
+2
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Our experiments with human participants demonstrate that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground truth.
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation.
AGENT: A Benchmark for Core Psychological Reasoning
For machine agents to successfully interact with humans in real-world settings, they will need to develop an understanding of human mental life.
Ranked #1 on
Core Psychological Reasoning
on AGENT
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents.
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Thus, in this paper, we propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations, where the robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications based on its online Bayesian inference of the user's mental state.
Bayesian Inference
Explainable Artificial Intelligence (XAI)
+1
Active Visual Information Gathering for Vision-Language Navigation
Vision-language navigation (VLN) is the task of entailing an agent to carry out navigational instructions inside photo-realistic environments.
M^3RL: Mind-aware Multi-agent Management Reinforcement Learning
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward.
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for training and testing AI agents.
Interactive Agent Modeling by Learning to Probe
In particular, the probing agent (i. e. a learner) learns to interact with the environment and with a target agent (i. e., a demonstrator) to maximize the change in the observed behaviors of that agent.
M$^3$RL: Mind-aware Multi-agent Management Reinforcement Learning
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward.
Where and Why Are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks
We built a new video dataset of tasks, intentions, and attention.
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills.
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition
This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities.
Ranked #12 on
Group Activity Recognition
on Volleyball
Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI).
Modeling and Inferring Human Intents and Latent Functional Objects for Trajectory Prediction
This paper is about detecting functional objects and inferring human intentions in surveillance videos of public spaces.
Learning Social Affordance for Human-Robot Interaction
In this paper, we present an approach for robot learning of social affordance from human activity videos.
Joint Inference of Groups, Events and Human Roles in Aerial Videos
This paper addresses a new problem of parsing low-resolution aerial videos of large spatial areas, in terms of 1) grouping, 2) recognizing events and 3) assigning roles to people engaged in events.
