Search Results for author:
Found 15 papers, 5 papers with code
MuMA-ToM: Multi-modal Multi-Agent Theory of Mind
To truly understand how and why people interact with one another, we must infer the underlying mental states that give rise to the social interactions, i. e., Theory of Mind reasoning in multi-agent interactions.
Baba Is AI: Break the Rules to Beat the Benchmark
Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives.
Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention
We introduce the Gaze-guided Action Anticipation algorithm, which establishes a visual-semantic graph from the video input.
A Transformer-Based Model for the Prediction of Human Gaze Behavior on Videos
Eye-tracking applications that utilize the human gaze in video understanding tasks have become increasingly important.
MMToM-QA: Multimodal Theory of Mind Question Answering
To engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models).
Neural Amortized Inference for Nested Multi-agent Reasoning
Multi-agent interactions, such as communication, teaching, and bluffing, often rely on higher-order social inference, i. e., understanding how others infer oneself.
NashFormer: Leveraging Local Nash Equilibria for Semantically Diverse Trajectory Prediction
Interactions between road agents present a significant challenge in trajectory prediction, especially in cases involving multiple agents.
Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation
This task requires an understanding of the spatio-temporal context formed by past actions on objects, coined action context.
Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors
We further qualitatively evaluate the effectiveness of our method on real images and demonstrate its generalizability towards interaction types and object categories.
Trajectory Prediction with Linguistic Representations
Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions.
Learning a natural-language to LTL executable semantic parser for grounded robotics
We take a step toward robots that can do the same by training a grounded semantic parser, which discovers latent linguistic representations that can be used for the execution of natural-language commands.
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding
Recent work has shown that while deep networks can mimic some human language abilities when presented with novel sentences, systematic variation uncovers the limitations in the language-understanding abilities of networks.
Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas
We demonstrate a reinforcement learning agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions.
Investigating the Decoders of Maximum Likelihood Sequence Models: A Look-ahead Approach
We evaluate our look-ahead module on three datasets of varying difficulties: IM2LATEX-100k OCR image to LaTeX, WMT16 multimodal machine translation, and WMT14 machine translation.
Multimodal Machine Translation
Optical Character Recognition (OCR)
+2
Deep compositional robotic planners that follow natural language commands
We demonstrate how a sampling-based robotic planner can be augmented to learn to understand a sequence of natural language commands in a continuous configuration space to move and manipulate objects.
