Search Results for author:
Found 24 papers, 9 papers with code
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens.
Ranked #2 on
Referring Expression Segmentation
on PhraseCut
Causal Language Modeling
Generalized Referring Expression Segmentation
+4
Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning
In this work, we tackle the problem of training a robot to understand multimodal prompts, interleaving vision signals with text descriptions.
Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AI
The Alexa Prize program has empowered numerous university students to explore, experiment, and showcase their talents in building conversational agents through challenges like the SocialBot Grand Challenge and the TaskBot Challenge.
Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
Offline reinforcement learning (RL) methods strike a balance between exploration and exploitation by conservative value estimation -- penalizing values of unseen states and actions.
LEMMA: Learning Language-Conditioned Multi-Robot Manipulation
We introduce a benchmark for LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) focused on task allocation and long-horizon object manipulation based on human language instructions in a tabletop setting.
Alexa Arena: A User-Centric Interactive Platform for Embodied AI
We introduce Alexa Arena, a user-centric simulation platform for Embodied AI (EAI) research.
Language-Informed Transfer Learning for Embodied Household Activities
For service robots to become general-purpose in everyday household environments, they need not only a large library of primitive skills, but also the ability to quickly learn novel tasks specified by users.
OpenD: A Benchmark for Language-Driven Door and Drawer Opening
We introduce OPEND, a benchmark for learning how to use a hand to open cabinet doors or drawers in a photo-realistic and physics-reliable simulation environment driven by language instruction.
CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning
We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems
Our proposed method allows a single transformer model to directly walk on a large-scale knowledge graph to generate responses.
DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following
Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural language command to the agent, and the agent can only follow the command passively.
Learning to Act with Affordance-Aware Multimodal Neural SLAM
With the proposed Affordance-aware Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement over prior published work on the ALFRED benchmark and set a new state-of-the-art generalization performance at a success rate of 23. 48% on the test unseen scenes.
Best of Both Worlds: A Hybrid Approach for Multi-Hop Explanation with Declarative Facts
Language-enabled AI systems can answer complex, multi-hop questions to high accuracy, but supporting answers with evidence is a more challenging task which is important for the transparency and trustworthiness to users.
LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
However, current simulators for Embodied AI (EAI) challenges only provide simulated indoor scenes with a limited number of layouts.
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding
However, evaluations only based on end task performance shed little light on machines' true ability in language understanding and reasoning.
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
Language-guided robots performing home and office tasks must navigate in and interact with the world.
Are We There Yet? Learning to Localize in Embodied Instruction Following
Embodied instruction following is a challenging problem requiring an agent to infer a sequence of primitive actions to achieve a goal environment state from complex language and visual inputs.
Interactive Teaching for Conversational AI
Current conversational AI systems aim to understand a set of pre-designed requests and execute related actions, which limits them to evolve naturally and adapt based on human interactions.
Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
In the NLP community, recent years have seen a surge of research activities that address machines' ability to perform deep language understanding which goes beyond what is explicitly stated in text, rather relying on reasoning and knowledge of the world.
Commonsense Justification for Action Explanation
To enable collaboration and communication between humans and agents, this paper investigates learning to acquire commonsense evidence for action justification.
What Action Causes This? Towards Naive Physical Action-Effect Prediction
Despite recent advances in knowledge representation, automated reasoning, and machine learning, artificial agents still lack the ability to understand basic action-effect relations regarding the physical world, for example, the action of cutting a cucumber most likely leads to the state where the cucumber is broken apart into smaller pieces.
Interactive Learning of State Representation through Natural Language Instruction and Explanation
One significant simplification in most previous work on robot learning is the closed-world assumption where the robot is assumed to know ahead of time a complete set of predicates describing the state of the physical world.
