Search Results for author:
Found 25 papers, 7 papers with code
Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AI
The Alexa Prize program has empowered numerous university students to explore, experiment, and showcase their talents in building conversational agents through challenges like the SocialBot Grand Challenge and the TaskBot Challenge.
Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
Offline reinforcement learning (RL) methods strike a balance between exploration and exploitation by conservative value estimation -- penalizing values of unseen states and actions.
LEMMA: Learning Language-Conditioned Multi-Robot Manipulation
We introduce a benchmark for LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) focused on task allocation and long-horizon object manipulation based on human language instructions in a tabletop setting.
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models
Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network.
Alexa Arena: A User-Centric Interactive Platform for Embodied AI
We introduce Alexa Arena, a user-centric simulation platform for Embodied AI (EAI) research.
Language-Informed Transfer Learning for Embodied Household Activities
For service robots to become general-purpose in everyday household environments, they need not only a large library of primitive skills, but also the ability to quickly learn novel tasks specified by users.
GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods
A key goal for the advancement of AI is to develop technologies that serve the needs not just of one group but of all communities regardless of their geographical region.
OpenD: A Benchmark for Language-Driven Door and Drawer Opening
We introduce OPEND, a benchmark for learning how to use a hand to open cabinet doors or drawers in a photo-realistic and physics-reliable simulation environment driven by language instruction.
TPA-Net: Generate A Dataset for Text to Physics-based Animation
Recent breakthroughs in Vision-Language (V&L) joint research have achieved remarkable results in various text-driven tasks.
Towards Reasoning-Aware Explainable VQA
Our method generates human-readable textual explanations while maintaining SOTA VQA accuracy on the GQA-REX (77. 49%) and VQA-E (71. 48%) datasets.
CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning
We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
A Multi-level Alignment Training Scheme for Video-and-Language Grounding
To solve video-and-language grounding tasks, the key is for the network to understand the connection between the two modalities.
DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following
Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural language command to the agent, and the agent can only follow the command passively.
Privacy Preserving Visual Question Answering
We introduce a novel privacy-preserving methodology for performing Visual Question Answering on the edge.
Learning to Act with Affordance-Aware Multimodal Neural SLAM
With the proposed Affordance-aware Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement over prior published work on the ALFRED benchmark and set a new state-of-the-art generalization performance at a success rate of 23. 48% on the test unseen scenes.
Learning Two-Step Hybrid Policy for Graph-Based Interpretable Reinforcement Learning
We present a two-step hybrid reinforcement learning (RL) policy that is designed to generate interpretable and robust hierarchical policies on the RL problem with graph-based input.
A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering
Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question.
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering
Most previous works address the problem by first fusing the image and question in the multi-modal space, which is inflexible for further fusion with a vast amount of external knowledge.
Ranked #19 on
Visual Question Answering (VQA)
on OK-VQA
Best of Both Worlds: A Hybrid Approach for Multi-Hop Explanation with Declarative Facts
Language-enabled AI systems can answer complex, multi-hop questions to high accuracy, but supporting answers with evidence is a more challenging task which is important for the transparency and trustworthiness to users.
LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
However, current simulators for Embodied AI (EAI) challenges only provide simulated indoor scenes with a limited number of layouts.
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
Language-guided robots performing home and office tasks must navigate in and interact with the world.
Learning Better Visual Dialog Agents with Pretrained Visual-Linguistic Representation
Most existing work for Guesser encode the dialog history as a whole and train the Guesser models from scratch on the GuessWhat?!
Are We There Yet? Learning to Localize in Embodied Instruction Following
Embodied instruction following is a challenging problem requiring an agent to infer a sequence of primitive actions to achieve a goal environment state from complex language and visual inputs.
Interactive Teaching for Conversational AI
Current conversational AI systems aim to understand a set of pre-designed requests and execute related actions, which limits them to evolve naturally and adapt based on human interactions.
LRTA: A Transparent Neural-Symbolic Reasoning Framework with Modular Supervision for Visual Question Answering
We show that LRTA makes a step towards truly understanding the question while the state-of-the-art model tends to learn superficial correlations from the training data.
