Search Results for author:
Found 23 papers, 8 papers with code
Polos: Multimodal Metric Learning from Human Feedback for Image Captioning
Establishing an automatic evaluation metric that closely aligns with human judgments is essential for effectively developing image captioning models.
Learning-To-Rank Approach for Identifying Everyday Objects Using a Physical-World Search Engine
Therefore, we focus on the task of retrieving target objects from open-vocabulary user instructions in a human-in-the-loop setting, which we define as the learning-to-rank physical objects (LTRPO) task.
DialMAT: Dialogue-Enabled Transformer with Moment-Based Adversarial Training
This paper focuses on the DialFRED task, which is the task of embodied instruction following in a setting where an agent can actively ask questions about the task.
JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models
Image captioning studies heavily rely on automatic evaluation metrics such as BLEU and METEOR.
Fully Automated Task Management for Generation, Execution, and Evaluation: A Framework for Fetch-and-Carry Tasks with Natural Language Instructions in Continuous Space
This paper aims to develop a framework that enables a robot to execute tasks based on visual information, in response to natural language instructions for Fetch-and-Carry with Object Grounding (FCOG) tasks.
Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions
In this study, we aim to develop a model that comprehends a natural language instruction (e. g., "Go to the living room and get the nearest pillow to the radio art on the wall") and generates a segmentation mask for the target everyday object.
Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks
The results show that our method outperforms the baseline method in terms of language comprehension accuracy.
Prototypical Contrastive Transfer Learning for Multimodal Language Understanding
Most conventional models have been trained on real-world datasets that are labor-intensive to collect, and they have not fully leveraged simulation data through a transfer learning framework.
Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Query
The decoder in AQT utilizes action queries, which represent the information of each action, as queries.
Relational Future Captioning Model for Explaining Likely Collisions in Daily Tasks
Domestic service robots that support daily tasks are a promising solution for elderly or disabled people.
Moment-based Adversarial Training for Embodied Language Comprehension
This is challenging because the robot needs to break down the instruction sentences into subgoals and execute them in the correct order.
LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation
Previous models for this task successfully generate images iteratively, given a sequence of instructions and a previously generated image.
Ranked #1 on
Text-to-Image Generation
on GeNeVA (CoDraw)
Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots
Currently, domestic service robots have an insufficient ability to interact naturally through language.
Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue
In addition, we build a goal-oriented visual dialogue task called CLEVR Ask.
Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning
A3C consists of a feature extractor that extracts features from an image, a policy branch that outputs the policy, and a value branch that outputs the state value.
CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation
Navigation guided by natural language instructions is particularly suitable for Domestic Service Robots that interacts naturally with users.
Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations
Thus, inferring the collision-risk before a placing motion is crucial for achieving the requested task.
Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder-Decoder Network
Domestic service robots (DSRs) are a promising solution to the shortage of home care workers.
A Multimodal Target-Source Classifier with Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects
To solve such a task, we propose the multimodal target-source classifier model with attention branches (MTCM-AB), which is an extension of the MTCM.
Multimodal Attention Branch Network for Perspective-Free Sentence Generation
In this paper, we address the automatic sentence generation of fetching instructions for domestic service robots.
A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks from Ambiguous Language Instructions
This paper focuses on a multimodal language understanding method for carry-and-place tasks with domestic service robots.
Grounded Language Understanding for Manipulation Instructions Using GAN-Based Classification
The target task of this study is grounded language understanding for domestic service robots (DSRs).
