Search Results for author:
Found 9 papers, 7 papers with code
Character Grounding and Re-Identification in Story of Videos and Text Descriptions
We address character grounding and re-identification in multiple story-based videos like movies and associated text descriptions.
Dense 2D-3D Indoor Prediction with Sound via Aligned Cross-Modal Distillation
Sound can convey significant information for spatial reasoning in our daily lives.
Fusing Pre-Trained Language Models With Multimodal Prompts Through Reinforcement Learning
Language models are capable of commonsense reasoning: while domain-specific models can learn from explicit knowledge (e. g. commonsense graphs [6], ethical norms [25]), and larger models like GPT-3 manifest broad commonsense reasoning capacity.
Panoramic Vision Transformer for Saliency Detection in 360° Videos
360$^\circ$ video saliency detection is one of the challenging benchmarks for 360$^\circ$ video understanding since non-negligible distortion and discontinuity occur in the projection of any format of 360$^\circ$ videos, and capture-worthy viewpoint in the omnidirectional sphere is ambiguous by nature.
Multimodal Knowledge Alignment with Reinforcement Learning
Large language models readily adapt to novel settings, even without task-specific training data.
Pano-AVQA: Grounded Audio-Visual Question Answering on 360$^\circ$ Videos
However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings.
Transitional Adaptation of Pretrained Models for Visual Storytelling
In this work, we claim that a transitional adaptation task is required between pretraining and finetuning to harmonize the visual encoder and the language model for challenging downstream target tasks like visual storytelling.
Ranked #1 on
Visual Storytelling
on VIST
(ROUGE-L metric, using extra
training data)
Pano-AVQA: Grounded Audio-Visual Question Answering on 360deg Videos
However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings.
A Mobile Robot Generating Video Summaries of Seniors' Indoor Activities
We develop a system which generates summaries from seniors' indoor-activity videos captured by a social robot to help remote family members know their seniors' daily activities at home.
