Search Results for author:
Found 17 papers, 8 papers with code
MarioQA: Answering Questions by Watching Gameplay Videos
To address this objective, we automatically generate a customized synthetic VideoQA dataset using {\em Super Mario Bros.} gameplay videos so that it contains events with different levels of reasoning complexity.
Text-guided Attention Model for Image Captioning
Visual attention plays an important role to understand images and demonstrates its effectiveness in generating natural language descriptions of images.
Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance.
Transfer Learning via Unsupervised Task Discovery for Visual Question Answering
Specifically, we employ linguistic knowledge sources such as structured lexical database (e. g. WordNet) and visual descriptions for unsupervised task discovery, and transfer a learned task conditional visual classifier as an answering unit in a visual question answering model.
Learning to Specialize with Knowledge Distillation for Visual Question Answering
The proposed framework is model-agnostic and applicable to any tasks other than VQA, e. g., image classification with a large number of labels but few per-class examples, which is known to be difficult under existing MCL schemes.
Streamlined Dense Video Captioning
Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events.
Towards Oracle Knowledge Distillation with Neural Architecture Search
We present a novel framework of knowledge distillation that is capable of learning powerful and efficient student models from ensemble teacher networks.
Local-Global Video-Text Interactions for Temporal Grounding
This paper addresses the problem of text-to-video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query.
Boundary-aware Pre-training for Video Scene Segmentation
Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks.
Winning the ICCV'2021 VALUE Challenge: Task-aware Ensemble and Transfer Learning with Visual Concepts
The VALUE (Video-And-Language Understanding Evaluation) benchmark is newly introduced to evaluate and analyze multi-modal representation learning algorithms on three video-and-language tasks: Retrieval, QA, and Captioning.
Boundary-aware Self-supervised Learning for Video Scene Segmentation
Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks.
MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection
Human-Object Interaction (HOI) detection is the task of identifying a set of <human, object, interaction> triplets from an image.
Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs
Existing open-world segmentation methods have shown impressive advances by employing contrastive learning (CL) to learn diverse visual concepts and transferring the learned image-level understanding to the segmentation task.
Ranked #2 on
Semantic Segmentation
on CC3M-TagMask
Contrastive Learning
Open Vocabulary Semantic Segmentation
+4
Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning
Image captioning is one of the straightforward tasks that can take advantage of large-scale web-crawled data which provides rich knowledge about the visual world for a captioning model.
NICE: CVPR 2023 Challenge on Zero-shot Image Captioning
In this report, we introduce NICE (New frontiers for zero-shot Image Captioning Evaluation) project and share the results and outcomes of 2023 challenge.
Learning Pseudo-Labeler beyond Noun Concepts for Open-Vocabulary Object Detection
Specifically, the proposed method aims to learn arbitrary image-to-text mapping for pseudo-labeling of arbitrary concepts, named Pseudo-Labeling for Arbitrary Concepts (PLAC).
Honeybee: Locality-enhanced Projector for Multimodal LLM
In Multimodal Large Language Models (MLLMs), a visual projector plays a crucial role in bridging pre-trained vision encoders with LLMs, enabling profound visual understanding while harnessing the LLMs' robust capabilities.
Ranked #1 on
Science Question Answering
on ScienceQA
(using extra training data)
