Search Results for author:
Found 10 papers, 4 papers with code
HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue
Video-grounded Dialogue (VGD) aims to answer questions regarding a given multi-modal input comprising video, audio, and dialogue history.
Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue
Despite the recent success of multi-modal reasoning to generate answer sentences, existing dialogue systems still suffer from a text hallucination problem, which denotes indiscriminate text-copying from input texts without an understanding of the question.
Selective Query-guided Debiasing for Video Corpus Moment Retrieval
Video moment retrieval (VMR) aims to localize target moments in untrimmed videos pertinent to a given textual query.
Scalable SoftGroup for 3D Instance Segmentation on Point Clouds
Furthermore, SoftGroup can be extended to perform object detection and panoptic segmentation with nontrivial improvements over existing methods.
Structured Co-reference Graph Attention for Video-grounded Dialogue
A video-grounded dialogue system referred to as the Structured Co-reference Graph Attention (SCGA) is presented for decoding the answer sequence to a question regarding a given video while keeping track of the dialogue context.
VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval
This paper explores methods for performing VMR in a weakly-supervised manner (wVMR): training is performed without temporal moment labels but only with the text query that describes a segment of the video.
Modality Shifting Attention Network for Multi-modal Video Question Answering
To this end, MSAN is based on (1) the moment proposal network (MPN) that attempts to locate the most appropriate temporal moment from each of the modalities, and also on (2) the heterogeneous reasoning network (HRN) that predicts the answer using an attention mechanism on both modalities.
Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering
This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering.
Progressive Attention Memory Network for Movie Story Question Answering
To overcome these challenges, PAMN involves three main features: (1) progressive attention mechanism that utilizes cues from both question and answer to progressively prune out irrelevant temporal parts in memory, (2) dynamic modality fusion that adaptively determines the contribution of each modality for answering the current question, and (3) belief correction answering scheme that successively corrects the prediction score on each candidate answer.
Ranked #2 on
Video Story QA
on MovieQA
Pivot Correlational Neural Network for Multimodal Video Categorization
The architecture is trained to maximizes the correlation between the hidden states as well as the predictions of the modal-agnostic pivot stream and modal-specific stream in the network.
