Search Results for author:
Found 23 papers, 11 papers with code
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
On the other hand, there are also many existing models and systems (symbolic-based or neural-based) that can do some domain-specific tasks very well.
MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must.
An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022
This technical report describes the CONE approach for Ego4D Natural Language Queries (NLQ) Challenge in ECCV 2022.
HORIZON: A High-Resolution Panorama Synthesis Framework
Panorama synthesis aims to generate a visual scene with all 360-degree views and enables an immersive virtual world.
CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding
Analysis reveals the effectiveness of components and higher efficiency in long video grounding as our system improves the inference speed by 2x on Ego4d-NLQ and 15x on MAD while keeping the SOTA performance of CONE.
ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
This paper proposes a fusion model named ScaleVLAD to gather multi-Scale representation from text, video, and audio with shared Vectors of Locally Aggregated Descriptors to improve unaligned multimodal sentiment analysis.
Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
To consider the interdependent knowledge between contextual clips into the network inference, we propose a Siamese Sampling and Reasoning (SiaSamRea) approach, which consists of a siamese sampling mechanism to generate sparse and similar clips (i. e., siamese clips) from the same video, and a novel reasoning strategy for integrating the interdependent knowledge between contextual clips into the network.
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively.
Ranked #1 on
Text-to-Video Generation
on Kinetics
Hybrid Reasoning Network for Video-based Commonsense Captioning
Firstly, we develop multi-commonsense learning for semantic-level reasoning by jointly training different commonsense types in a unified network, which encourages the interaction between the clues of multiple commonsense descriptions, event-wise captions and videos.
Hierarchical Context-aware Network for Dense Video Event Captioning
Dense video event captioning aims to generate a sequence of descriptive captions for each event in a long untrimmed video.
Control Image Captioning Spatially and Temporally
Moreover, the controllability and explainability of LoopCAG are validated by analyzing spatial and temporal sensitivity during the generation process.
Ranked #1 on
Image Captioning
on Localized Narratives
GEM: A General Evaluation Benchmark for Multimodal Tasks
Comparing with existing multimodal datasets such as MSCOCO and Flicker30K for image-language tasks, YouCook2 and MSR-VTT for video-language tasks, GEM is not only the largest vision-language dataset covering image-language tasks and video-language tasks at the same time, but also labeled in multiple languages.
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation.
Ranked #4 on
Text-to-Video Generation
on MSR-VTT
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner.
Ranked #1 on
Text to Video Retrieval
on MSR-VTT
GRACE: Gradient Harmonized and Cascaded Labeling for Aspect-based Sentiment Analysis
Specifically, a cascaded labeling module is developed to enhance the interchange between aspect terms and improve the attention of sentiment tokens when labeling sentiment polarities.
Tag and Correct: Question aware Open Information Extraction with Two-stage Decoding
The semi-structured answer has two advantages which are more readable and falsifiable compared to span answer.
A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Watching instructional videos are often used to learn about procedures.
XGPT: Cross-modal Generative Pre-Training for Image Captioning
While many BERT-based cross-modal pre-trained models produce excellent results on downstream understanding tasks like image-text retrieval and VQA, they cannot be applied to generation tasks directly.
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks.
Ranked #1 on
Action Segmentation
on COIN
(using extra training data)
Knowledge Aware Semantic Concept Expansion for Image-Text Matching
In this paper, we develop a Scene Concept Graph (SCG) by aggregating image scene graphs and extracting frequently co-occurred concept pairs as scene common-sense knowledge.
Dense Procedure Captioning in Narrated Instructional Videos
Understanding narrated instructional videos is important for both research and real-world web applications.
R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
To better utilize semantic knowledge in images, we propose a novel framework to learn visual relation facts for VQA.
