Search Results for author:
Found 16 papers, 9 papers with code
Investigating Information-Theoretic Properties of the Typology of Spatial Demonstratives
Using data from Nintemann et al. (2020), we explore the variability in complexity and informativity across spatial demonstrative systems using spatial deictic lexicons from 223 languages.
NTIRE 2024 Challenge on Image Super-Resolution ($\times$4): Methods and Results
This paper reviews the NTIRE 2024 challenge on image super-resolution ($\times$4), highlighting the solutions proposed and the outcomes obtained.
Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation
In this paper, we introduce CRFormer, a model that iteratively calibrates multi-modal features in the transformer decoder.
VL-Mamba: Exploring State Space Models for Multimodal Learning
The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.
Ranked #63 on
Visual Question Answering
on MM-Vet
GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER
Specifically, we propose a video auto-encoder, where a video encoder encodes videos into global features, and a video decoder, built on a diffusion model, decodes the global features and synthesizes video frames in a non-autoregressive manner.
EAVL: Explicitly Align Vision and Language for Referring Image Segmentation
In previous approaches, fused vision-language features are directly fed into a decoder and pass through a convolution with a fixed kernel to obtain the result, which follows a similar pattern as traditional image segmentation.
COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations.
Ranked #1 on
TGIF-Frame
on TGIF-QA
(using extra training data)
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA).
Ranked #1 on
Image Captioning
on COCO Captions
(SPICE metric, using extra
training data)
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
We show that only language-paired two-modality data is sufficient to connect all modalities.
VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending
Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature Adapting and Blending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks.
Ranked #1 on
Visual Question Answering (VQA)
on MSVD-QA
(using extra training data)
Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner
In this paper, we propose a novel method called Joint QA and DC GEneration (JADE), which utilizes a pre-trained multimodal model and easily-crawled image-text pairs to automatically generate and filter large-scale VQA and dense captioning datasets.
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner.
Ranked #1 on
Video Captioning
on VATEX
(using extra training data)
Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
In this work, we concentrate on a rarely investigated problem of text guided sounding video generation and propose the Sounding Video Generator (SVG), a unified framework for generating realistic videos along with audio signals.
TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving
The next-generation high-resolution automotive radar (4D radar) can provide additional elevation measurement and denser point clouds, which has great potential for 3D sensing in autonomous driving.
CPTR: Full Transformer Network for Image Captioning
Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.
Global-Local Propagation Network for RGB-D Semantic Segmentation
Depth information matters in RGB-D semantic segmentation task for providing additional geometric information to color images.
