Search Results for author:
Found 17 papers, 10 papers with code
Social Debiasing for Fair Multi-modal LLMs
Multi-modal Large Language Models (MLLMs) have advanced significantly, offering powerful vision-language understanding capabilities.
Hummer: Towards Limited Competitive Preference Dataset
Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback.
SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval
In particular, text-video retrieval, which aims to find the top matching videos given text descriptions from a vast video corpus, is an essential function, the primary challenge of which is to bridge the modality gap.
M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval
We then summarize this empirical study into the M2-RAAP recipe, where our technical contributions lie in 1) the data filtering and text re-writing pipeline resulting in 1M high-quality bilingual video-text pairs, 2) the replacement of video inputs with key-frames to accelerate pre-training, and 3) the Auxiliary-Caption-Guided (ACG) strategy to enhance video features.
SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
By employing one shared BERT-type network to refine textual and cross-modal features simultaneously, SNP is lightweight and could support various downstream applications.
M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining
Vision-language foundation models like CLIP have revolutionized the field of artificial intelligence.
Ranked #1 on
Zero-Shot Transfer Image Classification
on ImageNet
(using extra training data)
Zero-Shot Cross-Modal Retrieval
Zero-shot Image Retrieval
+3
Knowledge-enhanced Multi-perspective Video Representation Learning for Scene Recognition
With the explosive growth of video data in real-world applications, a comprehensive representation of videos becomes increasingly important.
SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment
To achieve multimodal alignment from both global and local perspectives, this paper proposes Symmetrizing Contrastive Captioners (SyCoCa), which introduces bidirectional interactions on images and texts across the global and local representation levels.
Text as Image: Learning Transferable Adapter for Multi-Label Classification
Pre-trained vision-language models have notably accelerated progress of open-world concept recognition.
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
Specifically, we present a new method for constructing the instruction tuning dataset at a low cost by leveraging annotations in existing datasets.
Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learning
We thereby present a new Triplet Partial Margin Contrastive Learning (TPM-CL) module to construct partial order triplet samples by automatically generating fine-grained hard negatives for matched text-video pairs.
Ranked #4 on
Video Retrieval
on MSR-VTT-1kA
EVE: Efficient zero-shot text-based Video Editing with Depth Map Guidance and Temporal Consistency Constraints
Compared with images, we conjecture that videos necessitate more constraints to preserve the temporal consistency during editing.
Temporal Sentence Grounding in Streaming Videos
The goal of TSGSV is to evaluate the relevance between a video stream and a given sentence query.
Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input
They can achieve exceptional performances on specific tasks, but face a particularly challenging problem of modality mismatch because of diversity of input modalities and their fixed structures.
Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval
For traditional model upgrades, the old model will not be replaced by the new one until the embeddings of all the images in the database are re-computed by the new model, which takes days or weeks for a large amount of data.
CNVid-3.5M: Build, Filter, and Pre-Train the Large-Scale Public Chinese Video-Text Dataset
Though there are certain methods studying the Chinese video-text pre-training, they pre-train their models on private datasets whose videos and text are unavailable.
LPSNet: A Lightweight Solution for Fast Panoptic Segmentation
Panoptic segmentation is a challenging task aiming to simultaneously segment objects (things) at instance level and background contents (stuff) at semantic level.
