Search Results for author:
Found 29 papers, 21 papers with code
UniAV: Unified Audio-Visual Perception for Multi-Task Video Localization
Video localization tasks aim to temporally locate specific instances in videos, including temporal action localization (TAL), sound event detection (SED) and audio-visual event localization (AVEL).
Video Understanding with Large Language Models: A Survey
With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly.
Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
Pre-trained vision-language models, e. g., CLIP, working with manually designed prompts have demonstrated great capacity of transfer learning.
Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
In this paper, we propose ViECap, a transferable decoding model that leverages entity-aware decoding to generate descriptions in both seen and unseen scenarios.
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks.
PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television Dramas
Art forms such as movies and television (TV) dramas are reflections of the real world, which have attracted much attention from the multimodal learning community recently.
LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning
This paper proposes an effective model LLMVA-GEBC (Large Language Model with Video Adapter for Generic Event Boundary Captioning): (1) We utilize a pretrained LLM for generating human-like captions with high quality.
Caption Anything: Interactive Image Description with Diverse Multimodal Controls
Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, $\textit{e. g.}$, looking at the specified regions or telling in a particular text style.
$π$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation
Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks.
Accelerating Vision-Language Pretraining with Free Language Modeling
FLM successfully frees the prediction rate from the tie-up with the corruption rate while allowing the corruption spans to be customized for each token to be predicted.
Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline
To better adapt to real-life applications, in this paper we focus on the task of dense-localizing audio-visual events, which aims to jointly localize and recognize all audio-visual events occurring in an untrimmed video.
Ranked #1 on
audio-visual event localization
on UnAV-100
Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos
Our framework is easily extensible to tasks covering visually-grounded language understanding and generation.
LANDMARK: Language-guided Representation Enhancement Framework for Scene Graph Generation
Besides, representation-level unbiased strategies endow LANDMARK the advantage of compatibility with other methods.
Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward
The existing method performs well at video segmentation stages but suffers from the problems of dependencies on extra cumbersome models and poor performance at the segment assemblage stage.
Exploiting Context Information for Generic Event Boundary Captioning
Previous methods only process the information of a single boundary at a time, which lacks utilization of video context information.
VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix
Existing vision-language pre-training (VLP) methods primarily rely on paired image-text datasets, which are either annotated by enormous human labors, or crawled from the internet followed by elaborate data cleaning techniques.
Transformer-Guided Convolutional Neural Network for Cross-View Geolocalization
Furthermore, we design a dual-branch Transformer head network to combine image features from multi-scale windows in order to improve details of the global feature representation.
Semantic-Aware Pretraining for Dense Video Captioning
This report describes the details of our approach for the event dense-captioning task in ActivityNet Challenge 2021.
Biasing Like Human: A Cognitive Bias Framework for Scene Graph Generation
Scene graph generation is a sophisticated task because there is no specific recognition pattern (e. g., "looking at" and "near" have no conspicuous difference concerning vision, whereas "near" could occur between entities with different morphology).
Ranked #1 on
Predicate Classification
on Visual Genome
(mean Recall @20 metric)
Auto-ABSA: Automatic Detection of Aspects in Aspect-Based Sentiment Analysis
In this paper, we proposed a method that uses an auxiliary sentence about aspects that the sentence contains to help sentiment prediction.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+2
A coarse-to-fine approach for dynamic-to-static image translation
Using the coarse predicted image, we explicitly infer a more accurate dynamic mask to identify both dynamic objects and their shadows, so that the task could be effectively converted to an image inpainting problem.
End-to-End Dense Video Captioning with Parallel Decoding
Dense video captioning aims to generate multiple associated captions with their temporal locations from the video.
Ranked #5 on
Dense Video Captioning
on YouCook2
Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images
Specifically, the dependency path is conducted based on the ResT, a novel Transformer backbone with memory-efficient multi-head self-attention, while the texture path is built on the stacked convolution operation.
Ranked #4 on
Semantic Segmentation
on UAVid
Multi-modal Visual Place Recognition in Dynamics-Invariant Perception Space
In this letter, we for the first time explore the use of multi-modal fusion of semantic and visual modalities in dynamics-invariant space to improve place recognition in dynamic environments.
Dense-Captioning Events in Videos: SYSU Submission to ActivityNet Challenge 2020
This technical report presents a brief description of our submission to the dense video captioning task of ActivityNet Challenge 2020.
Ranked #5 on
Dense Video Captioning
on ActivityNet Captions
Local Differential Privacy based Federated Learning for Internet of Things
To avoid the privacy threat and reduce the communication cost, in this paper, we propose to integrate federated learning and local differential privacy (LDP) to facilitate the crowdsourcing applications to achieve the machine learning model.
Reviewing and Improving the Gaussian Mechanism for Differential Privacy
Although both classical Gaussian mechanisms [1, 2] assume $0 < \epsilon \leq 1$, our review finds that many studies in the literature have used the classical Gaussian mechanisms under values of $\epsilon$ and $\delta$ where the added noise amounts of [1, 2] do not achieve $(\epsilon,\delta)$-DP.
Privacy-preserving Crowd-guided AI Decision-making in Ethical Dilemmas
To investigate such ethical dilemmas, recent studies have adopted preference aggregation, in which each voter expresses her/his preferences over decisions for the possible ethical dilemma scenarios, and a centralized system aggregates these preferences to obtain the winning decision.
A Survey of FPGA Based Deep Learning Accelerators: Challenges and Opportunities
With the rapid development of in-depth learning, neural network and deep learning algorithms have been widely used in various fields, e. g., image, video and voice processing.
