Search Results for author:
Found 66 papers, 35 papers with code
Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach
Self-improvement in multimodal large language models (MLLMs) is crucial for enhancing their reliability and robustness.
CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs
Large Vision-Language Model (LVLM) systems have demonstrated impressive vision-language reasoning capabilities but suffer from pervasive and severe hallucination issues, posing significant risks in critical domains such as healthcare and autonomous systems.
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
In this paper, we introduce Motion-Grounded Video Reasoning, a new motion understanding task that requires generating visual answers (video segmentation masks) according to the input question, and hence needs implicit spatiotemporal reasoning and grounding.
SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering
Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos.
Audio-visual Question Answering
Audio-Visual Question Answering (AVQA)
+2
Continual Audio-Visual Sound Separation
The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks to mitigate catastrophic forgetting during audio-visual continual learning.
Scaling Concept With Text-Guided Diffusion Models
Through an empirical study, we identify a trend where concepts can be decomposed in text-guided diffusion models.
CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIP
Machine unlearning (MU) has gained significant attention as a means to remove specific data from trained models without requiring a full retraining process.
Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation
Secondly, we introduce a cross-modal semantic enhancement approach.
DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures
Audio-driven talking video generation has advanced significantly, but existing methods often depend on video-to-video translation techniques and traditional generative networks like GANs and they typically generate taking heads and co-speech gestures separately, leading to less coherent outputs.
Semantic Grouping Network for Audio Source Separation
They extracted high-level semantics from visual inputs as the guidance to help disentangle sound representation for individual sources.
AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation
This shared backbone facilitates both audio and video generation.
MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers
Furthermore, to suppress the background features in each modality from foreground matched audio-visual features, we introduce a robust discriminative foreground mining scheme.
Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generation
Traditional diffusion transformers (DiT), which utilize self-attention blocks, are effective but their computational complexity scales quadratically with the input length, limiting their use for high-resolution images.
SignLLM: Sign Language Production Large Language Models
In this paper, we propose SignLLM, a multilingual Sign Language Production (SLP) large language model, which includes two novel multilingual SLP modes MLSF and Prompt2LangGloss that allow sign language gestures generation from query texts input and question-style prompts input respectively.
T-VSL: Text-Guided Visual Sound Source Localization in Mixtures
Visual sound source localization poses a significant challenge in identifying the semantic region of each sounding source within a video.
Robust Active Speaker Detection in Noisy Environments
Experiments demonstrate that non-speech audio noises significantly impact ASD models, and our proposed approach improves ASD performance in noisy environments.
Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition
To pave the way for further research on this new problem, we intensively explored leveraging foundation models and multimodal large language models across different modalities.
Text-to-Audio Generation Synchronized with Videos
Extensive evaluations on the AudioCaps and T2AV-Bench demonstrate that our T2AV sets a new standard for video-aligned TTA generation in ensuring visual alignment and temporal consistency.
OSCaR: Object State Captioning and State Change Representation
To address these challenges, in this paper, we introduce the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark.
Efficiently Leveraging Linguistic Priors for Scene Text Spotting
This paper proposes a method that leverages linguistic knowledge from a large text corpus to replace the traditional one-hot encoding used in auto-regressive scene text spotting and recognition models.
DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation
The generation of emotional talking faces from a single portrait image remains a significant challenge.
LAVSS: Location-Guided Audio-Visual Spatial Audio Separation
LAVSS is inspired by the correlation between spatial audio and visual location.
Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation
The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view.
Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields
Room impulse response (RIR), which measures the sound propagation within an environment, is critical for synthesizing high-fidelity audio for a given environment.
CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction
However, a limitation of CMR is its slow imaging speed, which causes patient discomfort and introduces artifacts in the images.
Class-Incremental Grouping Network for Continual Audio-Visual Learning
Our CIGN leverages learnable audio-visual class tokens and audio-visual grouping to continually aggregate class-aware features.
SignDiff: Diffusion Models for American Sign Language Production
In this paper, we propose a dual-condition diffusion pre-training model named SignDiff that can generate human sign language speakers from a skeleton pose.
DiffI2I: Efficient Diffusion Model for Image-to-Image Translation
Compared to traditional DMs, the compact IPR enables DiffI2I to obtain more accurate outcomes and employ a lighter denoising network and fewer iterations.
Audio-Visual Class-Incremental Learning
We demonstrate that joint audio-visual modeling can improve class-incremental learning, but current methods fail to preserve semantic similarity between audio and visual features as incremental step grows.
High-Quality Visually-Guided Sound Separation from Diverse Categories
We compare DAVIS to existing state-of-the-art discriminative audio-visual separation methods on the AVE and MUSIC datasets, and results show that DAVIS outperforms other methods in separation quality, demonstrating the advantages of our framework for tackling the audio-visual source separation task.
Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI
Limited by imaging systems, the reconstruction of Magnetic Resonance Imaging (MRI) images from partial measurement is essential to medical imaging research.
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect.
EgoVSR: Towards High-Quality Egocentric Video Super-Resolution
We explicitly tackle motion blurs in egocentric videos using a Dual Branch Deblur Network (DB$^2$Net) in the VSR framework.
DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
In this work, we propose a novel and personalized text-to-sound generation approach with visual alignment based on latent diffusion models, namely DiffAVA, that can simply fine-tune lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition.
AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation
In this work, we propose a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio.
Audio-Visual Grouping Network for Sound Localization from Mixtures
Sound source localization is a typical and challenging task that predicts the location of sound sources in a video.
Egocentric Audio-Visual Object Localization
In this paper, we explore the challenging egocentric audio-visual object localization task and observe that 1) egomotion commonly exists in first-person recordings, even within a short duration; 2) The out-of-view sound components can be created while wearers shift their attention.
DiffIR: Efficient Diffusion Model for Image Restoration
Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network.
Knowledge Distillation based Degradation Estimation for Blind Super-Resolution
It consists of a knowledge distillation based implicit degradation estimator network (KD-IDE) and an efficient SR network.
Basic Binary Convolution Unit for Binarized Image Restoration Network
In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks.
Learning in Audio-visual Context: A Review, Analysis, and New Perspective
A comprehensive survey that can systematically organize and analyze studies of the audio-visual field is expected.
Meta-Learning based Degradation Representation for Blind Super-Resolution
The most of CNN based super-resolution (SR) methods assume that the degradation is known (\eg, bicubic).
Structured Sparsity Learning for Efficient Video Super-Resolution
In SSL, we design pruning schemes for several key components in VSR models, including residual blocks, recurrent networks, and upsampling networks.
Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution
However, existing methods still have two shortcomings: (1) they neglect that the multi-contrast features at different scales contain different anatomical details and hence lack effective mechanisms to match and fuse these features for better reconstruction; and (2) they are still deficient in capturing long-range dependencies, which are essential for the regions with complicated anatomical structures.
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos.
Ranked #5 on
Audio-visual Question Answering
on MUSIC-AVQA
Learning Spatio-Temporal Downsampling for Effective Video Upscaling
Consequently, the inverse task of upscaling a low-resolution, low frame-rate video in space and time becomes a challenging ill-posed problem due to information loss and aliasing artifacts.
STDAN: Deformable Attention Network for Space-Time Video Super-Resolution
Second, we put forward a spatial-temporal deformable feature aggregation (STDFA) module, in which spatial and temporal contexts in dynamic video frames are adaptively captured and aggregated to enhance SR reconstruction.
Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution
To improve matching efficiency, we design a novel Embedded PatchMacth scheme with random samples propagation, which involves end-to-end training with asymptotic linear computational cost to the input size.
Efficient Non-Local Contrastive Attention for Image Super-Resolution
To demonstrate the effectiveness of ENLCA, we build an architecture called Efficient Non-Local Contrastive Network (ENLCN) by adding a few of our modules in a simple backbone.
Space-Time Memory Network for Sounding Object Localization in Videos
Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects.
Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution
A na\"ive method is to decompose it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR).
Space-time Video Super-resolution
Video Frame Interpolation
+1
Can audio-visual integration strengthen robustness under multimodal attacks?
In this paper, we propose to make a systematic study on machines multisensory perception under attacks.
Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
There are rich synchronized audio and visual events in our daily life.
Video Matting via Consistency-Regularized Graph Neural Networks
In this paper, we propose to enhance the temporal coherence by Consistency-Regularized Graph Neural Networks (CRGNN) with the aid of a synthesized video matting dataset.
Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
In this paper, we introduce a new problem, named audio-visual video parsing, which aims to parse a video into temporal event segments and label them as either audible, visible, or both.
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution
Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames).
Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
Rather than synthesizing missing LR video frames as VFI networks do, we firstly temporally interpolate LR frame features in missing LR video frames capturing local temporal contexts by the proposed feature temporal interpolation network.
Ranked #4 on
Video Frame Interpolation
on Vid4 - 4x upscaling
Space-time Video Super-resolution
Video Frame Interpolation
+1
LCSCNet: Linear Compressing Based Skip-Connecting Network for Image Super-Resolution
In this paper, we develop a concise but efficient network architecture called linear compressing based skip-connecting network (LCSCNet) for image super-resolution.
Ranked #18 on
Image Super-Resolution
on Set14 - 3x upscaling
CFSNet: Toward a Controllable Feature Space for Image Restoration
Deep learning methods have witnessed the great progress in image restoration with specific metrics (e. g., PSNR, SSIM).
Residual Dense Network for Image Restoration
We fully exploit the hierarchical features from all the convolutional layers.
Ranked #1 on
Color Image Denoising
on Kodak24 sigma30
An Attempt towards Interpretable Audio-Visual Video Captioning
To achieve this, we propose a multimodal convolutional neural network-based audio-visual video captioning framework and introduce a modality-aware module for exploring modality selection during sentence generation.
TDAN: Temporally Deformable Alignment Network for Video Super-Resolution
Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames).
Deep Learning for Single Image Super-Resolution: A Brief Review
Single image super-resolution (SISR) is a notoriously challenging ill-posed problem, which aims to obtain a high-resolution (HR) output from one of its low-resolution (LR) versions.
Audio-Visual Event Localization in Unconstrained Videos
In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos.
Residual Dense Network for Image Super-Resolution
In this paper, we propose a novel residual dense network (RDN) to address this problem in image SR. We fully exploit the hierarchical features from all the convolutional layers.
Ranked #5 on
Image Super-Resolution
on IXI
