Search Results for author:
Found 55 papers, 33 papers with code
Enhancing Modality Representation and Alignment for Multimodal Cold-start Active Learning
Additionally, most AL methods seldom address multimodal data, highlighting a research gap in this field.
On-the-fly Modulation for Balanced Multimodal Learning
Then, On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies are proposed to modulate the optimization of each modality, by monitoring the discriminative discrepancy between modalities during training.
KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidance
Within the intervals between semantic key states, optical flow is employed to capture motion key states to understand the mechanisms of "how to do".
Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation
Humans possess a remarkable talent for flexibly alternating to different senses when interacting with the environment.
Boosting Audio Visual Question Answering via Key Semantic-Aware Cues
The Audio Visual Question Answering (AVQA) task aims to answer questions related to various visual objects, sounds, and their interactions in videos.
Audio-visual Question Answering
Audio-Visual Question Answering (AVQA)
+2
Towards Effective and Efficient Continual Pre-training of Large Language Models
To make the CPT approach more traceable, this paper presents a technical report for continually pre-training Llama-3 (8B), which significantly enhances the Chinese language ability and scientific reasoning ability of the backbone model.
Unveiling and Mitigating Bias in Audio Visual Segmentation
For audio priming bias, to enhance audio sensitivity to different intensities and semantics, a perception module specifically for audio perceives the latent semantic information and incorporates information into a limited set of queries, namely active queries.
Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation
Audio-Visual Segmentation (AVS) aims to achieve pixel-level localization of sound sources in videos, while Audio-Visual Semantic Segmentation (AVSS), as an extension of AVS, further pursues semantic understanding of audio-visual scenes.
Can Textual Semantics Mitigate Sounding Object Segmentation Preference?
The Audio-Visual Segmentation (AVS) task aims to segment sounding objects in the visual space using audio cues.
Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
In this work, we introduce a novel task called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment objects within the visual domain based on expressions containing multimodal cues.
Diagnosing and Re-learning for Balanced Multimodal Learning
In this way, the over-emphasizing of scarcely informative modalities is avoided.
YuLan: An Open-source Large Language Model
This paper presents the development of YuLan, a series of open-source LLMs with $12$ billion parameters.
Learning Manipulation by Predicting Interaction
To this end, we propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction (MPI) and enhances the visual representation. Given a pair of keyframes representing the initial and final states, along with language instructions, our algorithm predicts the transition frame and detects the interaction object, respectively.
MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
However, in this paper, we identify the previously ignored gradient conflict between multimodal and unimodal learning objectives, potentially misleading the unimodal encoder optimization.
Multimodal Fusion on Low-quality Data: A Comprehensive Survey
Multimodal fusion focuses on integrating information from multiple modalities with the goal of more accurate prediction, which has achieved remarkable progress in a wide range of scenarios, including autonomous driving and medical diagnosis.
SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model
Controllable spherical panoramic image generation holds substantial applicative potential across a variety of domains. However, it remains a challenging task due to the inherent spherical distortion and geometry characteristics, resulting in low-quality content generation. In this paper, we introduce a novel framework of SphereDiffusion to address these unique challenges, for better generating high-quality and precisely controllable spherical panoramic images. For the spherical distortion characteristic, we embed the semantics of the distorted object with text encoding, then explicitly construct the relationship with text-object correspondence to better use the pre-trained knowledge of the planar images. Meanwhile, we employ a deformable technique to mitigate the semantic deviation in latent space caused by spherical distortion. For the spherical geometry characteristic, in virtue of spherical rotation invariance, we improve the data diversity and optimization objectives in the training process, enabling the model to better learn the spherical geometry characteristic. Furthermore, we enhance the denoising process of the diffusion model, enabling it to effectively use the learned geometric characteristic to ensure the boundary continuity of the generated images. With these specific techniques, experiments on Structured3D dataset show that SphereDiffusion significantly improves the quality of controllable spherical image generation and relatively reduces around 35% FID on average.
Quantifying and Enhancing Multi-modal Robustness with Modality Preference
Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective.
Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs
Generalizable articulated object manipulation is essential for home-assistant robots.
Enhancing multimodal cooperation via sample-level modality valuation
One primary topic of multimodal learning is to jointly incorporate heterogeneous information from different modalities.
Progressive Spatio-temporal Perception for Audio-Visual Question Answering
Such naturally multi-modal videos are composed of rich and complex dynamic audio-visual components, where most of which could be unrelated to the given questions, or even play as interference in answering the content of interest.
Ranked #2 on
Audio-Visual Question Answering (AVQA)
on AVQA
Audio-visual Question Answering
Audio-Visual Question Answering (AVQA)
+2
Supervised Knowledge May Hurt Novel Class Discovery Performance
Next, by using the proposed transfer flow, we conduct various empirical experiments with different levels of semantic similarity, yielding that supervised knowledge may hurt NCD performance.
Multi-Scale Attention for Audio Question Answering
Audio question answering (AQA), acting as a widely used proxy task to explore scene understanding, has got more attention.
Robust Cross-Modal Knowledge Distillation for Unconstrained Videos
However, such semantic consistency from the synchronization is hard to guarantee in unconstrained videos, due to the irrelevant modality noise and differentiated semantic correlation.
Balanced Audiovisual Dataset for Imbalance Analysis
We surprisingly find that: the multimodal models with existing imbalance algorithms consistently perform worse than the unimodal one on specific subsets, in accordance with the modality bias.
Revisiting Pre-training in Audio-Visual Learning
Specifically, we explore the effects of pre-trained models on two audio-visual learning scenarios: cross-modal initialization and multi-modal joint learning.
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Experimental results indicate that the models incorporating large language models (LLM) can generate more diverse responses, while the model utilizing knowledge graphs to introduce external knowledge performs the best overall.
A Closer Look at Novel Class Discovery from the Labeled Set
Novel class discovery (NCD) aims to infer novel categories in an unlabeled dataset leveraging prior knowledge of a labeled set comprising disjoint but related classes.
Learning in Audio-visual Context: A Review, Analysis, and New Perspective
A comprehensive survey that can systematically organize and analyze studies of the audio-visual field is expected.
Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction
The domain discrepancy induces to performance degradation on target testing data for CNN models.
Balanced Multimodal Learning via On-the-fly Gradient Modulation
Multimodal learning helps to comprehensively understand the world, by integrating different senses.
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos.
Ranked #5 on
Audio-visual Question Answering
on MUSIC-AVQA
SeCo: Separating Unknown Musical Visual Sounds with Consistency Guidance
Recent years have witnessed the success of deep learning on the visual sound separation task.
Towards Inadequately Pre-trained Models in Transfer Learning
Based on the contradictory phenomenon between FE and FT that better feature extractor fails to be fine-tuned better accordingly, we conduct comprehensive analyses on features before softmax layer to provide insightful explanations.
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
Specifically, we observe that the previous practice of learning only a single audio representation is insufficient due to the additive nature of audio signals.
Class-aware Sounding Objects Localization via Audiovisual Correspondence
To address this problem, we propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios using only the correspondence between audio and vision.
Self-supervised Audiovisual Representation Learning for Remote Sensing Data
By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pre-training strategies for remote sensing imagery.
Ranked #2 on
Cross-Modal Retrieval
on SoundingEarth
Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge
Additionally, CMD covers two special cases: zero-knowledge and all knowledge, leading to a unified MKD framework.
Unsupervised Multi-Source Domain Adaptation for Person Re-Identification
Although achieving great success, most of them only use limited data from a single-source domain for model pre-training, making the rich labeled data insufficiently exploited.
Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
There are rich synchronized audio and visual events in our daily life.
Model information as an analysis tool in deep learning
We demonstrate using model information as a general analysis tool to gain insight into problems that arise in deep learning.
Temporal Relational Modeling with Self-Supervision for Action Segmentation
The main reason is that large number of nodes (i. e., video frames) makes GCNs hard to capture and model temporal relations in videos.
Ranked #24 on
Action Segmentation
on 50 Salads
Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement
Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples.
Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching
First, we propose to learn robust object representations by aggregating the candidate sound localization results in the single source scenes.
Multiple Sound Sources Localization from Coarse to Fine
How to visually localize multiple sound sources in unconstrained videos is a formidable problem, especially when lack of the pairwise sound-object annotations.
Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition
With the help of this dataset, we evaluate three proposed approaches for transferring the sound event knowledge to the aerial scene recognition task in a multimodal learning framework, and show the benefit of exploiting the audio information for the aerial scene recognition.
Ambient Sound Helps: Audiovisual Crowd Counting in Extreme Conditions
Visual crowd counting has been recently studied as a way to enable people counting in crowd scenes from images.
Curriculum Audiovisual Learning
Associating sound and its producer in complex audiovisual scene is a challenging task, especially when we are lack of annotated training data.
Listen to the Image
different encoding schemes indicate that using machine model to accelerate optimization evaluation and reduce experimental cost is feasible to some extent, which could dramatically promote the upgrading of encoding scheme then help the blind to improve their visual perception ability.
Deep LDA Hashing
The conventional supervised hashing methods based on classification do not entirely meet the requirements of hashing technique, but Linear Discriminant Analysis (LDA) does.
Dense Multimodal Fusion for Hierarchically Joint Representation
Hence, it is highly expected to learn effective joint representation by fusing the features of different modalities.
Deep Multimodal Clustering for Unsupervised Audiovisual Learning
And such integrated multimodal clustering network can be effectively trained with max-margin loss in the end-to-end fashion.
Image2song: Song Retrieval via Bridging Image Content and Lyric Words
Image is usually taken for expressing some kinds of emotions or purposes, such as love, celebrating Christmas.
Deep Binary Reconstruction for Cross-modal Hashing
Based on the analysis, we provide a so-called Deep Binary Reconstruction (DBRC) network that can directly learn the binary hashing codes in an unsupervised fashion.
Temporal Multimodal Learning in Audiovisual Speech Recognition
Recently, audiovisual speech recognition based the MRBM has attracted much attention, and the MRBM shows its effectiveness in learning the joint representation across audiovisual modalities.
