Search Results for author:
Found 76 papers, 23 papers with code
ImViD: Immersive Volumetric Videos for Enhanced VR Engagement
To stimulate the reconstruction of immersive volumetric videos, we introduce ImViD, a multi-view, multi-modal dataset featuring complete space-oriented data capture and various indoor/outdoor scenarios.
Boosting Multimodal Reasoning with MCTS-Automated Structured Thinking
Multimodal large language models (MLLMs) exhibit impressive capabilities but still face challenges in complex visual reasoning.
DReSS: Data-driven Regularized Structured Streamlining for Large Language Models
Large language models (LLMs) have achieved significant progress across various domains, but their increasing scale results in high computational and memory costs.
MTPareto: A MultiModal Targeted Pareto Framework for Fake News Detection
Multimodal fake news detection is essential for maintaining the authenticity of Internet multimedia information.
Region-Based Optimization in Continual Learning for Audio Deepfake Detection
To address this issue, we propose a continual learning method named Region-Based Optimization (RegO) for audio deepfake detection.
Reject Threshold Adaptation for Open-Set Model Attribution of Deepfake Audio
To address the issues, we propose a novel framework for open set model attribution of deepfake audio with rejection threshold adaptation (ReTA).
Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS
In-context Learning (ICL) enables large language models (LLMs) to tackle downstream tasks through sophisticated prompting and high-quality demonstrations.
LetsTalk: Latent Diffusion Transformer for Talking Video Synthesis
Then we propose a suitable solution according to the modality differences of image, audio, and video generation.
DARNet: Dual Attention Refinement Network with Spatiotemporal Construction for Auditory Attention Detection
To address these issues, this paper proposes a dual attention refinement network with spatiotemporal construction for AAD, named DARNet, which consists of the spatiotemporal construction module, dual attention refinement module, and feature fusion \& classifier module.
DPI-TTS: Directional Patch Interaction for Fast-Converging and Style Temporal Modeling in Text-to-Speech
In recent years, speech diffusion models have advanced rapidly.
Text Prompt is Not Enough: Sound Event Enhanced Prompt Adapter for Target Style Audio Generation
Additionally, the Sound Event Reference Style Transfer Dataset (SERST) is introduced for the proposed target style audio generation task, enabling dual-prompt audio generation using both text and audio references.
Pandora's Box or Aladdin's Lamp: A Comprehensive Analysis Revealing the Role of RAG Noise in Large Language Models
Retrieval-Augmented Generation (RAG) has emerged as a crucial method for addressing hallucinations in large language models (LLMs).
Does Current Deepfake Audio Detection Model Effectively Detect ALM-based Deepfake Audio?
Currently, Audio Language Models (ALMs) are rapidly advancing due to the developments in large language models and audio neural codecs.
VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing
For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired, emphasizing the semantic content of the text modality while de-emphasizing the paralinguistic information of the speech modality.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
ADD 2023: Towards Audio Deepfake Detection and Analysis in the Wild
The growing prominence of the field of audio deepfake detection is driven by its wide range of applications, notably in protecting the public from potential fraud and other malicious activities, prompting the need for greater attention and research in this area.
MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics
We believe that MDPE will become a valuable resource for promoting research in the field of affective computing.
An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio
However, they overlook the characteristics in target domain that are absent in source domain.
ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation
The results indicate that the ASRRL method significantly outperforms traditional fine-tuning approaches, achieving higher speaker similarity and better overall speech quality with limited reference speeches.
Fake News Detection and Manipulation Reasoning via Large Vision-Language Models
To support the research, we introduce a benchmark for fake news detection and manipulation reasoning, referred to as Human-centric and Fact-related Fake News (HFFN).
MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation
Despite advancements in AIGC technologies for text and image generation, the foley audio dubbing remains rudimentary due to difficulties in cross-modal scene matching and content correlation.
Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio
To effectively detect LLM-based deepfake audio, we focus on the core of the generation process, the conversion from neural codec to waveform.
Generalized Source Tracing: Detecting Novel Audio Deepfake Algorithm with Real Emphasis and Fake Dispersion Strategy
For effective OOD detection, we first explore current post-hoc OOD methods and propose NSD, a novel OOD approach in identifying novel deepfake algorithms through the similarity consideration of both feature and logits scores.
Can large language models understand uncommon meanings of common words?
Large language models (LLMs) like ChatGPT have shown significant advancements across diverse natural language understanding (NLU) tasks, including intelligent dialogue and autonomous agents.
The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio
With the proliferation of Audio Language Model (ALM) based deepfake audio, there is an urgent need for generalized detection methods.
MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
However, this process may lead to inaccurate annotations, such as ignoring non-majority or non-candidate labels.
KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering
Large language models (LLMs) suffer from the hallucination problem and face significant challenges when applied to knowledge-intensive tasks.
Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild
Multimodal fusion is a significant method for most multimodal tasks.
Can Deception Detection Go Deeper? Dataset, Evaluation, and Benchmark for Deception Reasoning
Deception detection has attracted increasing attention due to its importance in real-world scenarios.
Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion
This paper proposes a progressive distillation method based on masked generation features for KGC task, aiming to significantly reduce the complexity of pre-trained models.
HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines.
Ranked #7 on
Dynamic Facial Expression Recognition
on MAFW
Contrastive Learning
Dynamic Facial Expression Recognition
+3
SVFAP: Self-supervised Video Facial Affect Perceiver
Video-based facial affect analysis has recently attracted increasing attention owing to its critical role in human-computer interaction.
Ranked #6 on
Dynamic Facial Expression Recognition
on FERV39k
Dynamic Facial Expression Recognition
Emotion Recognition
+2
What to Remember: Self-Adaptive Continual Learning for Audio Deepfake Detection
The rapid evolution of speech synthesis and voice conversion has raised substantial concerns due to the potential misuse of such technology, prompting a pressing need for effective audio deepfake detection mechanisms.
GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition
To bridge this gap, we present the quantitative evaluation results of GPT-4V on 21 benchmark datasets covering 6 tasks: visual sentiment analysis, tweet sentiment analysis, micro-expression recognition, facial emotion recognition, dynamic facial emotion recognition, and multimodal emotion recognition.
DGSD: Dynamical Graph Self-Distillation for EEG-Based Auditory Spatial Attention Detection
Specifically, to effectively represent the non-Euclidean properties of EEG signals, dynamical graph convolutional networks are applied to represent the graph structure of EEG signals, which can also extract crucial features related to auditory spatial attention in EEG signals.
Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detection
The orthogonal weight modification to overcome catastrophic forgetting does not consider the similarity of genuine audio across different datasets.
MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition
Dynamic facial expression recognition (DFER) is essential to the development of intelligent and empathetic machines.
Ranked #4 on
Dynamic Facial Expression Recognition
on FERV39k
Dynamic Facial Expression Recognition
Facial Expression Recognition
Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion
When decreasing the number of sampling steps (i. e., the number of line segments used to fit the path), the ease of fitting straight lines compared to curves allows us to generate higher quality samples from a random noise with fewer iterations.
Low-rank Adaptation Method for Wav2vec2-based Fake Audio Detection
Self-supervised speech models are a rapidly developing research topic in fake audio detection.
Adaptive Fake Audio Detection with Low-Rank Model Squeezing
During the inference stage, these adaptation matrices are combined with the existing model to generate the final prediction output.
M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis
However, it is still a challenge to comprehensively model the conversation, and a majority of conversational TTS systems only focus on extracting global information and omit local prosody features, which contain important fine-grained information like keywords and emphasis.
MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
The first Multimodal Emotion Recognition Challenge (MER 2023) was successfully held at ACM Multimedia.
UnifySpeech: A Unified Framework for Zero-shot Text-to-Speech and Voice Conversion
Text-to-speech (TTS) and voice conversion (VC) are two different tasks both aiming at generating high quality speaking voice according to different input modality.
Emotion Selectable End-to-End Text-based Speech Editing
To achieve this task, we propose Emo-CampNet (emotion CampNet), which can provide the option of emotional attributes for the generated speech in text-based speech editing and has the one-shot ability to edit unseen speakers' speech.
SceneFake: An Initial Dataset and Benchmarks for Scene Fake Audio Detection
Some scene fake audio detection benchmark results on the SceneFake dataset are reported in this paper.
IRNet: Iterative Refinement Network for Noisy Partial Label Learning
In this paper, we relax this assumption and focus on a more general problem, noisy PLL, where the ground-truth label may not exist in the candidate set.
An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
Speech is the fundamental mode of human communication, and its synthesis has long been a core priority in human-computer interaction research.
Audio Deepfake Attribution: An Initial Dataset and Investigation
To address the challenges of attribution of continuously emerging unknown audio generation tools in the real world, we propose the Class-Representation Multi-Center Learning (CRML) method for open-set audio deepfake attribution (OSADA).
Fully Automated End-to-End Fake Audio Detection
The existing fake audio detection systems often rely on expert experience to design the acoustic features or manually design the hyperparameters of the network structure.
An Initial Investigation for Detecting Vocoder Fingerprints of Fake Audio
Many effective attempts have been made for fake audio detection.
Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis
With the proliferation of user-generated online videos, Multimodal Sentiment Analysis (MSA) has attracted increasing attention recently.
Audio Deepfake Detection Based on a Combination of F0 Information and Real Plus Imaginary Spectrogram Features
Meanwhile, to make full use of the phase and full-band information, we also propose to use real and imaginary spectrogram features as complementary input features and model the disjoint subbands separately.
Two-Aspect Information Fusion Model For ABAW4 Multi-task Challenge
In this paper, we propose the solution to the Multi-Task Learning (MTL) Challenge of the 4th Affective Behavior Analysis in-the-wild (ABAW) competition.
Adaptive Pseudo-Siamese Policy Network for Temporal Knowledge Prediction
Specifically, we design the policy network in our model as a pseudo-siamese policy network that consists of two sub-policy networks.
EmotionNAS: Two-stream Neural Architecture Search for Speech Emotion Recognition
Speech emotion recognition (SER) is an important research topic in human-computer interaction.
NeuralDPS: Neural Deterministic Plus Stochastic Model with Multiband Excitation for Noise-Controllable Waveform Generation
We have also verified through experiments that this method can effectively control the noise components in the predicted speech and adjust the SNR of speech.
GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
To this end, we propose a novel framework for incomplete multimodal learning in conversations, called "Graph Complete Network (GCNet)", filling the gap of existing works.
CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing
It can solve unnatural prosody in the edited region and synthesize the speech corresponding to the unseen words in the transcript.
MixKG: Mixing for harder negative samples in knowledge graph
The representations of entities and relations are learned via contrasting the positive and negative triplets.
ADD 2022: the First Audio Deep Synthesis Detection Challenge
Audio deepfake detection is an emerging topic, which was included in the ASVspoof 2021.
Singing-Tacotron: Global duration control attention and dynamic filter for End-to-end singing voice synthesis
Firstly, we propose a global duration control attention mechanism for the SVS model.
Reducing language context confusion for end-to-end code-switching automatic speech recognition
We propose a language-related attention mechanism to reduce multilingual context confusion for the E2E code-switching ASR model based on the Equivalence Constraint (EC) Theory.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Knowledge graph enhanced recommender system
This is attributed to the rich attribute information contained in KG to improve item and user representations as side information.
Multi-Level Graph Contrastive Learning
In this paper, we propose a Multi-Level Graph Contrastive Learning (MLGCL) framework for learning robust representation of graph data by contrasting space views of graphs.
Continual Learning for Fake Audio Detection
However, fine-tuning leads to performance degradation on previous data.
Half-Truth: A Partially Fake Audio Detection Dataset
Therefore, this paper develops such a dataset for half-truth audio detection (HAD).
TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition
To address these two problems, we propose a new model named the two-step non-autoregressive transformer(TSNAT), which improves the performance and accelerating the convergence of the NAR model by learning prior knowledge from a parameters-sharing AR model.
Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
Based on this idea, we propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once).
Tucker decomposition-based Temporal Knowledge Graph Completion
Developing the model for temporal knowledge graphs completion is an increasingly important task.
Deep Time Delay Neural Network for Speech Enhancement with Full Data Learning
This paper proposes a deep time delay neural network (TDNN) for speech enhancement with full data learning.
Self-supervised Graph Representation Learning via Bootstrapping
In addition, we summarize three kinds of augmentation methods for graph-structured data and apply them to the DGB.
Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognition
The joint training framework for speech enhancement and recognition methods have obtained quite good performances for robust end-to-end automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Decoupling Pronunciation and Language for End-to-end Code-switching Automatic Speech Recognition
In this paper, we propose a decoupled transformer model to use monolingual paired data and unpaired text data to alleviate the problem of code-switching data shortage.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition
Emotion recognition remains a complex task due to speaker variations and low-resource training samples.
Ranked #2 on
Multimodal Emotion Recognition
on IEMOCAP-4
(Accuracy metric)
One In A Hundred: Select The Best Predicted Sequence from Numerous Candidates for Streaming Speech Recognition
Inspired by the success of two-pass end-to-end models, we introduce a transformer decoder and the two-stage inference method into the streaming CTC model.
Deep imitator: Handwriting calligraphy imitation via deep attention networks
Calligraphy imitation (CI) from a handful of target handwriting samples is such a challenging task that most of the existing writing style analysis or handwriting generation methods do not exhibit satisfactory performance.
