Search Results for author:
Found 69 papers, 18 papers with code
Grounded Dialogue Generation with Cross-encoding Re-ranker, Grounding Span Prediction, and Passage Dropout
For the response generator, we use grounding span prediction as an auxiliary task to be jointly trained with the main task of response generation.
Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction
Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others.
Naturalistic Language-related Movie-Watching fMRI Task for Detecting Neurocognitive Decline and Disorder
The study demonstrated the potential of the naturalistic language-related fMRI task for early detection of aging-related cognitive decline and NCD.
RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning
Robust evaluation is critical for deploying trustworthy retrieval-augmented generation (RAG) systems.
Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving
These speech LLMs (SLLMs) typically use supervised fine-tuning to align speech with text-based LLMs.
$C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction
Audio-Visual Target Speaker Extraction (AV-TSE) aims to mimic the human ability to enhance auditory perception using visual cues.
Generate, Discriminate, Evolve: Enhancing Context Faithfulness via Fine-Grained Sentence-Level Self-Evolution
Improving context faithfulness in large language models is essential for developing trustworthy retrieval augmented generation systems and mitigating hallucinations, especially in long-form question answering (LFQA) tasks or scenarios involving knowledge conflicts.
Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
Empathetic dialogue is crucial for natural human-computer interaction, allowing the dialogue system to respond in a more personalized and emotionally aware manner, improving user satisfaction and engagement.
Detecting Neurocognitive Disorders through Analyses of Topic Evolution and Cross-modal Consistency in Visual-Stimulated Narratives
Early detection of neurocognitive disorders (NCDs) is crucial for timely intervention and disease management.
Disambiguation of Chinese Polyphones in an End-to-End Framework with Semantic Features Extracted by Pre-trained BERT
The pre-trained BERT model extracts semantic features from a raw Chinese character sequence and the NN based classifier predicts the polyphonic character's pronunciation according to BERT output.
learning discriminative features from spectrograms using center loss for speech emotion recognition
Identifying the emotional state from speech is essential for the natural interaction of the machine with the speaker.
Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema
We propose an ontology-grounded approach to Knowledge Graph (KG) construction using Large Language Models (LLMs) on a knowledge base.
Not All Errors Are Equal: Investigation of Speech Recognition Errors in Alzheimer's Disease Detection
Automatic Speech Recognition (ASR) plays an important role in speech-based automatic detection of Alzheimer's disease (AD).
Devising a Set of Compact and Explainable Spoken Language Feature for Screening Alzheimer's Disease
Alzheimer's disease (AD) has become one of the most significant health challenges in an aging society.
Improving Grapheme-to-Phoneme Conversion through In-Context Knowledge Retrieval with Large Language Models
Using GPT-4 in the ICKR system can increase of 3. 5% absolute (3. 8% relative) on the Librig2p dataset.
Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains
We argue that this concept can serve as a principle for making faithful and sound reasoning for KGQA.
A Survey on the Honesty of Large Language Models
Honesty is a fundamental principle for aligning large language models (LLMs) with human values, requiring these models to recognize what they know and don't know and be able to faithfully express their knowledge.
On the Within-class Variation Issue in Alzheimer's Disease Detection
Alzheimer's Disease (AD) detection employs machine learning classification models to distinguish between individuals with AD and those without.
Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTC
SACTC is a tailored CTC variant for multi-talker scenarios, it explicitly models speaker disentanglement by constraining the encoder to represent different speakers' tokens at specific time frames.
AudioComposer: Towards Fine-grained Audio Generation with Natural Language Descriptions
Current Text-to-audio (TTA) models mainly use coarse text descriptions as inputs to generate audio, which hinders models from generating audio with fine-grained control of content and style.
Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models
With these improvement, we show a significant improvement in generation performance and generation speed compared to our previous work and other state-of-the-art (SOTA) large-scale TTS models.
Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models
Moreover, fine-grained prosody modeling is introduced to enhance the model's ability to capture subtle prosody variations in spontaneous speech. Experimental results show that our proposed method significantly outperforms the baseline methods in terms of prosody naturalness and spontaneous behavior naturalness.
Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder
This study highlights the potential of fMRI encoding models and brain scores for detecting early functional changes in NCD patients.
Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System
In this study, we propose a pioneering approach to empower Whisper, which is a speech foundation model, to tackle joint multi-talker and target-talker speech recognition tasks.
Autoregressive Speech Synthesis without Vector Quantization
We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS).
Purple-teaming LLMs with Adversarial Defender Training
In PAD, we automatically collect conversational data that cover the vulnerabilities of an LLM around specific safety risks in a self-play manner, where the attacker aims to elicit unsafe responses and the defender generates safe responses to these attacks.
Seamless Language Expansion: Enhancing Multilingual Mastery in Self-Supervised Models
Experiments show that our adaptation methods enable mHuBERT to be applied to a new language (Mandarin) with MOS value increased about 1. 6 and the relative value of WER reduced up to 61. 72%.
Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answers
A novel approach is then proposed to assess retriever's preference for these candidates by the probability of answers conditioned on the conversational query by marginalizing the Top-$K$ passages.
CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction
Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech.
CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction
Universal sound separation (USS) aims to extract arbitrary types of sounds from real-world recordings.
Ranked #1 on
Target Sound Extraction
on AudioSet
UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization
Dysarthric speech reconstruction (DSR) systems aim to automatically convert dysarthric speech into normal-sounding speech.
Cross-Speaker Encoding Network for Multi-Talker Speech Recognition
To the best of our knowledge, this work represents an early effort to integrate SIMO and SISO for multi-talker speech recognition.
StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis
Both objective and subjective evaluations demonstrate that our proposed method can effectively improve the naturalness and expressiveness of the synthesized speech in audiobook synthesis especially for the role and out-of-domain scenarios.
SimCalib: Graph Neural Network Calibration based on Similarity between Nodes
A novel calibration framework, named SimCalib, is accordingly proposed to consider similarity between nodes at global and local levels.
Injecting linguistic knowledge into BERT for Dialogue State Tracking
This correlation facilitates a comprehensive understanding of the linguistic features influencing the DST model's decision-making process.
Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning
How can we perform computations over natural language representations to solve tasks that require symbolic and numeric reasoning?
Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information
For text-to-speech (TTS) synthesis, prosodic structure prediction (PSP) plays an important role in producing natural and intelligible speech.
QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learning
This paper proposes a novel semi-supervised TTS framework, QS-TTS, to improve TTS quality with lower supervised data requirements via Vector-Quantized Self-Supervised Speech Representation Learning (VQ-S3RL) utilizing more unlabeled speech audio.
Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?
Making moral judgments is an essential step toward developing ethical AI systems.
Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator
Extending on this, we incorporate a diarization branch into the Sidecar, allowing for unified modeling of both ASR and diarization with a negligible overhead of only 768 parameters.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SAIL: Search-Augmented Instruction Learning
Large language models (LLMs) have been significantly improved by instruction fine-tuning, but still lack transparency and the ability to utilize up-to-date knowledge and information.
Interpretable Unified Language Checking
Despite recent concerns about undesirable behaviors generated by large language models (LLMs), including non-factual, biased, and hateful language, we find LLMs are inherent multi-task language checkers based on their latent representations of natural and social knowledge.
A Hierarchical Regression Chain Framework for Affective Vocal Burst Recognition
Experimental results based on the ACII Challenge 2022 dataset demonstrate the superior performance of the proposed system and the effectiveness of considering multiple relationships using hierarchical regression chain models.
Ranked #1 on
Vocal Bursts Intensity Prediction
on HUME-VB
Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detection
This paper presents several efficient methods to extract better AD-related cues from high-level acoustic and linguistic features.
A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One
Although automatic speech recognition (ASR) can perform well in common non-overlapping environments, sustaining performance in multi-talker overlapping speech recognition remains challenging.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Improving Rare Words Recognition through Homophone Extension and Unified Writing for Low-resource Cantonese Speech Recognition
Homophone characters are common in tonal syllable-based languages, such as Mandarin and Cantonese.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representations
Moreover, we optimize the training strategy by leveraging more audio to learn MSMCRs better for low-resource languages.
Disentangled Speech Representation Learning for One-Shot Cross-lingual Voice Conversion Using $β$-VAE
We propose an unsupervised learning method to disentangle speech into content representation and speaker identity representation.
A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS
A vector-quantized, variational autoencoder (VQ-VAE) based feature analyzer is used to encode Mel spectrograms of speech training data by down-sampling progressively in multiple stages into MSMC Representations (MSMCRs) with different time resolutions, and quantizing them with multiple VQ codebooks, respectively.
Exploring linguistic feature and model combination for speech recognition based automatic AD detection
Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care and delay progression.
Tackling Spoofing-Aware Speaker Verification with Multi-Model Fusion
However, previous works show that state-of-the-art ASV models are seriously vulnerable to voice spoofing attacks, and the recently proposed high-performance spoofing countermeasure (CM) models only focus solely on the standalone anti-spoofing tasks, and ignore the subsequent speaker verification process.
Neural Architecture Search for Speech Emotion Recognition
Deep neural networks have brought significant advancements to speech emotion recognition (SER).
Spoofing-Aware Speaker Verification by Multi-Level Fusion
In the second-level fusion, the CM score and ASV scores directly from ASV systems will be concatenated into a prediction block for the final decision.
Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors
In this paper, a novel Bayesian training loss based on per-utterance Dirichlet prior distributions is proposed for verbal emotion recognition, which models the uncertainty in one-hot labels created when human annotators assign the same utterance to different emotion classes.
Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation
The primary task of ASA fine-tunes the SE with the speech of the target dysarthric speaker to effectively capture identity-related information, and the secondary task applies adversarial training to avoid the incorporation of abnormal speaking patterns into the reconstructed speech, by regularizing the distribution of reconstructed speech to be close to that of reference speech with high quality.
The CUHK-TENCENT speaker diarization system for the ICASSP 2022 multi-channel multi-party meeting transcription challenge
This paper describes our speaker diarization system submitted to the Multi-channel Multi-party Meeting Transcription (M2MeT) challenge, where Mandarin meeting data were recorded in multi-channel format for diarization and automatic speech recognition (ASR) tasks.
Characterizing the adversarial vulnerability of speech self-supervised learning
As the paradigm of the self-supervised learning upstream model followed by downstream tasks arouses more attention in the speech community, characterizing the adversarial robustness of such paradigm is of high priority.
Channel-wise Gated Res2Net: Towards Robust Detection of Synthetic Speech Attacks
This argument motivates the current work that presents a novel, channel-wise gated Res2Net (CG-Res2Net), which modifies Res2Net to enable a channel-wise gating mechanism in the connection between feature groups.
Attention Forcing for Machine Translation
This approach guides the model with the generated output history and reference attention, and can reduce the training-inference mismatch without a schedule or a classifier.
Should Ensemble Members Be Calibrated?
It is shown that well calibrated ensemble members will not necessarily yield a well calibrated ensemble prediction, and if the ensemble prediction is well calibrated its performance cannot exceed that of the average performance of the calibrated ensemble members.
Learning Explicit Prosody Models and Deep Speaker Embeddings for Atypical Voice Conversion
Third, a conversion model takes phoneme embeddings and typical prosody features as inputs to generate the converted speech, conditioned on the target DSE that is learned via speaker encoder or speaker adaptation.
Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling
During the training stage, an encoder-decoder-based hybrid connectionist-temporal-classification-attention (CTC-attention) phoneme recognizer is trained, whose encoder has a bottle-neck layer.
Investigating Robustness of Adversarial Samples Detection for Automatic Speaker Verification
Orthogonal to prior approaches, this work proposes to defend ASV systems against adversarial attacks with a separate detection network, rather than augmenting adversarial data into ASV training.
Bayesian x-vector: Bayesian Neural Network based x-vector System for Speaker Verification
Our experiment results indicate that the DNN x-vector system could benefit from BNNs especially when the mismatch problem is severe for evaluations using out-of-domain data.
Deep segmental phonetic posterior-grams based discovery of non-categories in L2 English speech
And then we explore the non-categories by looking for the SPPGs with more than one peak.
Adversarial Attacks on GMM i-vector based Speaker Verification Systems
Experiment results show that GMM i-vector systems are seriously vulnerable to adversarial attacks, and the crafted adversarial samples prove to be transferable and pose threats to neuralnetwork speaker embedding based systems (e. g. x-vector systems).
Coupling Global and Local Context for Unsupervised Aspect Extraction
Aspect words, indicating opinion targets, are essential in expressing and understanding human opinions.
Maximizing Mutual Information for Tacotron
End-to-end speech synthesis methods already achieve close-to-human quality performance.
