Search Results for author:
Found 30 papers, 19 papers with code
Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent
Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners.
Learning Retrieval Augmentation for Personalized Dialogue Generation
Specifically, the proposed LAPDOG model consists of a story retriever and a dialogue generator.
Selective Prompting Tuning for Personalized Conversations with LLMs
To alleviate those issues, we propose \textbf{S}elective \textbf{P}rompt \textbf{T}uning (SPT), which softly prompts LLMs for personalized conversations in a selective way.
Speech Translation with Large Language Models: An Industrial Practice
Given the great success of large language models (LLMs) across various tasks, in this paper, we introduce LLM-ST, a novel and effective speech translation model constructed upon a pre-trained LLM.
RepCodec: A Speech Representation Codec for Speech Tokenization
To improve the performance of these discrete speech tokens, we present RepCodec, a novel speech representation codec for semantic speech tokenization.
Recent Advances in Direct Speech-to-text Translation
Recently, speech-to-text translation has attracted more and more attention and many studies have emerged rapidly.
MOSPC: MOS Prediction Based on Pairwise Comparison
And our framework also surpasses the strong baseline in ranking accuracy on each fine-grained segment.
PolyVoice: Language Models for Speech to Speech Translation
For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model.
CTC-based Non-autoregressive Speech Translation
Combining end-to-end speech translation (ST) and non-autoregressive (NAR) generation is promising in language and speech processing for their advantages of less error propagation and low latency.
DUB: Discrete Unit Back-translation for Speech Translation
The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST.
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions.
Ranked #1 on
Zero-Shot Environment Sound Classification
on ESC-50
(using extra training data)
M3ST: Mix at Three Levels for Speech Translation
How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)?
Leveraging per Image-Token Consistency for Vision-Language Pre-training
(2) Under-utilization of the unmasked tokens: CMLM primarily focuses on the masked token but it cannot simultaneously leverage other tokens to learn vision-language associations.
Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
Audio captioning aims to generate text descriptions of audio clips.
Personalized Dialogue Generation with Persona-Adaptive Attention
Persona-based dialogue systems aim to generate consistent responses based on historical context and predefined persona.
A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis
In human speech, the attitude of a speaker cannot be fully expressed only by the textual content.
Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation
Direct Speech-to-speech translation (S2ST) has drawn more and more attention recently.
GigaST: A 10,000-hour Pseudo Speech Translation Corpus
The training set is translated by a strong machine translation system and the test set is translated by human.
Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT
LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the HuBERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a $3. 5\times$ compression ratio in three SUPERB tasks, e. g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss.
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+8
Multi-View Self-Attention Based Transformer for Speaker Recognition
In this work, we propose a novel multi-view self-attention mechanism and present an empirical study of different Transformer variants with or without the proposed attention mechanism for speaker recognition.
An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning
Automated audio captioning aims to use natural language to describe the content of audio data.
CL4AC: A Contrastive Loss for Audio Captioning
Automated Audio captioning (AAC) is a cross-modal translation task that aims to use natural language to describe the content of an audio clip.
Token-Level Supervised Contrastive Learning for Punctuation Restoration
Punctuation is critical in understanding natural language text.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation
Machine Speech Chain, which integrates both end-to-end (E2E) automatic speech recognition (ASR) and text-to-speech (TTS) into one circle for joint training, has been proven to be effective in data augmentation by leveraging large amounts of unpaired data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Auto-KWS 2021 Challenge: Task, Datasets, and Baselines
Auto-KWS 2021 challenge calls for automated machine learning (AutoML) solutions to automate the process of applying machine learning to a customized keyword spotting task.
AutoSpeech 2020: The Second Automated Machine Learning Challenge for Speech Classification
The AutoSpeech challenge calls for automated machine learning (AutoML) solutions to automate the process of applying machine learning to speech processing tasks.
MetaMix: Improved Meta-Learning with Interpolation-based Consistency Regularization
MetaMix can be integrated with any of the MAML-based algorithms and learn the decision boundaries generalizing better to new tasks.
An Investigation of Few-Shot Learning in Spoken Term Classification
In this paper, we investigate the feasibility of applying few-shot learning algorithms to a speech task.
