Search Results for author:
Found 76 papers, 30 papers with code
Tacotron: Towards End-to-End Speech Synthesis
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.
Ranked #5 on
Speech Synthesis
on North American English
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text.
Ranked #2 on
Speech Synthesis
on North American English
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
In this work, we propose "global style tokens" (GSTs), a bank of embeddings that are jointly trained within Tacotron, a state-of-the-art end-to-end speech synthesis system.
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model.
Ranked #3 on
Audio Generation
on AudioCaps
GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music
In this article, we create a GiantMIDI-Piano (GP) dataset containing 38, 700, 838 transcribed notes and 10, 855 unique solo piano works composed by 2, 786 composers.
High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times
In addition, previous AMT systems are sensitive to the misaligned onset and offset labels of audio recordings.
Sound Audio and Speech Processing
Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation
This paper describes our system (HIT-SCIR) submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text to Universal Dependencies.
Ranked #3 on
Dependency Parsing
on Universal Dependencies
Separate Anything You Describe
In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries.
VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration
Speech restoration aims to remove distortions in speech signals.
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody.
Convolutional Embedding for Edit Distance
Edit-distance-based string similarity search has many applications such as spell correction, data de-duplication, and sequence alignment.
SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation
Adapting to a continuously evolving environment is a safety-critical challenge inevitably faced by all autonomous driving systems.
Trainable Frontend For Robust and Far-Field Keyword Spotting
Robust and far-field speech recognition is critical to enable true hands-free communication.
Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages.
Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task
However, CL on VQA involves not only the expansion of label sets (new Answer sets).
AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant
In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR).
HawkEye: Training Video-Text LLMs for Grounding Text in Videos
Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos.
Ranked #3 on
Video Question Answering
on MVBench
GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval
In this paper, we introduce a new dataset called Kinetic-GEB+.
Ranked #1 on
Boundary Captioning
on Kinetics-GEB+
LSTP: Language-guided Spatial-Temporal Prompt Learning for Long-form Video-Text Understanding
Despite progress in video-language modeling, the computational challenge of interpreting long-form videos in response to task-specific linguistic queries persists, largely due to the complexity of high-dimensional video data and the misalignment between language and visual cues over space and time.
Hierarchical Generative Modeling for Controllable Speech Synthesis
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions.
Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models
Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP).
VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions
Video-grounded dialogue understanding is a challenging problem that requires machine to perceive, parse and reason over situated semantics extracted from weakly aligned video and dialogues.
Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency
The information retrieval community has made significant progress in improving the efficiency of Dual Encoder (DE) dense passage retrieval systems, making them suitable for latency-sensitive settings.
Deep Superpixel-based Network for Blind Image Quality Assessment
In order to fill this gap, we propose a deep adaptive superpixel-based network, namely DSN-IQA, to assess the quality of image based on multi-scale and superpixel segmentation.
Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training
We introduce CDBERT, a new learning paradigm that enhances the semantics understanding ability of the Chinese PLMs with dictionary knowledge and structure of Chinese characters.
STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering
However, most models can only handle simple videos in terms of temporal reasoning, and their performance tends to drop when answering temporal-reasoning questions on long and informative videos.
Network-Level Adversaries in Federated Learning
Federated learning is a popular strategy for training models on distributed, sensitive data, while preserving data privacy.
Simple and Effective Graph-to-Graph Annotation Conversion
Annotation conversion is an effective way to construct datasets under new annotation guidelines based on existing datasets with little human labour.
Uncovering Latent Style Factors for Expressive Speech Synthesis
Prosodic modeling is a core problem in speech synthesis.
Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis
GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style.
Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis
We demonstrate that the proposed framework enables Tacotron to generate intelligible speech using less than half an hour of paired training data.
The HIT-SCIR System for End-to-End Parsing of Universal Dependencies
Our system includes three pipelined components: \textit{tokenization}, \textit{Part-of-Speech} (POS) \textit{tagging} and \textit{dependency parsing}.
Cocktail Party Processing via Structured Prediction
While human listeners excel at selectively attending to a conversation in a cocktail party, machine performance is still far inferior by comparison.
HIT-SCIR at MRP 2019: A Unified Pipeline for Meaning Representation Parsing via Efficient Training and Effective Encoding
This paper describes our system (HIT-SCIR) for CoNLL 2019 shared task: Cross-Framework Meaning Representation Parsing.
Ranked #1 on
UCCA Parsing
on CoNLL 2019
A hybrid text normalization system using multi-head self-attention for mandarin
In this paper, we propose a hybrid text normalization system using multi-head self-attention.
A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis
In Mandarin text-to-speech (TTS) system, the front-end text processing module significantly influences the intelligibility and naturalness of synthesized speech.
ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders
This paper presents ByteSing, a Chinese singing voice synthesis (SVS) system based on duration allocated Tacotron-like acoustic models and WaveRNN neural vocoders.
Adversarial Feature Learning and Unsupervised Clustering based Speech Synthesis for Found Data with Acoustic and Textual Noise
As for the speech-side noise, we propose to learn a noise-independent feature in the auto-regressive decoder through adversarial training and data augmentation, which does not need an extra speech enhancement model.
Review of Text Style Transfer Based on Deep Learning
In the traditional text style transfer model, the text style is generally relied on by experts knowledge and hand-designed rules, but with the application of deep learning in the field of natural language processing, the text style transfer method based on deep learning Started to be heavily researched.
Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech
Accent conversion (AC) transforms a non-native speaker's accent into a native accent while maintaining the speaker's voice timbre.
Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement
Firstly, the speech synthesis model is pre-trained with both multi-speaker clean data and noisy augmented data; then the pre-trained model is adapted on noisy low-resource new speaker data; finally, by setting the clean speech condition, the model can synthesize the new speaker's clean voice.
Xiaomingbot: A Multilingual Robot News Reporter
This paper proposes the building of Xiaomingbot, an intelligent, multilingual and multimodal software robot equipped with four integral capabilities: news generation, news translation, news reading and avatar animation.
Large-Scale MIDI-based Composer Classification
Music classification is a task to classify a music piece into labels such as genres or composers.
Listen, Read, and Identify: Multimodal Singing Language Identification of Music
Optionally, LRID-Net is facilitated with modality dropouts to handle a missing modality.
USTC-NELSLIP System Description for DIHARD-III Challenge
This system description describes our submission system to the Third DIHARD Speech Diarization Challenge.
Modeling the Compatibility of Stem Tracks to Generate Music Mashups
A music mashup combines audio elements from two or more songs to create a new work.
Supervised Chorus Detection for Popular Music Using Convolutional Neural Network and Multi-task Learning
This paper presents a novel supervised approach to detecting the chorus segments in popular music.
VeniBot: Towards Autonomous Venipuncture with Automatic Puncture Area and Angle Regression from NIR Images
For the automation, we focus on the positioning part and propose a Dual-In-Dual-Out network based on two-step learning and two-task learning, which can achieve fully automatic regression of the suitable puncture area and angle from near-infrared(NIR) images.
VeniBot: Towards Autonomous Venipuncture with Semi-supervised Vein Segmentation from Ultrasound Images
In the modern medical care, venipuncture is an indispensable procedure for both diagnosis and treatment.
Audiovisual Singing Voice Separation
Separating a song into vocal and accompaniment components is an active research topic, and recent years witnessed an increased performance from supervised training using deep learning techniques.
Cloning one's voice using very limited data in the wild
(2) How to clone a person's voice while controlling the style and prosody.
Neural Dubber: Dubbing for Videos According to Scripts
Neural Dubber is a multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the prosody of the generated speech.
The USTC-Ximalaya system for the ICASSP 2022 multi-channel multi-party meeting transcription (M2MeT) challenge
We propose two improvements to target-speaker voice activity detection (TS-VAD), the core component in our proposed speaker diarization system that was submitted to the 2022 Multi-Channel Multi-Party Meeting Transcription (M2MeT) challenge.
How Many Grid-Forming Converters do We Need? A Perspective From Small Signal Stability and Power Grid Strength
Based on our analysis, we further study the problem of how to configure GFM converters in the grid and how many GFM converters we will need.
Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation
We study video-grounded dialogue generation, where a response is generated based on the dialogue context and the associated video.
Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance
In this paper, we propose to use intermediate bottleneck features (IBFs) to replace PPGs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Interactive Context-Aware Network for RGB-T Salient Object Detection
However, most related works are based on RGB images, which lose massive useful information.
Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network
The goal of accent conversion (AC) is to convert the accent of speech into the target accent while preserving the content and speaker identity.
Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition
Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A unified front-end framework for English text-to-speech synthesis
The front-end is a critical component of English text-to-speech (TTS) systems, responsible for extracting linguistic features that are essential for a text-to-speech model to synthesize speech, such as prosodies and phonemes.
Language-universal phonetic encoder for low-resource speech recognition
Our main approach and adaptation are effective on extremely low-resource languages, even within domain- and language-mismatched scenarios.
Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition
Moreover, on 3 of the 4 languages, comparing to the standard HuBERT, the approach performs better, meanwhile is able to save supervised training data by 1. 5k hours (75%) at most.
MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning
We introduce MoviePuzzle, a novel challenge that targets visual narrative reasoning and holistic movie understanding.
PolyVoice: Language Models for Speech to Speech Translation
For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model.
InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models
We tested InstructME in instrument-editing, remixing, and multi-round editing.
Teaching Text-to-Image Models to Communicate in Dialog
In this paper, we focus on the innovative dialog-to-image generation task, where the model synthesizes a high-resolution image aligned with the given dialog context as a response.
LLaMA Rider: Spurring Large Language Models to Explore the Open World
Recently, various studies have leveraged Large Language Models (LLMs) to help decision-making and planning in environments, and try to align the LLMs' knowledge with the world conditions.
Audio Prompt Tuning for Universal Sound Separation
Specifically, APT improves the separation performance of specific sources through training a small number of prompt parameters with limited audio samples, while maintaining the generalization of the USS model by keeping its parameters frozen.
A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization
The Stable Diffusion Model (SDM) is a popular and efficient text-to-image (t2i) generation and image-to-image (i2i) generation model.
StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion
Specifically, to enable streaming capability, StreamVoice employs a fully causal context-aware LM with a temporal-independent acoustic predictor, while alternately processing semantic and acoustic features at each time step of autoregression which eliminates the dependence on complete source speech.
TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling
To emphasize temporal correlation modeling, this paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks.
TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables
Experimentally, TimeXer significantly improves time series forecasting with exogenous variables and achieves consistent state-of-the-art performance in twelve real-world forecasting benchmarks.
View-Consistent 3D Editing with Gaussian Splatting
The advent of 3D Gaussian Splatting (3DGS) has revolutionized 3D editing, offering efficient, high-fidelity rendering and enabling precise local manipulations.
Improving Scene Graph Generation with Relation Words' Debiasing in Vision-Language Models
After that, we ensemble VLMs with SGG models to enhance representation.
VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing
We present VoiceShop, a novel speech-to-speech framework that can modify multiple attributes of speech, such as age, gender, accent, and speech style, in a single forward pass while preserving the input speaker's timbre.
