Search Results for author:
Found 66 papers, 27 papers with code
Simple and Effective Graph-to-Graph Annotation Conversion
Annotation conversion is an effective way to construct datasets under new annotation guidelines based on existing datasets with little human labour.
Audio Prompt Tuning for Universal Sound Separation
Specifically, APT improves the separation performance of specific sources through training a small number of prompt parameters with limited audio samples, while maintaining the generalization of the USS model by keeping its parameters frozen.
LLaMA Rider: Spurring Large Language Models to Explore the Open World
Recently, various studies have leveraged Large Language Models (LLMs) to help decision-making and planning in environments, and try to align the LLMs' knowledge with the world conditions.
Teaching Text-to-Image Models to Communicate
Various works have been extensively studied in the research of text-to-image generation.
InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models
We tested InstructME in instrument-editing, remixing, and multi-round editing.
Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models
Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP).
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model.
Ranked #2 on
Audio Generation
on AudioCaps
Separate Anything You Describe
In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries.
Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency
The information retrieval community has made significant progress in improving the efficiency of Dual Encoder (DE) dense passage retrieval systems, making them suitable for latency-sensitive settings.
PolyVoice: Language Models for Speech to Speech Translation
For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model.
MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning
We introduce MoviePuzzle, a novel challenge that targets visual narrative reasoning and holistic movie understanding.
VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions
Video-grounded dialogue understanding is a challenging problem that requires machine to perceive, parse and reason over situated semantics extracted from weakly aligned video and dialogues.
Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training
We introduce CDBERT, a new learning paradigm that enhances the semantics understanding ability of the Chinese PLMs with dictionary knowledge and structure of Chinese characters.
Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition
Moreover, on 3 of the 4 languages, comparing to the standard HuBERT, the approach performs better, meanwhile is able to save supervised training data by 1. 5k hours (75%) at most.
Language-universal phonetic encoder for low-resource speech recognition
Our main approach and adaptation are effective on extremely low-resource languages, even within domain- and language-mismatched scenarios.
a unified front-end framework for english text-to-speech synthesis
The front-end is a critical component of English text-to-speech (TTS) systems, responsible for extracting linguistic features that are essential for a text-to-speech model to synthesize speech, such as prosodies and phonemes.
Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition
Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network
The goal of accent conversion (AC) is to convert the accent of speech into the target accent while preserving the content and speaker identity.
Interactive Context-Aware Network for RGB-T Salient Object Detection
However, most related works are based on RGB images, which lose massive useful information.
Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance
In this paper, we propose to use intermediate bottleneck features (IBFs) to replace PPGs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation
We study video-grounded dialogue generation, where a response is generated based on the dialogue context and the associated video.
How Many Grid-Forming Converters do We Need? A Perspective From Small Signal Stability and Power Grid Strength
Based on our analysis, we further study the problem of how to configure GFM converters in the grid and how many GFM converters we will need.
Network-Level Adversaries in Federated Learning
Federated learning is a popular strategy for training models on distributed, sensitive data, while preserving data privacy.
Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task
However, CL on VQA involves not only the expansion of label sets (new Answer sets).
SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation
Adapting to a continuously evolving environment is a safety-critical challenge inevitably faced by all autonomous driving systems.
VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration
Speech restoration aims to remove distortions in speech signals.
GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval
In this paper, we introduce a new dataset called Kinetic-GEB+.
Ranked #1 on
Text to Video Retrieval
on Kinetics-GEB+
(text-to-video R@1 metric)
The USTC-Ximalaya system for the ICASSP 2022 multi-channel multi-party meeting transcription (M2MeT) challenge
We propose two improvements to target-speaker voice activity detection (TS-VAD), the core component in our proposed speaker diarization system that was submitted to the 2022 Multi-Channel Multi-Party Meeting Transcription (M2MeT) challenge.
AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant
In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR).
Neural Dubber: Dubbing for Videos According to Scripts
Neural Dubber is a multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the prosody of the generated speech.
Deep Superpixel-based Network for Blind Image Quality Assessment
In order to fill this gap, we propose a deep adaptive superpixel-based network, namely DSN-IQA, to assess the quality of image based on multi-scale and superpixel segmentation.
Cloning one's voice using very limited data in the wild
(2) How to clone a person's voice while controlling the style and prosody.
Audiovisual Singing Voice Separation
Separating a song into vocal and accompaniment components is an active research topic, and recent years witnessed an increased performance from supervised training using deep learning techniques.
VeniBot: Towards Autonomous Venipuncture with Semi-supervised Vein Segmentation from Ultrasound Images
In the modern medical care, venipuncture is an indispensable procedure for both diagnosis and treatment.
VeniBot: Towards Autonomous Venipuncture with Automatic Puncture Area and Angle Regression from NIR Images
For the automation, we focus on the positioning part and propose a Dual-In-Dual-Out network based on two-step learning and two-task learning, which can achieve fully automatic regression of the suitable puncture area and angle from near-infrared(NIR) images.
Supervised Chorus Detection for Popular Music Using Convolutional Neural Network and Multi-task Learning
This paper presents a novel supervised approach to detecting the chorus segments in popular music.
Modeling the Compatibility of Stem Tracks to Generate Music Mashups
A music mashup combines audio elements from two or more songs to create a new work.
USTC-NELSLIP System Description for DIHARD-III Challenge
This system description describes our submission system to the Third DIHARD Speech Diarization Challenge.
Listen, Read, and Identify: Multimodal Singing Language Identification of Music
Optionally, LRID-Net is facilitated with modality dropouts to handle a missing modality.
Large-Scale MIDI-based Composer Classification
Music classification is a task to classify a music piece into labels such as genres or composers.
GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music
In this article, we create a GiantMIDI-Piano (GP) dataset containing 38, 700, 838 transcribed notes and 10, 855 unique solo piano works composed by 2, 786 composers.
High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times
In addition, previous AMT systems are sensitive to the misaligned onset and offset labels of audio recordings.
Sound Audio and Speech Processing
Xiaomingbot: A Multilingual Robot News Reporter
This paper proposes the building of Xiaomingbot, an intelligent, multilingual and multimodal software robot equipped with four integral capabilities: news generation, news translation, news reading and avatar animation.
Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement
Firstly, the speech synthesis model is pre-trained with both multi-speaker clean data and noisy augmented data; then the pre-trained model is adapted on noisy low-resource new speaker data; finally, by setting the clean speech condition, the model can synthesize the new speaker's clean voice.
Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech
Accent conversion (AC) transforms a non-native speaker's accent into a native accent while maintaining the speaker's voice timbre.
Review of Text Style Transfer Based on Deep Learning
In the traditional text style transfer model, the text style is generally relied on by experts knowledge and hand-designed rules, but with the application of deep learning in the field of natural language processing, the text style transfer method based on deep learning Started to be heavily researched.
Adversarial Feature Learning and Unsupervised Clustering based Speech Synthesis for Found Data with Acoustic and Textual Noise
As for the speech-side noise, we propose to learn a noise-independent feature in the auto-regressive decoder through adversarial training and data augmentation, which does not need an extra speech enhancement model.
ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders
This paper presents ByteSing, a Chinese singing voice synthesis (SVS) system based on duration allocated Tacotron-like acoustic models and WaveRNN neural vocoders.
Convolutional Embedding for Edit Distance
Edit-distance-based string similarity search has many applications such as spell correction, data de-duplication, and sequence alignment.
A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis
In Mandarin text-to-speech (TTS) system, the front-end text processing module significantly influences the intelligibility and naturalness of synthesized speech.
A hybrid text normalization system using multi-head self-attention for mandarin
In this paper, we propose a hybrid text normalization system using multi-head self-attention.
HIT-SCIR at MRP 2019: A Unified Pipeline for Meaning Representation Parsing via Efficient Training and Effective Encoding
This paper describes our system (HIT-SCIR) for CoNLL 2019 shared task: Cross-Framework Meaning Representation Parsing.
Ranked #1 on
UCCA Parsing
on CoNLL 2019
Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages.
Hierarchical Generative Modeling for Controllable Speech Synthesis
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions.
Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis
We demonstrate that the proposed framework enables Tacotron to generate intelligible speech using less than half an hour of paired training data.
Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis
GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style.
Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation
This paper describes our system (HIT-SCIR) submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text to Universal Dependencies.
Ranked #3 on
Dependency Parsing
on Universal Dependencies
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody.
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
In this work, we propose "global style tokens" (GSTs), a bank of embeddings that are jointly trained within Tacotron, a state-of-the-art end-to-end speech synthesis system.
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text.
Ranked #2 on
Speech Synthesis
on North American English
Uncovering Latent Style Factors for Expressive Speech Synthesis
Prosodic modeling is a core problem in speech synthesis.
The HIT-SCIR System for End-to-End Parsing of Universal Dependencies
Our system includes three pipelined components: \textit{tokenization}, \textit{Part-of-Speech} (POS) \textit{tagging} and \textit{dependency parsing}.
Tacotron: Towards End-to-End Speech Synthesis
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.
Ranked #5 on
Speech Synthesis
on North American English
Trainable Frontend For Robust and Far-Field Keyword Spotting
Robust and far-field speech recognition is critical to enable true hands-free communication.
Cocktail Party Processing via Structured Prediction
While human listeners excel at selectively attending to a conversation in a cocktail party, machine performance is still far inferior by comparison.
