Search Results for author:
Found 79 papers, 30 papers with code
Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?
Recently, self-supervised learning (SSL) has demonstrated strong performance in speaker recognition, even if the pre-training objective is designed for speech recognition.
Ultra Fast Speech Separation Model with Teacher Student Learning
In this paper, an ultra fast speech separation Transformer model is proposed to achieve both better performance and efficiency with teacher student learning (T-S learning).
A collaborative decomposition-based evolutionary algorithm integrating normal and penalty-based boundary intersection for many-objective optimization
On the other hand, the parallel reference lines of the parallel decomposition methods including the normal boundary intersection (NBI) might result in poor diversity because of under-sampling near the boundaries for MaOPs with concave frontiers.
Quantized GAN for Complex Music Generation from Dance Videos
We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates complex musical samples conditioned on dance videos.
Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings
The proposed speaker embedding, named t-vector, is extracted synchronously with the t-SOT ASR model, enabling joint execution of speaker identification (SID) or speaker diarization (SD) with the multi-talker transcription with low latency.
Streaming Multi-Talker ASR with Token-Level Serialized Output Training
This paper proposes a token-level serialized output training (t-SOT), a novel framework for streaming multi-talker automatic speech recognition (ASR).
Multi-query Video Retrieval
In this paper, we focus on the less-studied setting of multi-query video retrieval, where multiple queries are provided to the model for searching over the video archive.
Self-Supervised Learning for speech recognition with Intermediate layer supervision
Recently, pioneer work finds that speech pre-trained models can solve full-stack speech processing tasks, because the model utilizes bottom layers to learn speaker-related information and top layers to encode content-related information.
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks.
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
We integrate the proposed methods into the HuBERT framework.
Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition
In this paper we propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech via contrastive learning.
Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition
ILMA enables a fast text-only adaptation of the E2E model without increasing the run-time computational cost.
Cell2State: Learning Cell State Representations From Barcoded Single-Cell Gene-Expression Transitions
Genetic barcoding coupled with single-cell sequencing technology enables direct measurement of cell-to-cell transitions and gene-expression evolution over a long timespan.
Contrastive Video-Language Segmentation
We focus on the problem of segmenting a certain object referred by a natural language sentence in video content, at the core of formulating a pinpoint vision-language relation.
Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generation
To deal with this problem, instead of introducing knowledge base as the input, we force the model to learn a better semantic representation by predicting the information in the knowledge base, only based on the input context.
Detecting Speaker Personas from Conversational Texts
This is a many-to-many semantic matching task because both contexts and personas in SPD are composed of multiple sentences.
Flexible Clustered Federated Learning for Client-Level Data Distribution Shift
Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally.
UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset.
Investigation of Practical Aspects of Single Channel Speech Separation for ASR
Speech separation has been successfully applied as a frontend processing module of conversation transcription systems thanks to its ability to handle overlapped speech and its flexibility to combine with downstream tasks such as automatic speech recognition (ASR).
Saying the Unseen: Video Descriptions via Dialog Agents
Current vision and language tasks usually take complete visual data (e. g., raw images or videos) as input, however, practical scenarios may often consist the situations where part of the visual information becomes inaccessible due to various reasons e. g., restricted view with fixed camera or intentional vision block for security concerns.
VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild
To the best of our knowledge, our VSPW is the first attempt to tackle the challenging video scene parsing task in the wild by considering diverse scenarios.
Exploring Heterogeneous Clues for Weakly-Supervised Audio-Visual Video Parsing
Previous works take the overall event labels to supervise both audio and visual model predictions.
Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition
In this work, we perform LM fusion in the minimum WER (MWER) training of an E2E model to obviate the need for LM weights tuning during inference.
Template-Based Named Entity Recognition Using BART
To address the issue, we propose a template-based method for NER, treating NER as a language model ranking problem in a sequence-to-sequence framework, where original sentences and statement templates filled by candidate named entity span are regarded as the source sequence and the target sequence, respectively.
Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentation
Referring video object segmentation (RVOS) aims to segment video objects with the guidance of natural language reference.
One-shot visual object segmentation
Semantic Segmentation
+2
Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone
Transcribing meetings containing overlapped speech with only a single distant microphone (SDM) has been one of the most challenging problems for automatic speech recognition (ASR).
ClawCraneNet: Leveraging Object-level Relation for Text-based Video Segmentation
Text-based video segmentation is a challenging task that segments out the natural language referred objects in videos.
Ranked #3 on
Referring Expression Segmentation
on A2D Sentences
(Precision@0.9 metric)
Learning Audio-Visual Correlations from Variational Cross-Modal Generation
People can easily imagine the potential sound while seeing an event.
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner.
Learning to Anticipate Egocentric Actions by Imagination
We further improve ImagineRNN by residual anticipation, i. e., changing its target to predicting the feature difference of adjacent frames instead of the frame content.
Connection-Adaptive Meta-Learning
In this paper, we aim to obtain better meta-learners by co-optimizing the architecture and meta-weights simultaneously.
Formality Style Transfer with Shared Latent Space
Conventional approaches for formality style transfer borrow models from neural machine translation, which typically requires massive parallel data for training.
Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformer
With its strong modeling capacity that comes from a multi-head and multi-layer structure, Transformer is a very powerful model for learning a sequential representation and has been successfully applied to speech separation recently.
Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset
Recently, Transformer based end-to-end models have achieved great success in many areas including speech recognition.
Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents
With the arising concerns for the AI systems provided with direct access to abundant sensitive information, researchers seek to develop more reliable AI with implicit information sources.
Continuous Speech Separation with Conformer
Continuous speech separation plays a vital role in complicated speech related tasks such as conversation transcription.
On Commonsense Cues in BERT for Solving Commonsense Tasks
We quantitatively investigate the presence of structural commonsense cues in BERT when solving commonsense tasks, and the importance of such cues for the model prediction.
A Retrieve-and-Rewrite Initialization Method for Unsupervised Machine Translation
The commonly used framework for unsupervised machine translation builds initial translation models of both translation directions, and then performs iterative back-translation to jointly boost their translation performance.
Abnormal activity capture from passenger flow of elevator based on unsupervised learning and fine-grained multi-label recognition
We present a work-flow which aims at capturing residents' abnormal activities through the passenger flow of elevator in multi-storey residence buildings.
On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
Among all three E2E models, transformer-AED achieved the best accuracy in both streaming and non-streaming mode.
Curriculum Pre-training for End-to-End Speech Translation
End-to-end speech translation poses a heavy burden on the encoder, because it has to transcribe, understand, and learn cross-lingual semantics simultaneously.
MuTual: A Dataset for Multi-Turn Dialogue Reasoning
Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques.
Unsupervised Person Re-identification via Softened Similarity Learning
Person re-identification (re-ID) is an important topic in computer vision.
Low Latency End-to-End Streaming Speech Recognition with a Scout Network
The attention-based Transformer model has achieved promising results for speech recognition (SR) in the offline mode.
Audio and Speech Processing
Symbiotic Attention with Privileged Information for Egocentric Action Recognition
Due to the large action vocabulary in egocentric video datasets, recent studies usually utilize a two-branch structure for action recognition, ie, one branch for verb classification and the other branch for noun classification.
Ranked #2 on
Egocentric Activity Recognition
on EGTEA
Semantic Mask for Transformer based End-to-End Speech Recognition
Attention-based encoder-decoder model has achieved impressive results for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks.
Harnessing Pre-Trained Neural Networks with Rules for Formality Style Transfer
Formality text style transfer plays an important role in various NLP applications, such as non-native speaker assistants and child education.
Unsupervised Context Rewriting for Open Domain Conversation
Context modeling has a pivotal role in open domain conversation.
Dual Attention Matching for Audio-Visual Event Localization
The duration of these segments is usually short, making the visual and acoustic feature of each segment possibly not well aligned.
Gated Channel Transformation for Visual Recognition
This lightweight layer incorporates a simple l2 normalization, enabling our transformation unit applicable to operator-level without much increase of additional parameters.
Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
End-to-end speech translation, a hot topic in recent years, aims to translate a segment of audio into a specific language with an end-to-end model.
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
Pre-training has proven to be effective in unsupervised machine translation due to its ability to model deep context information in cross-lingual scenarios.
Cascaded Revision Network for Novel Object Captioning
By this novel cascaded captioning-revising mechanism, CRN can accurately describe images with unseen objects.
Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
In this report, we present the Baidu-UTS submission to the EPIC-Kitchens Action Recognition Challenge in CVPR 2019.
Revisiting EmbodiedQA: A Simple Baseline and Beyond
In this paper, we empirically study this problem and introduce 1) a simple yet effective baseline that achieves promising performance; 2) an easier and practical setting for EmbodiedQA where an agent has a chance to adapt the trained model to a new environment before it actually answers users questions.
Auto-ReID: Searching for a Part-aware ConvNet for Person Re-Identification
We propose to automatically search for a CNN architecture that is specifically suitable for the reID task.
Ranked #8 on
Person Re-Identification
on CUHK03 detected
Text Morphing
In this paper, we introduce a novel natural language generation task, termed as text morphing, which targets at generating the intermediate sentences that are fluency and smooth with the two input sentences.
Neural Melody Composition from Lyrics
In this paper, we study a novel task that learns to compose music from natural language.
Keyphrase Generation with Correlation Constraints
In this paper, we study automatic keyphrase generation.
Towards Explainable and Controllable Open Domain Dialogue Generation with Dialogue Acts
We study open domain dialogue generation with dialogue acts designed to explain how people engage in social chat.
Dictionary-Guided Editing Networks for Paraphrase Generation
An intuitive way for a human to write paraphrase sentences is to replace words or phrases in the original sentence with their corresponding synonyms and make necessary changes to ensure the new sentences are fluent and grammatically correct.
Response Generation by Context-aware Prototype Editing
Open domain response generation has achieved remarkable progress in recent years, but sometimes yields short and uninformative responses.
Exploit the Unknown Gradually: One-Shot Video-Based Person Re-Identification by Stepwise Learning
We focus on the one-shot learning for video-based person re-Identification (re-ID).
Learning Matching Models with Weak Supervision for Response Selection in Retrieval-based Chatbots
We propose a method that can leverage unlabeled data to learn a matching model for response selection in retrieval-based chatbots.
Decoupled Novel Object Captioner
Thus, the sequence model can be decoupled from the novel object descriptions.
Towards Interpretable Chit-chat: Open Domain Dialogue Generation with Dialogue Acts
Conventional methods model open domain dialogue generation as a black box through end-to-end learning from large scale conversation data.
Neural Response Generation with Dynamic Vocabularies
We study response generation for open domain conversation in chatbots.
A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots
The task requires matching a response candidate with a conversation context, whose challenges include how to recognize important parts of the context, and how to model the relationships among utterances in the context.
Beihang-MSRA at SemEval-2017 Task 3: A Ranking System with Neural Matching Features for Community Question Answering
This paper presents the system in SemEval-2017 Task 3, Community Question Answering (CQA).
Non-Convex Weighted Lp Nuclear Norm based ADMM Framework for Image Restoration
Since the matrix formed by nonlocal similar patches in a natural image is of low rank, the nuclear norm minimization (NNM) has been widely used in various image processing studies.
Non-Convex Weighted Lp Minimization based Group Sparse Representation Framework for Image Denoising
Nonlocal image representation or group sparsity has attracted considerable interest in various low-level vision tasks and has led to several state-of-the-art image denoising techniques, such as BM3D, LSSC.
Improving Person Re-identification by Attribute and Identity Learning
Person re-identification (re-ID) and attribute recognition share a common target at learning pedestrian descriptions.
Ranked #66 on
Person Re-Identification
on DukeMTMC-reID
Hierarchical Recurrent Attention Network for Response Generation
With the word level attention, hidden vectors of a word level encoder are synthesized as utterance vectors and fed to an utterance level encoder to construct hidden representations of the context.
Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots
Existing work either concatenates utterances in context or matches a response with a highly abstract context vector finally, which may lose relationships among utterances or important contextual information.
Ranked #7 on
Conversational Response Selection
on RRS
Knowledge Enhanced Hybrid Neural Network for Text Matching
Long text brings a big challenge to semantic matching due to their complicated semantic and syntactic structures.
Detecting Context Dependent Messages in a Conversational Environment
While automatic response generation for building chatbot systems has drawn a lot of attention recently, there is limited understanding on when we need to consider the linguistic context of an input text in the generation process.
Topic Aware Neural Response Generation
We consider incorporating topic information into the sequence-to-sequence framework to generate informative and interesting responses for chatbots.
Response Selection with Topic Clues for Retrieval-based Chatbots
The message vector, the response vector, and the two topic vectors are fed to neural tensors to calculate a matching score.
