Search Results for author:
Found 97 papers, 32 papers with code
Prior Knowledge and Memory Enriched Transformer for Sign Language Translation
This paper attacks the challenging problem of sign language translation (SLT), which involves not only visual and textual understanding but also additional prior knowledge learning (i. e. performing style, syntax).
AntPivot: Livestream Highlight Detection via Hierarchical Attention Mechanism
In recent days, streaming technology has greatly promoted the development in the field of livestream.
Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech
This paper tackles the polyphone disambiguation problem from a concise and novel perspective: we propose Dict-TTS, a semantic-aware generative text-to-speech model with an online website dictionary (the existing prior information in the natural language).
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation
To alleviate the acoustic multimodal problem, we propose bilateral perturbation, which consists of the style normalization and information enhancement stages, to learn only the linguistic information from speech samples and generate more deterministic representations.
GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis
Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e. g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data.
FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time.
Ranked #6 on
Text-To-Speech Synthesis
on LJSpeech
Fine-Grained Predicates Learning for Scene Graph Generation
The performance of current Scene Graph Generation models is severely hampered by some hard-to-distinguish predicates, e. g., "woman-on/standing on/walking on-beach" or "woman-near/looking at/in front of-child".
Contrastive Learning with Positive-Negative Frame Mask for Music Representation
We devise a novel contrastive learning objective to accommodate both self-augmented positives/negatives sampled from the same music.
End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding
To achieve effective grounding under a limited annotation budget, we investigate one-shot video grounding, and learn to ground natural language in all video frames with solely one frame labeled, in an end-to-end manner.
Learning the Beauty in Songs: Neural Singing Voice Beautifier
Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one.
Revisiting Over-Smoothness in Text to Speech
Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods.
VLAD-VSA: Cross-Domain Face Presentation Attack Detection with Vocabulary Separation and Adaptation
In this paper, the VLAD aggregation method is adopted to quantize local features with visual vocabulary locally partitioning the feature space, and hence preserve the local discriminability.
Pseudo Numerical Methods for Diffusion Models on Manifolds
Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs).
Ranked #4 on
Image Generation
on CelebA 64x64
ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech
Specifically, we first introduce a word-level prosody encoder, which quantizes the low-frequency band of the speech and compresses prosody attributes in the latent prosody vector (LPV).
MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder
However, they face several challenges: 1) the fixed-size speaker embedding is not powerful enough to capture full details of the target timbre; 2) single reference audio does not contain sufficient timbre information of the target speaker; 3) the pitch inconsistency between different speakers also leads to a degradation in the generated voice.
MyoPS: A Benchmark of Myocardial Pathology Segmentation Combining Three-Sequence Cardiac Magnetic Resonance Images
Assessment of myocardial viability is essential in diagnosis and treatment management of patients suffering from myocardial infarction, and classification of pathology on myocardium is the key to this assessment.
Cross-Modal Background Suppression for Audio-Visual Event Localization
Audiovisual Event (AVE) localization requires the model to jointly localize an event by observing audio and visual information.
Wnet: Audio-Guided Video Object Segmentation via Wavelet-Based Cross-Modal Denoising Networks
Audio-Guided video semantic segmentation is a challenging problem in visual analysis and editing, which automatically separates foreground objects from background in a video sequence according to the referring audio expressions.
MLSLT: Towards Multilingual Sign Language Translation
In addition, we also explore zero-shot translation in sign language and find that our model can achieve comparable performance to the supervised BSLT model on some language pairs.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost.
SimulSLT: End-to-End Simultaneous Sign Language Translation
Sign language translation as a kind of technology with profound social significance has attracted growing researchers' interest in recent years.
Generalizable Multi-linear Attention Network
The majority of existing multimodal sequential learning methods focus on how to obtain effective representations and ignore the importance of multimodal fusion.
A Chinese Multi-type Complex Questions Answering Dataset over Wikidata
We finally analyze the performance of SOTA KBQA models on this dataset and identify the challenges facing Chinese KBQA.
FedSpeech: Federated Text-to-Speech with Continual Learning
Federated learning enables collaborative training of machine learning models under strict privacy restrictions and federated text-to-speech aims to synthesize natural speech of multiple users with a few audio training samples stored in their devices locally.
SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation
In this work, we propose SingGAN, a generative adversarial network designed for high-fidelity singing voice synthesis.
Multi-trends Enhanced Dynamic Micro-video Recommendation
In this paper, we propose the DMR framework to explicitly model dynamic multi-trends of users' current preference and make predictions based on both the history and future potential trends.
Stable Prediction on Graphs with Agnostic Distribution Shift
The results demonstrate that our method outperforms various SOTA GNNs for stable prediction on graphs with agnostic distribution shift, including shift caused by node labels and attributes.
Test-time Batch Statistics Calibration for Covariate Shift
An previous solution is test time normalization, which substitutes the source statistics in BN layers with the target batch statistics.
PortaSpeech: Portable and High-Quality Generative Text-to-Speech
Non-autoregressive text-to-speech (NAR-TTS) models such as FastSpeech 2 and Glow-TTS can synthesize high-quality speech from the given text in parallel.
PhaseFool: Phase-oriented Audio Adversarial Examples via Energy Dissipation
In this work, we propose a novel phase-oriented algorithm named PhaseFool that can efficiently construct imperceptible audio adversarial examples with energy dissipation.
SynCLR: A Synthesis Framework for Contrastive Learning of out-of-domain Speech Representations
Learning generalizable speech representations for unseen samples in different domains has been a challenge with ever increasing importance to date.
ST-DDPM: Explore Class Clustering for Conditional Diffusion Probabilistic Models
Score-based generative models involve sequentially corrupting the data distribution with noise and then learns to recover the data distribution based on score matching.
Why Do We Click: Visual Impression-aware News Recommendation
In this work, inspired by the fact that users make their click decisions mostly based on the visual impression they perceive when browsing news, we propose to capture such visual impression information with visual-semantic modeling for news recommendation.
CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation
In this paper, we propose to learn accurate and robust user representations, which are required to be less sensitive to (attack on) noisy behaviors and trust more on the indispensable ones, by modeling counterfactual data distribution.
SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory
Lip reading, aiming to recognize spoken sentences according to the given video of lip movements without relying on the audio stream, has attracted great interest due to its application in many scenarios.
Parallel and High-Fidelity Text-to-Lip Generation
However, the AR decoding manner generates current lip frame conditioned on frames generated previously, which inherently hinders the inference speed, and also has a detrimental effect on the quality of generated lip frames due to error propagation.
Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval
Toward this end, we contribute a multi-modal relational graph to capture the interactions among objects from the visual and textual content to identify the differences among similar video moment candidates.
Cascaded Prediction Network via Segment Tree for Temporal Video Grounding
Temporal video grounding aims to localize the target segment which is semantically aligned with the given sentence in an untrimmed video.
EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model
Finally, by showing a comparable performance in the emotional speech synthesis task, we successfully demonstrate the ability of the proposed model.
Learning to Rehearse in Long Sequence Memorization
Further, we design a history sampler to select informative fragments for rehearsal training, making the memory focus on the crucial information.
DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e. g., mel-spectrogram) given a music score.
Modeling High-order Interactions across Multi-interests for Micro-video Recommendation
Personalized recommendation system has become pervasive in various video platform.
To Learn Effective Features: Understanding the Task-Specific Adaptation of MAML
Model Agnostic Meta-Learning~(MAML)~(\cite{finn2017model}) is one of the most well-known gradient-based meta learning algorithms, that learns the meta-initialization through the inner and outer optimization loop.
Continual Memory: Can We Reason After Long-Term Memorization?
Existing reasoning tasks often follow the setting of "reasoning while experiencing", which has an important assumption that the raw contents can be always accessed while reasoning.
Cortical Surface Shape Analysis Based on Alexandrov Polyhedra
Shape analysis has been playing an important role in early diagnosis and prognosis of neurodegenerative diseases such as Alzheimer's diseases (AD).
Counterfactual Contrastive Learning for Weakly-Supervised Vision-Language Grounding
Weakly-supervised vision-language grounding aims to localize a target moment in a video or a specific region in an image according to the given sentence query, where only video-level or image-level sentence annotations are provided during training.
Future-Aware Diverse Trends Framework for Recommendation
By diverse trends, supposing the future preferences can be diversified, we propose the diverse trends extractor and the time-aware mechanism to represent the possible trends of preferences for a given user with multiple vectors.
MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination
In this paper, we investigate the problem of text-to-pedestrian synthesis, which has many potential applications in art, design, and video surveillance.
Regularized Two-Branch Proposal Networks for Weakly-Supervised Moment Retrieval in Videos
Thus, these methods fail to distinguish the target moment from plausible negative moments.
PopMAG: Pop Music Accompaniment Generation
To improve harmony, in this paper, we propose a novel MUlti-track MIDI representation (MuMIDI), which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks.
Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video Grounding
Spatio-temporal video grounding aims to retrieve the spatio-temporal tube of a queried object according to the given sentence.
DeVLBert: Learning Deconfounded Visio-Linguistic Representations
In this paper, we propose to investigate the problem of out-of-domain visio-linguistic pretraining, where the pretraining data distribution differs from that of downstream data on which the pretrained model will be fine-tuned.
Poet: Product-oriented Video Captioner for E-commerce
Then, based on the aspects of the video-associated product, we perform knowledge-enhanced spatial-temporal inference on those graphs for capturing the dynamic change of fine-grained product-part characteristics.
FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire
NAR lipreading is a challenging task that has many difficulties: 1) the discrepancy of sequence lengths between source and target makes it difficult to estimate the length of the output sequence; 2) the conditionally independent behavior of NAR generation lacks the correlation across time which leads to a poor approximation of target distribution; 3) the feature representation ability of encoder can be weak due to lack of effective alignment mechanism; and 4) the removal of AR language model exacerbates the inherent ambiguity problem of lipreading.
Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
SAT contains a hyperparameter k, and each k value defines a SAT task with different degrees of parallelism.
DeepSinger: Singing Voice Synthesis with Data Mined From the Web
DeepSinger has several advantages over previous SVS systems: 1) to the best of our knowledge, it is the first SVS system that directly mines training data from music websites, 2) the lyrics-to-singing alignment model further avoids any human efforts for alignment labeling and greatly reduces labeling cost, 3) the singing model based on a feed-forward Transformer is simple and efficient, by removing the complicated acoustic feature modeling in parametric synthesis and leveraging a reference encoder to capture the timbre of a singer from noisy singing data, and 4) it can synthesize singing voices in multiple languages and multiple singers.
SimulSpeech: End-to-End Simultaneous Speech to Text Translation
In this work, we develop SimulSpeech, an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently.
Comprehensive Information Integration Modeling Framework for Video Titling
In e-commerce, consumer-generated videos, which in general deliver consumers' individual preferences for the different aspects of certain products, are massive in volume.
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e. g., pitch, energy and more accurate duration) as conditional inputs.
A Study of Non-autoregressive Model for Sequence Generation
In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all?
A Generic Network Compression Framework for Sequential Recommender Systems
Current state-of-the-art sequential recommender models are typically based on a sandwich-structured deep neural network, where one or more middle (hidden) layers are placed between the input embedding layer and output softmax layer.
Grounded and Controllable Image Completion by Incorporating Lexical Semantics
Existing image completion procedure is highly subjective by considering only visual context, which may trigger unpredictable results which are plausible but not faithful to a grounded knowledge.
Convolutional Hierarchical Attention Network for Query-Focused Video Summarization
This paper addresses the task of query-focused video summarization, which takes user's query and a long video as inputs and aims to generate a query-focused video summary.
Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences
In this paper, we consider a novel task, Spatio-Temporal Video Grounding for Multi-Form Sentences (STVG).
Bi-Decoder Augmented Network for Neural Machine Translation
Neural Machine Translation (NMT) has become a popular technology in recent years, and the encoder-decoder framework is the mainstream among all the methods.
Leveraging Long and Short-Term Information in Content-Aware Movie Recommendation via Adversarial Training
Movie recommendation systems provide users with ranked lists of movies based on individual’s preferences and constraints.
Weakly-Supervised Video Moment Retrieval via Semantic Completion Network
Video moment retrieval is to search the moment that is most relevant to the given natural language query.
Video Dialog via Progressive Inference and Cross-Transformer
Video dialog is a new and challenging task, which requires the agent to answer questions combining video information with dialog history.
A Better Way to Attend: Attention with Trees for Video Question Answering
We propose a new attention model for video question answering.
Personalized Hashtag Recommendation for Micro-videos
The hashtags, that a user provides to a post (e. g., a micro-video), are the ones which in her mind can well describe the post content where she is interested in.
Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference
We observe that people usually use some discourse markers such as "so" or "but" to represent the logical relationship between two sentences.
Ranked #12 on
Natural Language Inference
on SNLI
MacNet: Transferring Knowledge from Machine Comprehension to Sequence-to-Sequence Models
In this paper, we transfer knowledge learned from machine comprehension to the sequence-to-sequence tasks to deepen the understanding of the text.
Weak Supervision Enhanced Generative Network for Question Generation
More specifically, we devise a discriminator, Relation Guider, to capture the relations between the whole passage and the associated answer and then the Multi-Interaction mechanism is deployed to transfer the knowledge dynamically for our question generation system.
Localizing Unseen Activities in Video via Image Query
Thus, we consider a new task to localize unseen activities in videos via image queries, named Image-Based Activity Localization.
Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks
Concretely, we first develop a hierarchical convolutional self-attention encoder to efficiently model long-form video contents, which builds the hierarchical structure for video sequences and captures question-aware long-range dependencies from video context.
Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
In this work, we propose a deep progressive quantization (DPQ) model, as an alternative to PQ, for large scale image retrieval.
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA).
Ranked #4 on
Video Question Answering
on ActivityNet-QA
Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos
Query-based moment retrieval aims to localize the most relevant moment in an untrimmed video according to the given natural language query.
FastSpeech: Fast,Robustand Controllable Text-to-Speech
Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i. e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control).
FastSpeech: Fast, Robust and Controllable Text to Speech
In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS.
Ranked #9 on
Text-To-Speech Synthesis
on LJSpeech
Almost Unsupervised Text to Speech and Automatic Speech Recognition
Text to speech (TTS) and automatic speech recognition (ASR) are two dual tasks in speech processing and both achieve impressive performance thanks to the recent advance in deep learning and large amount of aligned speech and text data.
Multilingual Neural Machine Translation with Knowledge Distillation
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving.
Improving Automatic Source Code Summarization via Deep Reinforcement Learning
To the best of our knowledge, most state-of-the-art approaches follow an encoder-decoder framework which encodes the code into a hidden space and then decode it into natural language space, suffering from two major drawbacks: a) Their encoders only consider the sequential content of code, ignoring the tree structure which is also critical for the task of code summarization, b) Their decoders are typically trained to predict the next word by maximizing the likelihood of next ground-truth word with previous ground-truth word given.
Improved Dynamic Memory Network for Dialogue Act Classification with Adversarial Training
Dialogue Act (DA) classification is a challenging problem in dialogue interpretation, which aims to attach semantic labels to utterances and characterize the speaker's intention.
Ranked #4 on
Dialogue Act Classification
on Switchboard corpus
Dial2Desc: End-to-end Dialogue Description Generation
We first propose a new task named Dialogue Description (Dial2Desc).
Textually Guided Ranking Network for Attentional Image Retweet Modeling
In this paper, we study the problem of image retweet prediction in social media, which predicts the image sharing behavior that the user reposts the image tweets from their followees.
Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding
Visual grounding aims to localize an object in an image referred to by a textual query phrase.
Ranked #7 on
Phrase Grounding
on Flickr30k Entities Test
Investigating Capsule Networks with Dynamic Routing for Text Classification
In this study, we explore capsule networks with dynamic routing for text classification.
Ranked #6 on
Sentiment Analysis
on MR
Dialogue Act Recognition via CRF-Attentive Structured Network
Dialogue Act Recognition (DAR) is a challenging problem in dialogue interpretation, which aims to attach semantic labels to utterances and characterize the speaker's intention.
Keyword-based Query Comprehending via Multiple Optimized-Demand Augmentation
In this paper, we propose a novel neural network system that consists a Demand Optimization Model based on a passage-attention neural machine translation and a Reader Model that can find the answer given the optimized question.
Smarnet: Teaching Machines to Read and Comprehend Like Human
Machine Comprehension (MC) is a challenging task in Natural Language Processing field, which aims to guide the machine to comprehend a passage and answer the given question.
Ranked #34 on
Question Answering
on SQuAD1.1 dev
Identifying and Tracking Sentiments and Topics from Social Media Texts during Natural Disasters
We study the problem of identifying the topics and sentiments and tracking their shifts from social media texts in different geographical regions during emergencies and disasters.
MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
Machine comprehension(MC) style question answering is a representative problem in natural language processing.
Ranked #19 on
Question Answering
on TriviaQA
Video Question Answering via Attribute-Augmented Attention Network Learning
Video Question Answering is a challenging problem in visual information retrieval, which provides the answer to the referenced video content according to the question.
The Forgettable-Watcher Model for Video Question Answering
Then we propose a TGIF-QA dataset for video question answering with the help of automatic question generation.
Question Retrieval for Community-based Question Answering via Heterogeneous Network Integration Learning
The challenges in this task are the lexical gaps between questions for the word ambiguity and word mismatch problem.
User Personalized Satisfaction Prediction via Multiple Instance Deep Learning
However, it is not effective for users to attain answers within minutes.
