Search Results for author:
Found 150 papers, 53 papers with code
Prior Knowledge and Memory Enriched Transformer for Sign Language Translation
This paper attacks the challenging problem of sign language translation (SLT), which involves not only visual and textual understanding but also additional prior knowledge learning (i. e. performing style, syntax).
Beyond Two-Tower Matching: Learning Sparse Retrievable Cross-Interactions for Recommendation
Two-tower models are a prevalent matching framework for recommendation, which have been widely deployed in industrial applications.
Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks
This mechanism leverages audio and visual modalities as soft prompts to dynamically adjust the parameters of pre-trained models based on the current multi-modal input features.
Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models
We propose the first unsupervised and learning-based method to identify interpretable directions in h-space of pre-trained diffusion models.
Extending Multi-modal Contrastive Representations
Inspired by recent C-MCR, this paper proposes Extending Multimodal Contrastive Representation (Ex-MCR), a training-efficient and paired-data-free method to flexibly learn unified contrastive representation space for more than three modalities by integrating the knowledge of existing MCR spaces.
Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer
Direct speech-to-speech translation (S2ST) with discrete self-supervised representations has achieved remarkable accuracy, but is unable to preserve the speaker timbre of the source speech during translation.
TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models
The dataset comprises 236, 220 pairs of style prompt in natural text descriptions with five style factors and corresponding speech samples.
Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes
This paper presents Chat-3D, which combines the 3D visual perceptual ability of pre-trained 3D representations and the impressive reasoning and conversation capabilities of advanced LLMs to achieve the first universal dialogue systems for 3D scenes.
3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding
3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description.
DisCover: Disentangled Music Representation Learning for Cover Song Identification
We analyze the CSI task in a disentanglement view with the causal graph technique, and identify the intra-version and inter-version effects biasing the invariant learning.
Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Grounding
To accomplish this, we design a novel semantic matching model that analyzes the semantic similarity between object proposals and sentences in a coarse-to-fine manner.
Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts
In this paper, we introduce Mega-TTS 2, a generic zero-shot multispeaker TTS model that is capable of synthesizing speech for unseen speakers with arbitrary-length prompts.
Gloss Attention for Gloss-free Sign Language Translation
We find that it can provide two aspects of information for the model, 1) it can help the model implicitly learn the location of semantic boundaries in continuous sign language videos, 2) it can help the model understand the sign language video globally.
Ranked #2 on
Gloss-free Sign Language Translation
on PHOENIX14T
MSSRNet: Manipulating Sequential Style Representation for Unsupervised Text Style Transfer
Our proposed method addresses this issue by assigning individual style vector to each token in a text, allowing for fine-grained control and manipulation of the style strength.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and full-shot lip-reading methods.
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
3) We further use a VQGAN-based acoustic model to generate the spectrogram and a latent code language model to fit the distribution of prosody, since prosody changes quickly over time in a sentence, and language models can capture both local and long-range dependencies.
Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
We are interested in a novel task, namely low-resource text-to-talking avatar.
Detector Guidance for Multi-Object Text-to-Image Generation
Previous works identify the problem of information mixing in the CLIP text encoder and introduce the T5 text encoder or incorporate strong prior knowledge to assist with the alignment.
Make-A-Voice: Unified Voice Synthesis With Discrete Representation
Various applications of voice synthesis have been developed independently despite the fact that they generate "voice" as output in common.
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Finally, we use LLMs to augment and transform a large amount of audio-label data into audio-text datasets to alleviate the problem of scarcity of temporal data.
Ranked #4 on
Audio Generation
on AudioCaps
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation
Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date.
Connecting Multi-modal Contrastive Representations
This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR).
ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer
To mitigate the data scarcity in learning visual acoustic information, we 1) introduce a self-supervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leverage the diffusion transformer scalable in terms of parameters and capacity to learn visual scene information.
Wav2SQL: Direct Generalizable Speech-To-SQL Parsing
Speech-to-SQL (S2SQL) aims to convert spoken questions into SQL queries given relational databases, which has been traditionally implemented in a cascaded manner while facing the following challenges: 1) model training is faced with the major issue of data scarcity, where limited parallel data is available; and 2) the systems should be robust enough to handle diverse out-of-domain speech samples that differ from the source data.
RMSSinger: Realistic-Music-Score based Singing Voice Synthesis
To tackle these challenges, we propose RMSSinger, the first RMS-SVS method, which takes realistic music scores as input, eliminating most of the tedious manual annotation and avoiding the aforementioned inconvenience.
AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment
The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings while facing a major challenge: the alignment between the target (singing) pitch contour and the source (speech) content is difficult to learn in a text-free situation.
Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations
Moreover, a Knowledge-Enhance Encoder (KEE) is proposed to leverage SGK as input to further enhance structured representations.
ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos
A recent benchmark AGQA poses a promising paradigm to generate QA pairs automatically from pre-annotated scene graphs, enabling it to measure diverse reasoning abilities with granular control.
Denoising Multi-modal Sequential Recommenders with Contrastive Learning
In contrast, modalities that do not cause users' behaviors are potential noises and might mislead the learning of a recommendation model.
GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video.
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
In this work, we propose a multi-modal AI system named AudioGPT, which complements LLMs (i. e., ChatGPT) with 1) foundation models to process complex audio information and solve numerous understanding and generation tasks; and 2) the input/output interface (ASR, TTS) to support spoken dialogue.
Set-Based Face Recognition Beyond Disentanglement: Burstiness Suppression With Variance Vocabulary
Thus we propose to separate the identity features with the variance features in a light-weighted set-based disentanglement framework.
DATE: Domain Adaptive Product Seeker for E-commerce
Then, we present two cooperative seekers to simultaneously search the image for PR and localize the product for PG.
Overview of the ICASSP 2023 General Meeting Understanding and Generation Challenge (MUG)
ICASSP2023 General Meeting Understanding and Generation Challenge (MUG) focuses on prompting a wide range of spoken language processing (SLP) research on meeting transcripts, as SLP applications are critical to improve users' efficiency in grasping important information in meetings.
MUG: A General Meeting Understanding and Generation Benchmark
To prompt SLP advancement, we establish a large-scale general Meeting Understanding and Generation Benchmark (MUG) to benchmark the performance of a wide range of SLP tasks, including topic segmentation, topic-level and session-level extractive summarization and topic title generation, keyphrase extraction, and action item detection.
MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition
However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on visual speech.
ShiftDDPMs: Exploring Conditional Diffusion Models by Shifting Diffusion Trajectories
In this paper, we propose a novel and flexible conditional diffusion model by introducing conditions into the forward process.
GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality.
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data.
Ranked #8 on
Audio Generation
on AudioCaps
WINNER: Weakly-Supervised hIerarchical decompositioN and aligNment for Spatio-tEmporal Video gRounding
WINNER first builds the language decomposition tree in a bottom-up manner, upon which the structural attention mechanism and top-down feature backtracking jointly build a multi-modal decomposition tree, permitting a hierarchical understanding of unstructured videos.
Exploring Group Video Captioning with Efficient Relational Approximation
In this study, we propose a new task, group video captioning, which aims to infer the desired content among a group of target videos and describe it with another group of related reference videos.
Open-Vocabulary Object Detection With an Open Corpus
For the classification task, we introduce an open corpus classifier by reconstructing original classifier with similar words in text space.
Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models
These imply that the gap corresponds to the lost information of the image, and we can reconstruct the image by filling the gap.
DopplerBAS: Binaural Audio Synthesis Addressing Doppler Effect
Recently, binaural audio synthesis (BAS) has emerged as a promising research field for its applications in augmented and virtual realities.
Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection
In this paper, we introduce a new perceptron bias assumption that suggests discriminator models are more sensitive to certain features of the input, leading to the overconfidence problem.
VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement
In this paper, we pose the task of generating sound with a specific timbre given a video input and a reference audio sample.
Frame-Subtitle Self-Supervision for Multi-Modal Video Question Answering
Multi-modal video question answering aims to predict correct answer and localize the temporal boundary relevant to the question.
Video-Guided Curriculum Learning for Spoken Video Grounding
To rectify the discriminative phonemes and extract video-related information from noisy audio, we develop a novel video-guided curriculum learning (VGCL) during the audio pre-training process, which can make use of the vital visual perceptions to help understand the spoken language and suppress the external noise.
AntCritic: Argument Mining for Free-Form and Visually-Rich Financial Comments
The task of argument mining aims to detect all possible argumentative components and identify their relationships automatically.
CCL4Rec: Contrast over Contrastive Learning for Micro-video Recommendation
Micro-video recommender systems suffer from the ubiquitous noises in users' behaviors, which might render the learned user representation indiscriminating, and lead to trivial recommendations (e. g., popular items) or even weird ones that are far beyond users' interests.
Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation
Specifically, Re4 encapsulates three backward flows, i. e., 1) Re-contrast, which drives each interest embedding to be distinct from other interests using contrastive learning; 2) Re-attend, which ensures the interest-item correlation estimation in the forward flow to be consistent with the criterion used in final recommendation; and 3) Re-construct, which ensures that each interest embedding can semantically reflect the information of representative items that relate to the corresponding interest.
ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech
Through the preliminary study on diffusion model parameterization, we find that previous gradient-based TTS models require hundreds or thousands of iterations to guarantee high sample quality, which poses a challenge for accelerating sampling.
FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis
To tackle these problems, we propose FastLTS, a non-autoregressive end-to-end model which can directly synthesize high-quality speech audios from unconstrained talking videos with low latency, and has a relatively small model size.
AntPivot: Livestream Highlight Detection via Hierarchical Attention Mechanism
In recent days, streaming technology has greatly promoted the development in the field of livestream.
Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech
This paper tackles the polyphone disambiguation problem from a concise and novel perspective: we propose Dict-TTS, a semantic-aware generative text-to-speech model with an online website dictionary (the existing prior information in the natural language).
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation
Specifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e. g., rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing to take full advantage of parallelism.
GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech
Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e. g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data.
FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time.
Ranked #7 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
Fine-Grained Predicates Learning for Scene Graph Generation
The performance of current Scene Graph Generation models is severely hampered by some hard-to-distinguish predicates, e. g., "woman-on/standing on/walking on-beach" or "woman-near/looking at/in front of-child".
Contrastive Learning with Positive-Negative Frame Mask for Music Representation
We devise a novel contrastive learning objective to accommodate both self-augmented positives/negatives sampled from the same music.
End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding
To achieve effective grounding under a limited annotation budget, we investigate one-shot video grounding, and learn to ground natural language in all video frames with solely one frame labeled, in an end-to-end manner.
Learning the Beauty in Songs: Neural Singing Voice Beautifier
Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one.
Revisiting Over-Smoothness in Text to Speech
Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods.
VLAD-VSA: Cross-Domain Face Presentation Attack Detection with Vocabulary Separation and Adaptation
In this paper, the VLAD aggregation method is adopted to quantize local features with visual vocabulary locally partitioning the feature space, and hence preserve the local discriminability.
Pseudo Numerical Methods for Diffusion Models on Manifolds
Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs).
Ranked #10 on
Image Generation
on CelebA 64x64
ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech
Specifically, we first introduce a word-level prosody encoder, which quantizes the low-frequency band of the speech and compresses prosody attributes in the latent prosody vector (LPV).
MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder
However, they face several challenges: 1) the fixed-size speaker embedding is not powerful enough to capture full details of the target timbre; 2) single reference audio does not contain sufficient timbre information of the target speaker; 3) the pitch inconsistency between different speakers also leads to a degradation in the generated voice.
MyoPS: A Benchmark of Myocardial Pathology Segmentation Combining Three-Sequence Cardiac Magnetic Resonance Images
Assessment of myocardial viability is essential in diagnosis and treatment management of patients suffering from myocardial infarction, and classification of pathology on myocardium is the key to this assessment.
Wnet: Audio-Guided Video Object Segmentation via Wavelet-Based Cross-Modal Denoising Networks
Audio-Guided video semantic segmentation is a challenging problem in visual analysis and editing, which automatically separates foreground objects from background in a video sequence according to the referring audio expressions.
Cross-Modal Background Suppression for Audio-Visual Event Localization
Audiovisual Event (AVE) localization requires the model to jointly localize an event by observing audio and visual information.
MLSLT: Towards Multilingual Sign Language Translation
In addition, we also explore zero-shot translation in sign language and find that our model can achieve comparable performance to the supervised BSLT model on some language pairs.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost.
SimulSLT: End-to-End Simultaneous Sign Language Translation
Sign language translation as a kind of technology with profound social significance has attracted growing researchers' interest in recent years.
Generalizable Multi-linear Attention Network
The majority of existing multimodal sequential learning methods focus on how to obtain effective representations and ignore the importance of multimodal fusion.
A Chinese Multi-type Complex Questions Answering Dataset over Wikidata
We finally analyze the performance of SOTA KBQA models on this dataset and identify the challenges facing Chinese KBQA.
FedSpeech: Federated Text-to-Speech with Continual Learning
Federated learning enables collaborative training of machine learning models under strict privacy restrictions and federated text-to-speech aims to synthesize natural speech of multiple users with a few audio training samples stored in their devices locally.
SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation
In this work, we propose SingGAN, a generative adversarial network designed for high-fidelity singing voice synthesis.
Stable Prediction on Graphs with Agnostic Distribution Shift
The results demonstrate that our method outperforms various SOTA GNNs for stable prediction on graphs with agnostic distribution shift, including shift caused by node labels and attributes.
Multi-trends Enhanced Dynamic Micro-video Recommendation
In this paper, we propose the DMR framework to explicitly model dynamic multi-trends of users' current preference and make predictions based on both the history and future potential trends.
Test-time Batch Statistics Calibration for Covariate Shift
An previous solution is test time normalization, which substitutes the source statistics in BN layers with the target batch statistics.
PortaSpeech: Portable and High-Quality Generative Text-to-Speech
Non-autoregressive text-to-speech (NAR-TTS) models such as FastSpeech 2 and Glow-TTS can synthesize high-quality speech from the given text in parallel.
Text-To-Speech Synthesis
Vocal Bursts Intensity Prediction
+1
PhaseFool: Phase-oriented Audio Adversarial Examples via Energy Dissipation
In this work, we propose a novel phase-oriented algorithm named PhaseFool that can efficiently construct imperceptible audio adversarial examples with energy dissipation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SynCLR: A Synthesis Framework for Contrastive Learning of out-of-domain Speech Representations
Learning generalizable speech representations for unseen samples in different domains has been a challenge with ever increasing importance to date.
ST-DDPM: Explore Class Clustering for Conditional Diffusion Probabilistic Models
Score-based generative models involve sequentially corrupting the data distribution with noise and then learns to recover the data distribution based on score matching.
Why Do We Click: Visual Impression-aware News Recommendation
In this work, inspired by the fact that users make their click decisions mostly based on the visual impression they perceive when browsing news, we propose to capture such visual impression information with visual-semantic modeling for news recommendation.
CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation
In this paper, we propose to learn accurate and robust user representations, which are required to be less sensitive to (attack on) noisy behaviors and trust more on the indispensable ones, by modeling counterfactual data distribution.
SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory
Lip reading, aiming to recognize spoken sentences according to the given video of lip movements without relying on the audio stream, has attracted great interest due to its application in many scenarios.
Parallel and High-Fidelity Text-to-Lip Generation
However, the AR decoding manner generates current lip frame conditioned on frames generated previously, which inherently hinders the inference speed, and also has a detrimental effect on the quality of generated lip frames due to error propagation.
Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval
Toward this end, we contribute a multi-modal relational graph to capture the interactions among objects from the visual and textual content to identify the differences among similar video moment candidates.
Cascaded Prediction Network via Segment Tree for Temporal Video Grounding
Temporal video grounding aims to localize the target segment which is semantically aligned with the given sentence in an untrimmed video.
EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model
Finally, by showing a comparable performance in the emotional speech synthesis task, we successfully demonstrate the ability of the proposed model.
Learning to Rehearse in Long Sequence Memorization
Further, we design a history sampler to select informative fragments for rehearsal training, making the memory focus on the crucial information.
DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e. g., mel-spectrogram) given a music score.
Modeling High-order Interactions across Multi-interests for Micro-video Recommendation
Personalized recommendation system has become pervasive in various video platform.
Cortical Surface Shape Analysis Based on Alexandrov Polyhedra
Shape analysis has been playing an important role in early diagnosis and prognosis of neurodegenerative diseases such as Alzheimer's diseases (AD).
To Learn Effective Features: Understanding the Task-Specific Adaptation of MAML
Model Agnostic Meta-Learning~(MAML)~(\cite{finn2017model}) is one of the most well-known gradient-based meta learning algorithms, that learns the meta-initialization through the inner and outer optimization loop.
Continual Memory: Can We Reason After Long-Term Memorization?
Existing reasoning tasks often follow the setting of "reasoning while experiencing", which has an important assumption that the raw contents can be always accessed while reasoning.
Counterfactual Contrastive Learning for Weakly-Supervised Vision-Language Grounding
Weakly-supervised vision-language grounding aims to localize a target moment in a video or a specific region in an image according to the given sentence query, where only video-level or image-level sentence annotations are provided during training.
Future-Aware Diverse Trends Framework for Recommendation
By diverse trends, supposing the future preferences can be diversified, we propose the diverse trends extractor and the time-aware mechanism to represent the possible trends of preferences for a given user with multiple vectors.
MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination
In this paper, we investigate the problem of text-to-pedestrian synthesis, which has many potential applications in art, design, and video surveillance.
Hierarchical memory decoder for visual narrating
In this article, we devise a novel memory decoder for visual narrating.
Ranked #13 on
Visual Storytelling
on VIST
Regularized Two-Branch Proposal Networks for Weakly-Supervised Moment Retrieval in Videos
Thus, these methods fail to distinguish the target moment from plausible negative moments.
PopMAG: Pop Music Accompaniment Generation
To improve harmony, in this paper, we propose a novel MUlti-track MIDI representation (MuMIDI), which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks.
Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video Grounding
Spatio-temporal video grounding aims to retrieve the spatio-temporal tube of a queried object according to the given sentence.
Poet: Product-oriented Video Captioner for E-commerce
Then, based on the aspects of the video-associated product, we perform knowledge-enhanced spatial-temporal inference on those graphs for capturing the dynamic change of fine-grained product-part characteristics.
DeVLBert: Learning Deconfounded Visio-Linguistic Representations
In this paper, we propose to investigate the problem of out-of-domain visio-linguistic pretraining, where the pretraining data distribution differs from that of downstream data on which the pretrained model will be fine-tuned.
FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire
NAR lipreading is a challenging task that has many difficulties: 1) the discrepancy of sequence lengths between source and target makes it difficult to estimate the length of the output sequence; 2) the conditionally independent behavior of NAR generation lacks the correlation across time which leads to a poor approximation of target distribution; 3) the feature representation ability of encoder can be weak due to lack of effective alignment mechanism; and 4) the removal of AR language model exacerbates the inherent ambiguity problem of lipreading.
Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
SAT contains a hyperparameter k, and each k value defines a SAT task with different degrees of parallelism.
DeepSinger: Singing Voice Synthesis with Data Mined From the Web
DeepSinger has several advantages over previous SVS systems: 1) to the best of our knowledge, it is the first SVS system that directly mines training data from music websites, 2) the lyrics-to-singing alignment model further avoids any human efforts for alignment labeling and greatly reduces labeling cost, 3) the singing model based on a feed-forward Transformer is simple and efficient, by removing the complicated acoustic feature modeling in parametric synthesis and leveraging a reference encoder to capture the timbre of a singer from noisy singing data, and 4) it can synthesize singing voices in multiple languages and multiple singers.
SimulSpeech: End-to-End Simultaneous Speech to Text Translation
In this work, we develop SimulSpeech, an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Comprehensive Information Integration Modeling Framework for Video Titling
In e-commerce, consumer-generated videos, which in general deliver consumers' individual preferences for the different aspects of certain products, are massive in volume.
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e. g., pitch, energy and more accurate duration) as conditional inputs.
Ranked #6 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
A Study of Non-autoregressive Model for Sequence Generation
In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all?
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
A Generic Network Compression Framework for Sequential Recommender Systems
Current state-of-the-art sequential recommender models are typically based on a sandwich-structured deep neural network, where one or more middle (hidden) layers are placed between the input embedding layer and output softmax layer.
Grounded and Controllable Image Completion by Incorporating Lexical Semantics
Existing image completion procedure is highly subjective by considering only visual context, which may trigger unpredictable results which are plausible but not faithful to a grounded knowledge.
Convolutional Hierarchical Attention Network for Query-Focused Video Summarization
This paper addresses the task of query-focused video summarization, which takes user's query and a long video as inputs and aims to generate a query-focused video summary.
Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences
In this paper, we consider a novel task, Spatio-Temporal Video Grounding for Multi-Form Sentences (STVG).
Bi-Decoder Augmented Network for Neural Machine Translation
Neural Machine Translation (NMT) has become a popular technology in recent years, and the encoder-decoder framework is the mainstream among all the methods.
Leveraging Long and Short-Term Information in Content-Aware Movie Recommendation via Adversarial Training
Movie recommendation systems provide users with ranked lists of movies based on individual’s preferences and constraints.
Weakly-Supervised Video Moment Retrieval via Semantic Completion Network
Video moment retrieval is to search the moment that is most relevant to the given natural language query.
Video Dialog via Progressive Inference and Cross-Transformer
Video dialog is a new and challenging task, which requires the agent to answer questions combining video information with dialog history.
A Better Way to Attend: Attention with Trees for Video Question Answering
We propose a new attention model for video question answering.
Personalized Hashtag Recommendation for Micro-videos
The hashtags, that a user provides to a post (e. g., a micro-video), are the ones which in her mind can well describe the post content where she is interested in.
Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference
We observe that people usually use some discourse markers such as "so" or "but" to represent the logical relationship between two sentences.
Ranked #14 on
Natural Language Inference
on SNLI
MacNet: Transferring Knowledge from Machine Comprehension to Sequence-to-Sequence Models
In this paper, we transfer knowledge learned from machine comprehension to the sequence-to-sequence tasks to deepen the understanding of the text.
Weak Supervision Enhanced Generative Network for Question Generation
More specifically, we devise a discriminator, Relation Guider, to capture the relations between the whole passage and the associated answer and then the Multi-Interaction mechanism is deployed to transfer the knowledge dynamically for our question generation system.
Localizing Unseen Activities in Video via Image Query
Thus, we consider a new task to localize unseen activities in videos via image queries, named Image-Based Activity Localization.
Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks
Concretely, we first develop a hierarchical convolutional self-attention encoder to efficiently model long-form video contents, which builds the hierarchical structure for video sequences and captures question-aware long-range dependencies from video context.
Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
In this work, we propose a deep progressive quantization (DPQ) model, as an alternative to PQ, for large scale image retrieval.
Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos
Query-based moment retrieval aims to localize the most relevant moment in an untrimmed video according to the given natural language query.
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA).
Ranked #15 on
Video Question Answering
on ActivityNet-QA
Visual Question Answering (VQA)
Zero-Shot Video Question Answer
FastSpeech: Fast, Robust and Controllable Text to Speech
In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS.
Ranked #10 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
FastSpeech: Fast,Robustand Controllable Text-to-Speech
Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i. e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control).
Almost Unsupervised Text to Speech and Automatic Speech Recognition
Text to speech (TTS) and automatic speech recognition (ASR) are two dual tasks in speech processing and both achieve impressive performance thanks to the recent advance in deep learning and large amount of aligned speech and text data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Multilingual Neural Machine Translation with Knowledge Distillation
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving.
Improving Automatic Source Code Summarization via Deep Reinforcement Learning
To the best of our knowledge, most state-of-the-art approaches follow an encoder-decoder framework which encodes the code into a hidden space and then decode it into natural language space, suffering from two major drawbacks: a) Their encoders only consider the sequential content of code, ignoring the tree structure which is also critical for the task of code summarization, b) Their decoders are typically trained to predict the next word by maximizing the likelihood of next ground-truth word with previous ground-truth word given.
Improved Dynamic Memory Network for Dialogue Act Classification with Adversarial Training
Dialogue Act (DA) classification is a challenging problem in dialogue interpretation, which aims to attach semantic labels to utterances and characterize the speaker's intention.
Ranked #5 on
Dialogue Act Classification
on Switchboard corpus
Dial2Desc: End-to-end Dialogue Description Generation
We first propose a new task named Dialogue Description (Dial2Desc).
Textually Guided Ranking Network for Attentional Image Retweet Modeling
In this paper, we study the problem of image retweet prediction in social media, which predicts the image sharing behavior that the user reposts the image tweets from their followees.
Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding
Visual grounding aims to localize an object in an image referred to by a textual query phrase.
Ranked #9 on
Phrase Grounding
on Flickr30k Entities Test
Investigating Capsule Networks with Dynamic Routing for Text Classification
In this study, we explore capsule networks with dynamic routing for text classification.
Ranked #7 on
Sentiment Analysis
on MR
Dialogue Act Recognition via CRF-Attentive Structured Network
Dialogue Act Recognition (DAR) is a challenging problem in dialogue interpretation, which aims to attach semantic labels to utterances and characterize the speaker's intention.
Keyword-based Query Comprehending via Multiple Optimized-Demand Augmentation
In this paper, we propose a novel neural network system that consists a Demand Optimization Model based on a passage-attention neural machine translation and a Reader Model that can find the answer given the optimized question.
Smarnet: Teaching Machines to Read and Comprehend Like Human
Machine Comprehension (MC) is a challenging task in Natural Language Processing field, which aims to guide the machine to comprehend a passage and answer the given question.
Ranked #33 on
Question Answering
on SQuAD1.1 dev
Identifying and Tracking Sentiments and Topics from Social Media Texts during Natural Disasters
We study the problem of identifying the topics and sentiments and tracking their shifts from social media texts in different geographical regions during emergencies and disasters.
MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
Machine comprehension(MC) style question answering is a representative problem in natural language processing.
Ranked #32 on
Question Answering
on TriviaQA
Video Question Answering via Attribute-Augmented Attention Network Learning
Video Question Answering is a challenging problem in visual information retrieval, which provides the answer to the referenced video content according to the question.
The Forgettable-Watcher Model for Video Question Answering
Then we propose a TGIF-QA dataset for video question answering with the help of automatic question generation.
User Personalized Satisfaction Prediction via Multiple Instance Deep Learning
However, it is not effective for users to attain answers within minutes.
Question Retrieval for Community-based Question Answering via Heterogeneous Network Integration Learning
The challenges in this task are the lexical gaps between questions for the word ambiguity and word mismatch problem.
