Search Results for author:
Found 95 papers, 37 papers with code
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation
Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date.
GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video.
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
In this work, we propose a multi-modal AI system named AudioGPT, which complements LLMs (i. e., ChatGPT) with 1) foundation models to process complex audio information and solve numerous understanding and generation tasks; and 2) the input/output interface (ASR, TTS) to support spoken dialogue.
Attributing Image Generative Models using Latent Fingerprints
We use StyleGAN2 and the latent diffusion model to demonstrate the efficacy of our method.
Unbiased Pairwise Learning from Implicit Feedback for Recommender Systems without Biased Variance Control
As pairwise learning suits well for the ranking tasks, the previously proposed unbiased pairwise learning algorithm already achieves state-of-the-art performance.
Overview of the ICASSP 2023 General Meeting Understanding and Generation Challenge (MUG)
ICASSP2023 General Meeting Understanding and Generation Challenge (MUG) focuses on prompting a wide range of spoken language processing (SLP) research on meeting transcripts, as SLP applications are critical to improve users' efficiency in grasping important information in meetings.
MUG: A General Meeting Understanding and Generation Benchmark
To prompt SLP advancement, we establish a large-scale general Meeting Understanding and Generation Benchmark (MUG) to benchmark the performance of a wide range of SLP tasks, including topic segmentation, topic-level and session-level extractive summarization and topic title generation, keyphrase extraction, and action item detection.
Unbiased Learning to Rank with Biased Continuous Feedback
It is a well-known challenge to learn an unbiased ranker with biased feedback.
Item Cold Start Recommendation via Adversarial Variational Auto-encoder Warm-up
The gap between the randomly initialized item ID embedding and the well-trained warm item ID embedding makes the cold items hard to suit the recommendation system, which is trained on the data of historical warm items.
Slate-Aware Ranking for Recommendation
Therefore, the ranking stage is still essential for most applications to provide high-quality candidate set for the re-ranking stage.
How to prepare your task head for finetuning
We identify a significant trend in the effect of changes in this initial energy on the resulting features after fine-tuning.
GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality.
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data.
Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding
To evaluate the effectiveness, we perform multi-level difficulty part language grounding grasping experiments and deploy our proposed model on a real robot.
XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
Our research explores how to provide explanations for NLIs to help users locate the problems and further revise the queries.
Toward Global Sensing Quality Maximization: A Configuration Optimization Scheme for Camera Networks
Then, we form a single quantity that measures the sensing quality of the targets by the camera network.
Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection
Under this assumption, we design new detection methods and indicator scores.
VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement
In this paper, we pose the task of generating sound with a specific timbre given a video input and a reference audio sample.
Principles for generation of reverberation
With these understandings, we developed a STDP based learning rule which could drive the network to remember any presupposed sequence.
Video-Guided Curriculum Learning for Spoken Video Grounding
To rectify the discriminative phonemes and extract video-related information from noisy audio, we develop a novel video-guided curriculum learning (VGCL) during the audio pre-training process, which can make use of the vital visual perceptions to help understand the spoken language and suppress the external noise.
DA$^2$ Dataset: Toward Dexterity-Aware Dual-Arm Grasping
In this paper, we introduce DA$^2$, the first large-scale dual-arm dexterity-aware dataset for the generation of optimal bimanual grasping pairs for arbitrary large objects.
ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech
Through the preliminary study on diffusion model parameterization, we find that previous gradient-based TTS models require hundreds or thousands of iterations to guarantee high sample quality, which poses a challenge for accelerating sampling.
A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation
Furthermore, we take the best of both and design a new loss function to better handle the complicated syntactic multi-modality in real-world datasets.
Approximating Discontinuous Nash Equilibrial Values of Two-Player General-Sum Differential Games
This paper investigates two potential solutions to this problem: a hybrid method that leverages both supervised Nash equilibria and the HJI PDE, and a value-hardening method where a sequence of HJIs are solved with a gradually hardening reward.
Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech
This paper tackles the polyphone disambiguation problem from a concise and novel perspective: we propose Dict-TTS, a semantic-aware generative text-to-speech model with an online website dictionary (the existing prior information in the natural language).
Improving Item Cold-start Recommendation via Model-agnostic Conditional Variational Autoencoder
This paper attempts to tackle the item cold-start problem by generating enhanced warmed-up ID embeddings for cold items with historical data and limited interaction records.
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation
Specifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e. g., rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing to take full advantage of parallelism.
GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech
Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e. g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data.
SSR-GNNs: Stroke-based Sketch Representation with Graph Neural Networks
This paper follows cognitive studies to investigate a graph representation for sketches, where the information of strokes, i. e., parts of a sketch, are encoded on vertices and information of inter-stroke on edges.
FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time.
Ranked #7 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
Better Supervisory Signals by Observing Learning Paths
Observing the learning path not only provides a new perspective for understanding knowledge distillation, overfitting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks.
Learning the Beauty in Songs: Neural Singing Voice Beautifier
Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one.
Revisiting Over-Smoothness in Text to Speech
Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods.
Pseudo Numerical Methods for Diffusion Models on Manifolds
Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs).
Ranked #8 on
Image Generation
on CelebA 64x64
ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech
Specifically, we first introduce a word-level prosody encoder, which quantizes the low-frequency band of the speech and compresses prosody attributes in the latent prosody vector (LPV).
A Mini-Block Fisher Method for Deep Neural Networks
Specifically, our method uses a block-diagonal approximation to the empirical Fisher matrix, where for each layer in the DNN, whether it is convolutional or feed-forward and fully connected, the associated diagonal block is itself block-diagonal and is composed of a large number of mini-blocks of modest size.
Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide.
MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder
However, they face several challenges: 1) the fixed-size speaker embedding is not powerful enough to capture full details of the target timbre; 2) single reference audio does not contain sufficient timbre information of the target speaker; 3) the pitch inconsistency between different speakers also leads to a degradation in the generated voice.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost.
Unbiased Pairwise Learning to Rank in Recommender Systems
To provide personalized high quality recommendation results, conventional systems usually train pointwise rankers to predict the absolute value of objectives and leverage a distinct shallow tower to estimate and alleviate the impact of position bias.
MIC: Model-agnostic Integrated Cross-channel Recommenders
However, existing work are primarily built upon pre-defined retrieval channels, including User-CF (U2U), Item-CF (I2I), and Embedding-based Retrieval (U2I), thus access to the limited correlation between users and items which solely entail from partial information of latent interactions.
SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation
In this work, we propose SingGAN, a generative adversarial network designed for high-fidelity singing voice synthesis.
FedSpeech: Federated Text-to-Speech with Continual Learning
Federated learning enables collaborative training of machine learning models under strict privacy restrictions and federated text-to-speech aims to synthesize natural speech of multiple users with a few audio training samples stored in their devices locally.
PortaSpeech: Portable and High-Quality Generative Text-to-Speech
Non-autoregressive text-to-speech (NAR-TTS) models such as FastSpeech 2 and Glow-TTS can synthesize high-quality speech from the given text in parallel.
Text-To-Speech Synthesis
Vocal Bursts Intensity Prediction
+1
Expressivity of Emergent Languages is a Trade-off between Contextual Complexity and Unpredictability
Researchers are using deep learning models to explore the emergence of language in various language games, where simulated agents interact and develop an emergent language to solve a task.
PhaseFool: Phase-oriented Audio Adversarial Examples via Energy Dissipation
In this work, we propose a novel phase-oriented algorithm named PhaseFool that can efficiently construct imperceptible audio adversarial examples with energy dissipation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SynCLR: A Synthesis Framework for Contrastive Learning of out-of-domain Speech Representations
Learning generalizable speech representations for unseen samples in different domains has been a challenge with ever increasing importance to date.
Targeted Attack on Deep RL-based Autonomous Driving with Learned Visual Patterns
This paper investigates the feasibility of targeted attacks through visually learned patterns placed on physical objects in the environment, a threat model that combines the practicality and effectiveness of the existing ones.
Data Generation Method for Learning a Low-dimensional Safe Region in Safe Reinforcement Learning
Safe reinforcement learning aims to learn a control policy while ensuring that neither the system nor the environment gets damaged during the learning process.
Data-Driven Learning of 3-Point Correlation Functions as Microstructure Representations
This paper considers the open challenge of identifying complete, concise, and explainable quantitative microstructure representations for disordered heterogeneous material systems.
Parallel and High-Fidelity Text-to-Lip Generation
However, the AR decoding manner generates current lip frame conditioned on frames generated previously, which inherently hinders the inference speed, and also has a detrimental effect on the quality of generated lip frames due to error propagation.
EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model
Finally, by showing a comparable performance in the emotional speech synthesis task, we successfully demonstrate the ability of the proposed model.
Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability
Researchers are using deep learning models to explore the emergence of language in various language games, where agents interact and develop an emergent language to solve tasks.
Tensor Normal Training for Deep Learning Models
Based on the so-called tensor normal (TN) distribution, we propose and analyze a brand new approximate natural gradient method, Tensor Normal Training (TNT), which like Shampoo, only requires knowledge of the shape of the training parameters.
DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e. g., mel-spectrogram) given a music score.
Graph Intention Network for Click-through Rate Prediction in Sponsored Search
Moreover, it is difficult for user to jump out of their specific historical behaviors for possible interest exploration, namely weak generalization problem.
Kronecker-factored Quasi-Newton Methods for Deep Learning
Several improvements to the methods in Goldfarb et al. (2020) are also proposed that can be applied to both MLPs and CNNs.
Evolved Massive Stars at Low-metallicity IV. Using 1.6 $μ$m "H-bump" to identify red supergiant stars: a case study of NGC 6822
Further separating RSG candidates from the rest of the LSG candidates is done by using semi-empirical criteria on NIR CMDs and resulted in 323 RSG candidates.
Solar and Stellar Astrophysics Astrophysics of Galaxies
Denoising Text to Speech with Frame-Level Noise Modeling
In DenoiSpeech, we handle real-world noisy speech by modeling the fine-grained frame-level noise with a noise condition module, which is jointly trained with the TTS model.
SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint
Automatic song writing aims to compose a song (lyric and/or melody) by machine, which is an interesting topic in both academia and industry.
Inductive Bias and Language Expressivity in Emergent Communication
Referential games and reconstruction games are the most common game types for studying emergent languages.
TSSRGCN: Temporal Spectral Spatial Retrieval Graph Convolutional Network for Traffic Flow Forecasting
Further analysis indicates that the locality and globality of the traffic networks are critical to traffic flow prediction and the proposed TSSRGCN model can adapt to the various temporal traffic patterns.
Invariant Deep Compressible Covariance Pooling for Aerial Scene Categorization
In this article, we propose a novel invariant deep compressible covariance pooling (IDCCP) to solve nuisance variations in aerial scene categorization.
Decentralized Attribution of Generative Models
Growing applications of generative models have led to new threats such as malicious personation and digital copyright infringement.
PopMAG: Pop Music Accompaniment Generation
To improve harmony, in this paper, we propose a novel MUlti-track MIDI representation (MuMIDI), which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks.
LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
However, there are more than 6, 000 languages in the world and most languages are lack of speech training data, which poses significant challenges when building TTS and ASR systems for extremely low-resource languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire
NAR lipreading is a challenging task that has many difficulties: 1) the discrepancy of sequence lengths between source and target makes it difficult to estimate the length of the output sequence; 2) the conditionally independent behavior of NAR generation lacks the correlation across time which leads to a poor approximation of target distribution; 3) the feature representation ability of encoder can be weak due to lack of effective alignment mechanism; and 4) the removal of AR language model exacerbates the inherent ambiguity problem of lipreading.
Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
SAT contains a hyperparameter k, and each k value defines a SAT task with different degrees of parallelism.
DeepSinger: Singing Voice Synthesis with Data Mined From the Web
DeepSinger has several advantages over previous SVS systems: 1) to the best of our knowledge, it is the first SVS system that directly mines training data from music websites, 2) the lyrics-to-singing alignment model further avoids any human efforts for alignment labeling and greatly reduces labeling cost, 3) the singing model based on a feed-forward Transformer is simple and efficient, by removing the complicated acoustic feature modeling in parametric synthesis and leveraging a reference encoder to capture the timbre of a singer from noisy singing data, and 4) it can synthesize singing voices in multiple languages and multiple singers.
Deep Blue Sonics' Submission to IWSLT 2020 Open Domain Translation Task
We present in this report our submission to IWSLT 2020 Open Domain Translation Task.
SimulSpeech: End-to-End Simultaneous Speech to Text Translation
In this work, we develop SimulSpeech, an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Practical Quasi-Newton Methods for Training Deep Neural Networks
We consider the development of practical stochastic quasi-Newton, and in particular Kronecker-factored block-diagonal BFGS and L-BFGS methods, for training deep neural networks (DNNs).
UWSpeech: Speech to Speech Translation for Unwritten Languages
Existing speech to speech translation systems heavily rely on the text of target language: they usually translate source language either to target text and then synthesize target speech from text, or directly to target speech with target text for auxiliary training.
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e. g., pitch, energy and more accurate duration) as conditional inputs.
Ranked #6 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
MultiSpeech: Multi-Speaker Text to Speech with Transformer
Transformer-based text to speech (TTS) model (e. g., Transformer TTS~\cite{li2019neural}, FastSpeech~\cite{ren2019fastspeech}) has shown the advantages of training and inference efficiency over RNN-based model (e. g., Tacotron~\cite{shen2018natural}) due to its parallel computation in training and/or inference.
Targeting SARS-CoV-2 with AI- and HPC-enabled Lead Generation: A First Data Release
Researchers across the globe are seeking to rapidly repurpose existing drugs or discover new drugs to counter the the novel coronavirus disease (COVID-19) caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).
A Study of Non-autoregressive Model for Sequence Generation
In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all?
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Compositional Languages Emerge in a Neural Iterated Learning Model
The principle of compositionality, which enables natural language to represent complex concepts via a structured combination of simpler ones, allows us to convey an open-ended set of messages using a limited vocabulary.
A Study of Multilingual Neural Machine Translation
Multilingual neural machine translation (NMT) has recently been investigated from different aspects (e. g., pivot translation, zero-shot translation, fine-tuning, or training from scratch) and in different settings (e. g., rich resource and low resource, one-to-many, and many-to-one translation).
Good, Better, Best: Textual Distractors Generation for Multiple-Choice Visual Question Answering via Reinforcement Learning
Multiple-choice VQA has drawn increasing attention from researchers and end-users recently.
The Emergence of Compositional Languages for Numeric Concepts Through Iterated Learning in Neural Agents
Since first introduced, computer simulation has been an increasingly important tool in evolutionary linguistics.
Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks
We present practical Levenberg-Marquardt variants of Gauss-Newton and natural gradient methods for solving non-convex optimization problems that arise in training deep neural networks involving enormous numbers of variables and huge data sets.
FastSpeech: Fast,Robustand Controllable Text-to-Speech
Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i. e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control).
FastSpeech: Fast, Robust and Controllable Text to Speech
In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS.
Ranked #10 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
Almost Unsupervised Text to Speech and Automatic Speech Recognition
Text to speech (TTS) and automatic speech recognition (ASR) are two dual tasks in speech processing and both achieve impressive performance thanks to the recent advance in deep learning and large amount of aligned speech and text data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Multilingual Neural Machine Translation with Knowledge Distillation
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving.
The Period-Luminosity Relations of Red Supergiants in M33 and M31
The period-luminosity (P-L) relation is analyzed for the RSGs in the fundamental mode.
Solar and Stellar Astrophysics Astrophysics of Galaxies
Image Decomposition and Classification through a Generative Model
We demonstrate in this paper that a generative model can be designed to perform classification tasks under challenging settings, including adversarial attacks and input distribution shifts.
Low-cost Measurement of Industrial Shock Signals via Deep Learning Calibration
The results show that the proposed network is capable to map low-end shock signals to its high-end counterparts with satisfactory accuracy.
Augmenting Model Robustness with Transformation-Invariant Attacks
The vulnerability of neural networks under adversarial attacks has raised serious concerns and motivated extensive research.
How Shall I Drive? Interaction Modeling and Motion Planning towards Empathetic and Socially-Graceful Driving
While intelligence of autonomous vehicles (AVs) has significantly advanced in recent years, accidents involving AVs suggest that these autonomous systems lack gracefulness in driving when interacting with human drivers.
Robotics Computer Science and Game Theory
One-Shot Generation of Near-Optimal Topology through Theory-Driven Machine Learning
We introduce a theory-driven mechanism for learning a neural network model that performs generative topology design in one shot given a problem setting, circumventing the conventional iterative process that computational design tasks usually entail.
Gear Training: A new way to implement high-performance model-parallel training
The training of Deep Neural Networks usually needs tremendous computing resources.
Improving Direct Physical Properties Prediction of Heterogeneous Materials from Imaging Data via Convolutional Neural Network and a Morphology-Aware Generative Model
Direct prediction of material properties from microstructures through statistical models has shown to be a potential approach to accelerating computational material design with large design spaces.
Computational Physics Materials Science
Example-Based Image Synthesis via Randomized Patch-Matching
Image and texture synthesis is a challenging task that has long been drawing attention in the fields of image processing, graphics, and machine learning.
