Search Results for author:
Found 283 papers, 118 papers with code
Global Ranking Using Continuous Conditional Random Fields
It can naturally represent the content information of objects as well as the relation information between objects, necessary for global ranking.
Ranking Measures and Loss Functions in Learning to Rank
We show that these loss functions are upper bounds of the measure-based ranking errors.
Statistical Consistency of Top-k Ranking
This paper aims to analyze whether existing listwise ranking methods are statistically consistent in the top-k setting.
A New Probabilistic Model for Rank Aggregation
To avoid these limitations, in this paper, we propose a new model, which is defined with a coset-permutation distance, and models the generation of a permutation as a stagewise process.
Two-Layer Generalization Analysis for Ranking Using Rademacher Average
sampling of queries and the conditional i. i. d sampling of documents per query.
Statistical Consistency of Ranking Methods in A Rank-Differentiable Probability Space
This paper is concerned with the statistical consistency of ranking methods.
A Theoretical Analysis of NDCG Type Ranking Measures
We show that NDCG with logarithmic discount has consistent distinguishability although it converges to the same limit for all ranking functions.
Pure Price of Anarchy for Generalized Second Price Auction
Previous studies have shown that the pure Price Of Anarchy (POA) of GSP is 1. 25 when there are two ad slots and 1. 259 when three ad slots.
Computer Science and Game Theory
Estimation Bias in Multi-Armed Bandit Algorithms for Search Advertising
In search advertising, the search engine needs to select the most profitable advertisements to display, which can be formulated as an instance of online learning with partial feedback, also known as the stochastic multi-armed bandit (MAB) problem.
Agent Behavior Prediction and Its Generalization Analysis
Then we prove a generalization bound for the machine learning algorithms on the behavior data generated by the new Markov chain, which depends on both the Markovian parameters and the covering number of the function class compounded by the loss function for behavior prediction and the behavior prediction model.
Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks
Click prediction is one of the fundamental problems in sponsored search.
A Game-theoretic Machine Learning Approach for Revenue Maximization in Sponsored Search
Sponsored search is an important monetization channel for search engines, in which an auction mechanism is used to select the ads shown to users and determine the prices charged from advertisers.
WordRep: A Benchmark for Research on Learning Word Representations
In this paper, we describe the details of the WordRep collection and show how to use it in different types of machine learning research related to word embedding.
KNET: A General Framework for Learning Word Embedding using Morphological Knowledge
In particular, we introduce a novel neural network architecture called KNET that leverages both contextual information and morphological word similarity built based on morphological knowledge to learn word embeddings.
Generalization Analysis for Game-Theoretic Machine Learning
For Internet applications like sponsored search, cautions need to be taken when using machine learning to optimize their mechanisms (e. g., auction) since self-interested agents in these applications may change their behaviors (and thus the data distribution) in response to the mechanisms.
LightLDA: Big Topic Models on Modest Compute Clusters
When building large-scale machine learning (ML) programs, such as big topic models or deep neural nets, one usually assumes such tasks can only be attempted with industrial-sized clusters with thousands of nodes, which are out of reach for most practitioners or academic researchers.
Thompson Sampling for Budgeted Multi-armed Bandits
In this paper, we extend the Thompson sampling to Budgeted MAB, where there is random cost for pulling an arm and the total cost is constrained by a budget.
Learning Better Word Embedding by Asymmetric Low-Rank Projection of Knowledge Graph
Although these works have achieved certain success, they have neglected some important facts about knowledge graphs: (i) many relationships in knowledge graphs are \emph{many-to-one}, \emph{one-to-many} or even \emph{many-to-many}, rather than simply \emph{one-to-one}; (ii) most head entities and tail entities in knowledge graphs come from very different semantic spaces.
Solving Verbal Comprehension Questions in IQ Test by Knowledge-Powered Word Embedding
Second, we obtain distributed representations of words and relations by leveraging a novel word embedding method that considers the multi-sense nature of words and the relational knowledge among words (or their senses) contained in dictionaries.
On the Depth of Deep Neural Networks: A Theoretical View
First, we derive an upper bound for RA of DNN, and show that it increases with increasing depth.
Sentence Level Recurrent Topic Model: Letting Topics Speak for Themselves
We propose Sentence Level Recurrent Topic Model (SLRTM), a new topic model that assumes the generation of each word within a sentence to depend on both the topic of the sentence and the whole history of its preceding words in the sentence.
Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks
In this framework, we propose to aggregate the local models by ensemble, i. e., averaging the outputs of local models instead of the parameters.
Asynchronous Stochastic Proximal Optimization Algorithms with Variance Reduction
The results verified our theoretical findings and demonstrated the practical efficiency of the asynchronous stochastic proximal algorithms with variance reduction.
Asynchronous Stochastic Gradient Descent with Delay Compensation
We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD.
Generalization Error Bounds for Optimization Algorithms via Stability
Many machine learning tasks can be formulated as Regularized Empirical Risk Minimization (R-ERM), and solved by optimization algorithms such as gradient descent (GD), stochastic gradient descent (SGD), and stochastic variance reduction (SVRG).
LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
Based on the 2-Component shared embedding, we design a new RNN algorithm and evaluate it using the language modeling task on several benchmark datasets.
Dual Learning for Machine Translation
Based on the feedback signals generated during this process (e. g., the language-model likelihood of the output of a model, and the reconstruction error of the original sentence after the primal and dual translations), we can iteratively update the two models until convergence (e. g., using the policy gradient methods).
A Communication-Efficient Parallel Algorithm for Decision Tree
After partitioning the training data onto a number of (e. g., $M$) machines, this algorithm performs both local voting and global voting in each iteration.
Randomized Mechanisms for Selling Reserved Instances in Cloud
We then extend our mechanism to the general case and achieve a competitive ratio $\frac{1}{42\log k\log T}$ for both social welfare and revenue, where $T$ is the ratio of the maximum request length to the minimum request length and $k$ is the ratio of the maximum request value density to the minimum request value density.
Efficient Inexact Proximal Gradient Algorithm for Nonconvex Problems
The proximal gradient algorithm has been popularly used for convex optimization.
Optimization and Control
Adversarial Neural Machine Translation
The goal of the adversary is to differentiate the translation result generated by the NMT model from that by human.
Reinforcement Learning for Learning Rate Control
Stochastic gradient descent (SGD), which updates the model parameters by adding a local gradient times a learning rate at each step, is widely used in model training of machine learning algorithms such as neural networks.
Word-Entity Duet Representations for Document Ranking
This paper presents a word-entity duet framework for utilizing knowledge bases in ad-hoc retrieval.
Dual Supervised Learning
Many supervised learning tasks are emerged in dual forms, e. g., English-to-French translation vs. French-to-English translation, speech recognition vs. text to speech, and image classification vs. image generation.
Large-Scale Low-Rank Matrix Learning with Nonconvex Regularizers
Based on it, we develop a proximal gradient algorithm (and its accelerated variant) with inexact proximal splitting and prove that a convergence rate of O(1/T) where T is the number of iterations is guaranteed.
Slim-DP: A Light Communication Data Parallelism for DNN
However, with the increasing size of DNN models and the large number of workers in practice, this typical data parallelism cannot achieve satisfactory training acceleration, since it usually suffers from the heavy communication cost due to transferring huge amount of information between workers and the parameter server.
Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling
First, we give a mathematical formulation for the practical data processing procedure in distributed machine learning, which we call data partition with global/local shuffling.
Decoding with Value Networks for Neural Machine Translation
Inspired by the success and methodology of AlphaGo, in this paper we propose using a prediction network to improve beam search, which takes the source sentence $x$, the currently available decoding output $y_1,\cdots, y_{t-1}$ and a candidate word $w$ at step $t$ as inputs and predicts the long-term value (e. g., BLEU score) of the partial target sentence if it is completed by the NMT model.
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size.
Deliberation Networks: Sequence Generation Beyond One-Pass Decoding
In this work, we introduce the deliberation process into the encoder-decoder framework and propose deliberation networks for sequence generation.
Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction
Stock trend prediction plays a critical role in seeking maximized profit from stock investment.
Ranked #16 on
Stock Market Prediction
on Astock
$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation
This inconsistence of gradient magnitude across different layers renders optimization of deep neural network with a single learning rate problematic.
Achieving Human Parity on Automatic Chinese to English News Translation
Machine translation has made rapid advances in recent years.
Ranked #3 on
Machine Translation
on WMT 2017 English-Chinese
Conditional Image-to-Image Translation
In this paper, we study a new problem, conditional image-to-image translation, which is to translate an image from the source domain to the target domain conditioned on a given image in the target domain.
Towards Better Text Understanding and Retrieval through Kernel Entity Salience Modeling
The salience model also improves ad hoc search accuracy, providing effective ranking features by modeling the salience of query entities in candidate documents.
Differential Equations for Modeling Asynchronous Algorithms
Then we conduct theoretical analysis on the convergence rates of ASGD algorithm based on the continuous approximation.
Learning to Teach
Teaching plays a very important role in our society, by spreading human knowledge and educating our next generations.
Efficient Sequence Learning with Group Recurrent Networks
Recurrent neural networks have achieved state-of-the-art results in many artificial intelligence tasks, such as language modeling, neural machine translation, speech recognition and so on.
Dense Information Flow for Neural Machine Translation
Recently, neural machine translation has achieved remarkable progress by introducing well-designed deep neural networks into its encoder-decoder framework.
Ranked #68 on
Machine Translation
on WMT2014 English-German
Towards Binary-Valued Gates for Robust LSTM Training
Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling.
Double Path Networks for Sequence to Sequence Learning
In this work we propose Double Path Networks for Sequence to Sequence learning (DPN-S2S), which leverage the advantages of both models by using double path information fusion.
Model-Level Dual Learning
Many artificial intelligence tasks appear in dual forms like English$\leftrightarrow$French translation and speech$\leftrightarrow$text transformation.
Neural Architecture Optimization
The performance predictor and the encoder enable us to perform gradient based optimization in the continuous space to find the embedding of a new architecture with potentially better accuracy.
A Study of Reinforcement Learning for Neural Machine Translation
Recent studies have shown that reinforcement learning (RL) is an effective approach for improving the performance of neural machine translation (NMT) system.
Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
Many previous works have discussed the relationship between error propagation and the \emph{accuracy drop} (i. e., the left part of the translated sentence is often better than its right part in left-to-right decoding models) problem.
FRAGE: Frequency-Agnostic Word Representation
Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks.
Ranked #3 on
Machine Translation
on IWSLT2015 German-English
Capacity Control of ReLU Neural Networks by Basis-path Norm
Motivated by this, we propose a new norm \emph{Basis-path Norm} based on a group of linearly independent paths to measure the capacity of neural networks more accurately.
Target Transfer Q-Learning and Its Convergence Analysis
In this paper, we propose to transfer the Q-function learned in the source task to the target of the Q-learning in the new task when certain safe conditions are satisfied.
Finite Sample Analysis of the GTD Policy Evaluation Algorithms in Markov Setting
(2) The convergence rate is determined by the step size, with the mixing time of the Markov process as the coefficient.
A Convergent Variant of the Boltzmann Softmax Operator in Reinforcement Learning
We then propose the dynamic Boltzmann softmax(DBS) operator to enable the convergence to the optimal value function in value iteration.
Expressiveness in Deep Reinforcement Learning
Based on our observations, we formally define expressiveness of the state extractor as the rank of the matrix composed by representations.
Learning to Teach with Dynamic Loss Functions
Different from typical learning settings in which the loss function of a machine learning model is predefined and fixed, in our framework, the loss function of a machine learning model (we call it student) is defined by another machine learning model (we call it teacher).
Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks
In this paper, we propose an improved variant of RNN, Multi-Channel RNN (MC-RNN), to dynamically capture and leverage local semantic structure information.
On the Local Hessian in Back-propagation
We study the Hessian of the local back-matching loss (local Hessian) and connect it to the efficiency of BP.
Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation
Neural Machine Translation (NMT) has achieved remarkable progress with the quick evolvement of model structures.
Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
Non-autoregressive translation (NAT) models, which remove the dependence on previous target tokens from the inputs of the decoder, achieve significantly inference speedup but at the cost of inferior accuracy compared to autoregressive translation (AT) models.
Non-Autoregressive Machine Translation with Auxiliary Regularization
However, the high efficiency has come at the cost of not capturing the sequential dependency on the target side of translation, which causes NAT to suffer from two kinds of translation errors: 1) repeated translations (due to indistinguishable adjacent decoder hidden states), and 2) incomplete translations (due to incomplete transfer of source side information via the decoder hidden states).
Multilingual Neural Machine Translation with Knowledge Distillation
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving.
A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Network
Traditional solutions on these problems leverage combinatorial optimization with demand and supply forecasting.
Positively Scale-Invariant Flatness of ReLU Neural Networks
That is to say, the minimum with balanced values of basis paths will more likely to be flatter and generalize better.
Reinforcement Learning with Dynamic Boltzmann Softmax Updates
In this paper, we propose to update the value function with dynamic Boltzmann softmax (DBS) operator, which has good convergence property in the setting of planning and learning.
Stabilize Deep ResNet with A Sharp Scaling Factor $τ$
Moreover, for ResNets with normalization layer, adding such a factor $\tau$ also stabilizes the training and obtains significant performance gain for deep ResNet.
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
Recently, G2P conversion is viewed as a sequence to sequence task and modeled by RNN or CNN based encoder-decoder framework.
Ranked #1 on
Text-To-Speech Synthesis
on CMUDict 0.7b
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Adaptive Regret of Convex and Smooth Functions
We investigate online convex optimization in changing environments, and choose the adaptive regret as the performance measure.
TabNN: A Universal Neural Network Solution for Tabular Data
Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all.
Dual Learning: Theoretical Study and Algorithmic Extensions
Based on the theoretical discoveries, we extend dual learning by introducing more related mappings and propose highly symmetric frameworks, cycle dual learning and multipath dual learning, in both of which we can leverage the feedback signals from additional domains to improve the qualities of the mappings.
Multi-Agent Dual Learning
Dual learning has attracted much attention in machine learning, computer vision and natural language processing communities.
Ranked #1 on
Machine Translation
on WMT2016 English-German
G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Optimization on Multiple Manifolds
Optimization on manifold has been widely used in machine learning, to handle optimization problems with constraint.
Hint-based Training for Non-Autoregressive Translation
To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a well-trained ART model to train the NART model.
MASS: Masked Sequence to Sequence Pre-training for Language Generation
Pre-training and fine-tuning, e. g., BERT, have achieved great success in language understanding by transferring knowledge from rich-resource pre-training task to the low/zero-resource downstream tasks.
Almost Unsupervised Text to Speech and Automatic Speech Recognition
Text to speech (TTS) and automatic speech recognition (ASR) are two dual tasks in speech processing and both achieve impressive performance thanks to the recent advance in deep learning and large amount of aligned speech and text data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
FastSpeech: Fast,Robustand Controllable Text-to-Speech
Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i. e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control).
FastSpeech: Fast, Robust and Controllable Text to Speech
In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS.
Ranked #10 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
Soft Contextual Data Augmentation for Neural Machine Translation
While data augmentation is an important trick to boost the accuracy of deep learning methods in computer vision tasks, its study in natural language tasks is still very limited.
Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
Based on that, We proposed an end-to-end algorithm to learn exploration policy by meta-learning.
Beyond Exponentially Discounted Sum: Automatic Learning of Return Function
In reinforcement learning, Return, which is the weighted accumulated future rewards, and Value, which is the expected return, serve as the objective that guides the learning of the policy.
Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
Our theory captures how the convergence of distributed algorithms behaves as the number of machines and the size of local data vary.
Unsupervised Pivot Translation for Distant Languages
In this work, we introduce unsupervised pivot translation for distant languages, which translates a language to a distant language through multiple hops, and the unsupervised translation on each hop is relatively easier than the original direct translation.
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system.
Depth Growing for Neural Machine Translation
While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of neural machine translation (NMT) models for better translation quality remains a challenging problem.
Ranked #11 on
Machine Translation
on WMT2014 English-French
Light Multi-segment Activation for Model Compression
Inspired by the nature of the expressiveness ability in Neural Networks, we propose to use multi-segment activation, which can significantly improve the expressiveness ability with very little cost, in the compact student model.
Representation Degeneration Problem in Training Natural Language Generation Models
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}.
LightMC: A Dynamic and Efficient Multiclass Decomposition Algorithm
Instead of using fixed coding matrix and decoding strategy, LightMC uses a differentiable decoding strategy, which enables it to dynamically optimize the coding matrix and decoding strategy, toward increasing the overall accuracy of multiclass classification, via back propagation jointly with the training of base learners in an iterative way.
Multilingual Neural Machine Translation with Language Clustering
We study two methods for language clustering: (1) using prior knowledge, where we cluster languages according to language family, and (2) using language embedding, in which we represent each language by an embedding vector and cluster them in the embedding space.
Self-paced Ensemble for Highly Imbalanced Massive Data Classification
To tackle this problem, we conduct deep investigations into the nature of class imbalance, which reveals that not only the disproportion between classes, but also other difficulties embedded in the nature of data, especially, noises and class overlapping, prevent us from learning effective classifiers.
Hint-Based Training for Non-Autoregressive Machine Translation
Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency.
Path Space for Recurrent Neural Networks with ReLU Activations
Optimization algorithms like stochastic gradient descent that optimize the neural networks in the vector space of weights, which are not positively scale-invariant.
THE EFFECT OF ADVERSARIAL TRAINING: A THEORETICAL CHARACTERIZATION
It has widely shown that adversarial training (Madry et al., 2018) is effective in defending adversarial attack empirically.
Independence-aware Advantage Estimation
Most of existing advantage function estimation methods in reinforcement learning suffer from the problem of high variance, which scales unfavorably with the time horizon.
STABILITY AND CONVERGENCE THEORY FOR LEARNING RESNET: A FULL CHARACTERIZATION
We show that for standard initialization used in practice, $\tau =1/\Omega(\sqrt{L})$ is a sharp value in characterizing the stability of forward/backward process of ResNet, where $L$ is the number of residual blocks.
Demonstration Actor Critic
One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations.
P-BN: Towards Effective Batch Normalization in the Path Space
Hence, some new algorithms that conduct optimizations directly in the path space (the path space is proven to be PSI) were developed, such as Stochastic Gradient Descent (SGD) in the path space, and it was shown that SGD in the path space is superior to that in the weight space.
Machine Translation With Weakly Paired Documents
1) We provide a simple approach to mine implicitly bilingual sentence pairs from document pairs which can then be used as supervised training signals.
Exploiting Monolingual Data at Scale for Neural Machine Translation
In this work, we study how to use both the source-side and target-side monolingual data for NMT, and propose an effective strategy leveraging both of them.
Ranked #1 on
Machine Translation
on WMT2016 English-German
(SacreBLEU metric, using extra
training data)
Fully Parameterized Quantile Function for Distributional Reinforcement Learning
The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution.
Ranked #3 on
Atari Games
on Atari 2600 Skiing
(using extra training data)
Distributional Reward Decomposition for Reinforcement Learning
Many reinforcement learning (RL) tasks have specific properties that can be leveraged to modify existing RL algorithms to adapt to those tasks and further improve performance, and a general class of such properties is the multiple reward channel.
Microsoft Research Asia's Systems for WMT19
We Microsoft Research Asia made submissions to 11 language directions in the WMT19 news translation tasks.
Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
Non-autoregressive translation (NAT) models remove the dependence on previous target tokens and generate all target tokens in parallel, resulting in significant inference speedup but at the cost of inferior translation accuracy compared to autoregressive translation (AT) models.
Gradient Perturbation is Underrated for Differentially Private Convex Optimization
By using the \emph{expected curvature}, we show that gradient perturbation can achieve a significantly improved utility guarantee that can theoretically justify the advantage of gradient perturbation over other perturbation methods.
Normalization Helps Training of Quantized LSTM
The long-short-term memory (LSTM), though powerful, is memory and computa\x02tion expensive.
Neural Machine Translation with Soft Prototype
Neural machine translation models usually use the encoder-decoder framework and generate translation from left to right (or right to left) without fully utilizing the target-side global information.
A Study of Multilingual Neural Machine Translation
Multilingual neural machine translation (NMT) has recently been investigated from different aspects (e. g., pivot translation, zero-shot translation, fine-tuning, or training from scratch) and in different settings (e. g., rich resource and low resource, one-to-many, and many-to-one translation).
On Layer Normalization in the Transformer Architecture
This motivates us to remove the warm-up stage for the training of Pre-LN Transformers.
Incorporating BERT into Neural Machine Translation
While BERT is more commonly used as fine-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for fine-tuning.
Semi-Supervised Neural Architecture Search
On ImageNet, it achieves 23. 5% top-1 error rate (under 600M FLOPS constraint) using 4 GPU-days for search.
Ranked #81 on
Neural Architecture Search
on ImageNet
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View.
In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method.
Suphx: Mastering Mahjong with Deep Reinforcement Learning
Artificial Intelligence (AI) has achieved great success in many domains, and game AI is widely regarded as its beachhead since the dawn of AI.
Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
Generative Adversarial Networks (GANs) have shown great promise in modeling high dimensional data.
Ranked #7 on
Image Generation
on STL-10
MPNet: Masked and Permuted Pre-training for Language Understanding
Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem.
Ranked #16 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(using extra training data)
A Study of Non-autoregressive Model for Sequence Generation
In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all?
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
While pre-training and fine-tuning, e. g., BERT~\citep{devlin2018bert}, GPT-2~\citep{radford2019language}, have achieved great success in language understanding and generation tasks, the pre-trained models are usually too big for online deployment in terms of both memory cost and inference speed, which hinders them from practical online usage.
SEEK: Segmented Embedding of Knowledge Graphs
In recent years, knowledge graph embedding becomes a pretty hot research topic of artificial intelligence and plays increasingly vital roles in various downstream applications, such as recommendation and question answering.
Ranked #1 on
Link Prediction
on YAGO37
Invertible Image Rescaling
High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover the original resolutions or the details in the zoom-in images.
Dual Learning: Theoretical Study and an Algorithmic Extension
Dual learning has been successfully applied in many machine learning applications including machine translation, image-to-image transformation, etc.
MultiSpeech: Multi-Speaker Text to Speech with Transformer
Transformer-based text to speech (TTS) model (e. g., Transformer TTS~\cite{li2019neural}, FastSpeech~\cite{ren2019fastspeech}) has shown the advantages of training and inference efficiency over RNN-based model (e. g., Tacotron~\cite{shen2018natural}) due to its parallel computation in training and/or inference.
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e. g., pitch, energy and more accurate duration) as conditional inputs.
Ranked #6 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
MC-BERT: Efficient Language Pre-Training via a Meta Controller
Pre-trained contextual representations (e. g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks.
UWSpeech: Speech to Speech Translation for Unwritten Languages
Existing speech to speech translation systems heavily rely on the text of target language: they usually translate source language either to target text and then synthesize target speech from text, or directly to target speech with target text for auxiliary training.
Multi-branch Attentive Transformer
While the multi-branch architecture is one of the key ingredients to the success of computer vision tasks, it has not been well investigated in natural language processing, especially sequence learning tasks.
Ranked #4 on
Machine Translation
on WMT2014 English-German
(SacreBLEU metric)
Modeling Lost Information in Lossy Image Compression
Specifically, ILC introduces an invertible encoding module to replace the encoder-decoder structure to produce the low dimensional informative latent representation, meanwhile, transform the lost information into an auxiliary latent variable that won't be further coded or stored.
Dynamic of Stochastic Gradient Descent with State-Dependent Noise
Specifically, we show that the covariance of the noise of SGD in the local region of the local minima is a quadratic function of the state.
Rethinking Positional Encoding in Language Pre-training
In this work, we investigate the positional encoding methods used in language pre-training (e. g., BERT) and identify several problems in the existing formulations.
SimulSpeech: End-to-End Simultaneous Speech to Text Translation
In this work, we develop SimulSpeech, an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
DeepSinger: Singing Voice Synthesis with Data Mined From the Web
DeepSinger has several advantages over previous SVS systems: 1) to the best of our knowledge, it is the first SVS system that directly mines training data from music websites, 2) the lyrics-to-singing alignment model further avoids any human efforts for alignment labeling and greatly reduces labeling cost, 3) the singing model based on a feed-forward Transformer is simple and efficient, by removing the complicated acoustic feature modeling in parametric synthesis and leveraging a reference encoder to capture the timbre of a singer from noisy singing data, and 4) it can synthesize singing voices in multiple languages and multiple singers.
Accuracy Prediction with Non-neural Model for Neural Architecture Search
Considering that most architectures are represented as sequences of discrete symbols which are more like tabular data and preferred by non-neural predictors, in this paper, we study an alternative approach which uses non-neural model for accuracy prediction.
Ranked #81 on
Neural Architecture Search
on ImageNet
Temporally Correlated Task Scheduling for Sequence Learning
In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks, which are different in terms of how much input information to use or which future step to predict.
Learning to Match Distributions for Domain Adaptation
However, it remains challenging to determine which method is suitable for a given application since they are built with certain priors or bias.
Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
SAT contains a hyperparameter k, and each k value defines a SAT task with different degrees of parallelism.
How Does Data Augmentation Affect Privacy in Machine Learning?
Even further, we show that the proposed approach can achieve higher MI attack success rates on models trained with some data augmentation than the existing methods on models trained without data augmentation.
Taking Notes on the Fly Helps BERT Pre-training
In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization.
LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
However, there are more than 6, 000 languages in the world and most languages are lack of speech training data, which poses significant challenges when building TTS and ASR systems for extremely low-resource languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
PopMAG: Pop Music Accompaniment Generation
To improve harmony, in this paper, we propose a novel MUlti-track MIDI representation (MuMIDI), which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks.
HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
To tackle the difficulty of singing modeling caused by high sampling rate (wider frequency band and longer waveform), we introduce multi-scale adversarial training in both the acoustic model and vocoder to improve singing modeling.
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets.
Ranked #25 on
Graph Property Prediction
on ogbg-molhiv
Qlib: An AI-oriented Quantitative Investment Platform
Quantitative investment aims to maximize the return and minimize the risk in a sequential trading period over a set of financial instruments.
COSEA: Convolutional Code Search with Layer-wise Attention
However, most existing studies overlook the code's intrinsic structural logic, which indeed contains a wealth of semantic information, and fails to capture intrinsic features of codes.
Learning Causal Semantic Representation for Out-of-Distribution Prediction
Conventional supervised learning methods, especially deep ones, are found to be sensitive to out-of-distribution (OOD) examples, largely because the learned representation mixes the semantic factor with the variation factor due to their domain-specific correlation, while only the semantic factor causes the output.
Latent Causal Invariant Model
To avoid spurious correlation, we propose a Latent Causal Invariance Model (LaCIM) which pursues causal prediction.
RD$^2$: Reward Decomposition with Representation Decomposition
In this work, we propose a set of novel reward decomposition principles by constraining uniqueness and compactness of different state features/representations relevant to different sub-rewards.
The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
Except GD, adaptive algorithms such as AdaGrad, RMSProp and Adam are popular owing to their rapid training process.
Denoising Text to Speech with Frame-Level Noise Modeling
In DenoiSpeech, we handle real-world noisy speech by modeling the fine-grained frame-level noise with a noise condition module, which is jointly trained with the TTS model.
Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations
While adopting complex GNN models with more informative message passing and aggregation mechanisms can obviously benefit heterogeneous vertex representations and cooperative policy learning, it could, on the other hand, increase the training difficulty of MARL and demand more intense and direct reward signals compared to the original global reward.
On the Stability of Multi-branch Network
More importantly, we propose a new design ``STAM aggregation" that can guarantee to STAbilize the forward/backward process of Multi-branch networks irrespective of the number of branches.
Taking Notes on the Fly Helps Language Pre-Training
In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization.
Task-Agnostic and Adaptive-Size BERT Compression
NAS-BERT trains a big supernet on a carefully designed search space containing various architectures and outputs multiple compressed models with adaptive sizes and latency.
Learning to Use Future Information in Simultaneous Translation
For wait-k inference, we observe that wait-m training with $m>k$ in simultaneous NMT (i. e., using more future information for training than inference) generally outperforms wait-k training.
Exploring the Regulatory Function of the N-terminal Domain of SARS-CoV-2 Spike Protein Through Molecular Dynamics Simulation
We performed molecular dynamics simulation on the S protein with a focus on the function of its N-terminal domains (NTDs).
BN-invariant sharpness regularizes the training model to better generalization
However, it has been pointed out that the usual definitions of sharpness, which consider either the maxima or the integral of loss over a $\delta$ ball of parameters around minima, cannot give consistent measurement for scale invariant neural networks, e. g., networks with batch normalization layer.
Universal Trading for Order Execution with Oracle Policy Distillation
As a fundamental problem in algorithmic trading, order execution aims at fulfilling a specific trading order, either liquidation or acquirement, for a given instrument.
LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search
Text to speech (TTS) has been broadly used to synthesize natural and intelligible speech in different scenarios.
REST: Relational Event-driven Stock Trend Forecasting
To remedy the first shortcoming, we propose to model the stock context and learn the effect of event information on the stocks under different contexts.
Revisiting Language Encoding in Learning Multilingual Representations
The language embedding can be either added to the word embedding or attached at the beginning of the sentence.
Return-Based Contrastive Representation Learning for Reinforcement Learning
Recently, various auxiliary tasks have been proposed to accelerate representation learning and improve sample efficiency in deep reinforcement learning (RL).
Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
The privacy leakage of the model about the training data can be bounded in the differential privacy mechanism.
LazyFormer: Self Attention with Lazy Update
In each lazy block, the self-attention distribution is only computed once in the first layer and then is reused in all upper layers.
AdaSpeech: Adaptive Text to Speech for Custom Voice
2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation.
Learning Invariant Representations across Domains and Tasks
Being expensive and time-consuming to collect massive COVID-19 image samples to train deep classification models, transfer learning is a promising approach by transferring knowledge from the abundant typical pneumonia datasets for COVID-19 image classification.
IOT: Instance-wise Layer Reordering for Transformer Structures
Based on this observation, in this work, we break the assumption of the fixed layer order in the Transformer and introduce instance-wise layer reordering into the model structure.
Impact of pandemic fatigue on the spread of COVID-19: a mathematical modelling study
We used a stochastic metapopulation model with a hierarchical structure and fitted the model to the positive cases in the US from the start of outbreak to the end of 2020.
UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models.
Ranked #4 on
Machine Translation
on IWSLT2014 English-German
AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data
In adaptation, we use untranscribed speech data for speech reconstruction and only fine-tune the TTS decoder.
FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition
A straightforward solution to reduce latency, inspired by non-autoregressive (NAR) neural machine translation, is to use an NAR sequence generation model for ASR error correction, which, however, comes at the cost of significantly increased ASR error rate.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
How could Neural Networks understand Programs?
Inspired by this, we propose a novel program semantics learning paradigm, that the model should learn from information composed of (1) the representations which align well with the fundamental operations in operational semantics, and (2) the information of environment transition, which is indispensable for program understanding.
Optimizing Information-theoretical Generalization Bound via Anisotropic Noise of SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Learning Structures for Deep Neural Networks
We further establish connections between this principle and the theory of Bayesian optimal classification, and empirically verify that larger entropy of the outputs of a deep neural network indeed corresponds to a better classification accuracy.
NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
The technical challenge of NAS-BERT is that training a big supernet on the pre-training task is extremely costly.
Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart
Along with this routine, we find that confidence and a rectified confidence (R-Con) can form two coupled rejection metrics, which could provably distinguish wrongly classified inputs from correctly classified ones.
Machine-Learning Non-Conservative Dynamics for New-Physics Detection
Energy conservation is a basic physics principle, the breakdown of which often implies new physics.
Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics
In this paper, to reduce the reliance on the numerical solver, we propose to enhance the supervised signal in the training of NODE.
Do Transformers Really Perform Bad for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Ranked #1 on
Graph Regression
on PCQM4M-LSC
MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
Inspired by the success of pre-training models in natural language processing, in this paper, we develop MusicBERT, a large-scale pre-trained model for music understanding.
PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior
Denoising diffusion probabilistic models have been recently proposed to generate high-quality samples by estimating the gradient of the data density.
Dual-view Molecule Pre-training
After pre-training, we can use either the Transformer branch (this one is recommended according to empirical results), the GNN branch, or both for downstream tasks.
Ranked #1 on
Molecular Property Prediction
on HIV dataset
Large Scale Private Learning via Low-rank Reparametrization
We propose a reparametrization scheme to address the challenges of applying differentially private SGD on large neural networks, which are 1) the huge memory cost of storing individual gradients, 2) the added noise suffering notorious dimensional dependence.
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Since in many state-of-the-art models, relative positional encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.
R-Drop: Regularized Dropout for Neural Networks
Dropout is a powerful and widely used technique to regularize the training of deep neural networks.
Ranked #4 on
Machine Translation
on WMT2014 English-French
A Survey on Neural Speech Synthesis
Text to speech (TTS), or speech synthesis, which aims to synthesize intelligible and natural speech given text, is a hot research topic in speech, language, and machine learning communities and has broad applications in the industry.
Regularized OFU: an Efficient UCB Estimator forNon-linear Contextual Bandit
However, it is in general unknown how to deriveefficient and effective EE trade-off methods for non-linearcomplex tasks, suchas contextual bandit with deep neural network as the reward function.
On the Generative Utility of Cyclic Conditionals
This is motivated by the observation that deep generative models, in addition to a likelihood model $p(x|z)$, often also use an inference model $q(z|x)$ for extracting representation, but they rely on a usually uninformative prior distribution $p(z)$ to define a joint distribution, which may render problems like posterior collapse and manifold mismatch.
Supervised Off-Policy Ranking
Inspired by the two observations, in this work, we study a new problem, supervised off-policy ranking (SOPR), which aims to rank a set of target policies based on supervised learning by leveraging off-policy data and policies with known performance.
DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling
In this paper, we develop DeepRapper, a Transformer-based rap generation system that can model both rhymes and rhythms.
AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style
While recent text to speech (TTS) models perform very well in synthesizing reading-style (e. g., audiobook) speech, it is still challenging to synthesize spontaneous-style speech (e. g., podcast or conversation), mainly because of two reasons: 1) the lack of training data for spontaneous speech; 2) the difficulty in modeling the filled pauses (um and uh) and diverse rhythms in spontaneous speech.
A Survey on Low-Resource Neural Machine Translation
Neural approaches have achieved state-of-the-art accuracy on machine translation but suffer from the high cost of collecting large scale parallel data.
