Search Results for author:
Found 98 papers, 18 papers with code
Deep Reinforcement Learning with Smooth Policy
In contrast to policy parameterized by linear/reproducing kernel functions, where simple regularization techniques suffice to control smoothness, for neural network based reinforcement learning algorithms, there is no readily available solution to learn a smooth policy.
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
Large Transformer-based models have exhibited superior performance in various natural language processing and computer vision tasks.
Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint
Overparameterized neural networks enjoy great representation power on complex data, and more importantly yield sufficiently smooth output, which is crucial to their generalization and robustness.
Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networks
We consider the off-policy evaluation problem of reinforcement learning using deep neural networks.
A Manifold Two-Sample Test Study: Integral Probability Metric with Neural Networks
Based on the approximation theory of neural networks, we show that the neural network IPM test has the type-II risk in the order of $n^{-(s+\beta)/d}$, which is in the same order of the type-II risk as the H\"older IPM test.
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
We propose MoEBERT, which uses a Mixture-of-Experts structure to increase model capacity and inference speed.
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing
To retain ensemble benefits while maintaining a low memory cost, we propose a consistency-regularized ensemble learning approach based on perturbed models, named CAMERO.
CERES: Pretraining of Graph-Conditioned Transformer for Semi-Structured Session Data
User sessions empower many search and recommendation tasks on a daily basis.
Noise Regularizes Over-parameterized Rank One Matrix Recovery, Provably
We investigate the role of noise in optimization algorithms for learning over-parameterized models.
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity
The superlinear convergence takes effect after no more than $\mathcal{O}(\log(1/\Delta^*))$ number of iterations, where $\Delta^*$ is defined via a gap quantity associated with the optimal state-action value function; (2) HPMD also exhibits last-iterate convergence of the policy, with the limiting policy corresponding exactly to the optimal policy with the maximal entropy for every state.
Block Policy Mirror Descent
Compared to the traditional PG methods with a batch update rule, which visits and updates the policy for every state, BPMD methods have cheap per-iteration computation via a partial update rule that performs the policy update on a sampled state.
Deep Learning Assisted End-to-End Synthesis of mm-Wave Passive Networks with 3D EM Structures: A Study on A Transformer-Based Matching Network
This paper presents a deep learning assisted synthesis approach for direct end-to-end generation of RF/mm-wave passive matching network with 3D EM structures.
Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces
Learning operators between infinitely dimensional spaces is an important learning task arising in wide applications in machine learning, imaging science, mathematical modeling and simulations, etc.
Pessimism Meets Invariance: Provably Efficient Offline Mean-Field Multi-Agent RL
Theoretically, under a weak coverage assumption that the experience dataset contains enough information about the optimal policy, we prove that for an episodic mean-field MDP with a horizon $H$ and $N$ training trajectories, SAFARI attains a sub-optimality gap of $\mathcal{O}(H^2d_{\rm eff} /\sqrt{N})$, where $d_{\rm eff}$ is the effective dimension of the function class for parameterizing the value function, but independent on the number of agents.
Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits
We show that incorporating frequency information of tokens in the embedding learning problems leads to provably efficient algorithms, and demonstrate that common adaptive algorithms implicitly exploit the frequency information to a large extent.
Taming Sparsely Activated Transformer with Stochastic Experts
While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i. e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts.
Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
Moreover, we rigorously establish an implicit bias of GD induced by such a large learning rate, termed 'balancing', meaning that magnitudes of $X$ and $Y$ at the limit of GD iterations will be close even if their initialization is significantly unbalanced.
A Principled Permutation Invariant Approach to Mean-Field Multi-Agent Reinforcement Learning
To exploit the permutation invariance therein, we propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation- invariant actor-critic neural architecture.
Adversarially Regularized Policy Learning Guided by Trajectory Optimization
Recent advancement in combining trajectory optimization with function approximation (especially neural networks) shows promise in learning complex control policies for diverse tasks in robot systems.
Self-Training with Differentiable Teacher
In self-training, the student contributes to the prediction performance, and the teacher controls the training process by generating pseudo-labels.
ARCH: Efficient Adversarial Regularized Training with Caching
Adversarial regularization can improve model generalization in many natural language processing tasks.
Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks
Most of existing statistical theories on deep neural networks have sample complexities cursed by the data dimension and therefore cannot well explain the empirical success of deep learning on high-dimensional data.
QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction
We study the problem of query attribute value extraction, which aims to identify named entities from user queries as diverse surface form attribute values and afterward transform them into formally canonical forms.
Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data
We show that BPPA attains non-trivial margin, which closely depends on the condition number of the distance generating function inducing the Bregman divergence.
Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data
Unfortunately, we observe that weakly labeled data does not necessarily improve, or even deteriorate the model performance (due to the extensive noise in the weak labels) when we train deep NER models over a simple or weighted combination of the strongly labeled and weakly labeled data.
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i. e., a subnetwork) can match the performance of the full model.
Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach
To exploit the permutation invariance therein, we propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation-invariant actor-critic neural architecture.
COUnty aggRegation mixup AuGmEntation (COURAGE) COVID-19 Prediction
The global spread of COVID-19, the disease caused by the novel coronavirus SARS-CoV-2, has cast a significant threat to mankind.
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks.
Token-wise Curriculum Learning for Neural Machine Translation
Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of "easy" samples from training data at the early training stage.
Reinforcement Learning for Adaptive Mesh Refinement
Large-scale finite element simulations of complex physical systems governed by partial differential equations (PDE) crucially depend on adaptive mesh refinement (AMR) to allocate computational budget to regions where higher resolution is required.
Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
Numerous empirical evidences have corroborated the importance of noise in nonconvex optimization problems.
Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
An ideal environment for evaluating dialog systems, also known as the Turing test, needs to involve human interaction, which is usually not affordable for large-scale experiments.
Model-based Reinforcement Learning
reinforcement-learning
+1
Differentiable Top-k with Optimal Transport
Finding the k largest or smallest elements from a collection of scores, i. e., top-k operation, is an important model component widely used in information retrieval, machine learning, and data mining.
Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? --- A Neural Tangent Kernel Perspective
We then compare the kernel of deep ResNets with that of deep FFNets and discover that the class of functions induced by the kernel of FFNets is asymptotically not learnable, as the depth goes to infinity.
A Hypergradient Approach to Robust Regression without Correspondence
Due to the combinatorial nature of the problem, most existing methods are only applicable when the sample size is small, and limited to linear regression models.
Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networks
Our theory shows that deep neural networks are adaptive to the low-dimensional geometric structures of the covariates, and partially explains the success of deep learning for causal inference.
Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
Fine-tuned pre-trained language models can suffer from severe miscalibration for both in-distribution and out-of-distribution (OOD) data due to over-parameterization.
Differentiable Top-$k$ with Optimal Transport
The top-$k$ operation, i. e., finding the $k$ largest or smallest elements from a collection of scores, is an important model component, which is widely used in information retrieval, machine learning, and data mining.
Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach
To address this problem, we develop a contrastive self-training framework, COSINE, to enable fine-tuning LMs with weak supervision.
Ranked #1 on
Word Sense Disambiguation
on Words in Context
How Important is the Train-Validation Split in Meta-Learning?
A common practice in meta-learning is to perform a train-validation split (\emph{train-val method}) where the prior adapts to the task on one split of the data, and the resulting predictor is evaluated on another split.
Residual Network Based Direct Synthesis of EM Structures: A Study on One-to-One Transformers
We propose using machine learning models for the direct synthesis of on-chip electromagnetic (EM) passive structures to enable rapid or even automated designs and optimizations of RF/mm-Wave circuits.
BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision
We study the open-domain named entity recognition (NER) problem under distant supervision.
Picasso: A Sparse Learning Library for High Dimensional Data Analysis in R and Python
We describe a new library named picasso, which implements a unified framework of pathwise coordinate optimization for a variety of sparse learning problems (e. g., sparse linear regression, sparse logistic regression, sparse Poisson regression and scaled sparse linear regression) combined with efficient active set selection strategies.
The flare Package for High Dimensional Linear Regression and Precision Matrix Estimation in R
This paper describes an R package named flare, which implements a family of new high dimensional regression methods (LAD Lasso, SQRT Lasso, $\ell_q$ Lasso, and Dantzig selector) and their extensions to sparse precision matrix estimation (TIGER and CLIME).
The huge Package for High-dimensional Undirected Graph Estimation in R
We describe an R package named huge which provides easy-to-use functions for estimating high dimensional undirected graphs from data.
Towards Understanding Hierarchical Learning: Benefits of Neural Representations
When the trainable network is the quadratic Taylor model of a wide two-layer network, we show that neural representation can achieve improved sample complexities compared with the raw input: For learning a low-rank degree-$p$ polynomial ($p \geq 4$) in $d$ dimension, neural representation requires only $\tilde{O}(d^{\lceil p/2 \rceil})$ samples, while the best-known sample complexity upper bound for the raw input is $\tilde{O}(d^{p-1})$.
Implicit Bias of Gradient Descent based Adversarial Training on Separable Data
Specifically, we show that for any fixed iteration $T$, when the adversarial perturbation during training has proper bounded L2 norm, the classifier learned by gradient descent based adversarial training converges in direction to the maximum L2 norm margin classifier at the rate of $O(1/\sqrt{T})$, significantly faster than the rate $O(1/\log T}$ of training with clean data.
Deep Reinforcement Learning with Robust and Smooth Policy
Deep reinforcement learning (RL) has achieved great empirical successes in various domains.
Transformer Hawkes Process
Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets.
Differentiable Top-k Operator with Optimal Transport
The top-k operation, i. e., finding the k largest or smallest elements from a collection of scores, is an important model component, which is widely used in information retrieval, machine learning, and data mining.
Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
We then compare the kernel of deep ResNets with that of deep FFNets and discover that the class of functions induced by the kernel of FFNets is asymptotically not learnable, as the depth goes to infinity.
Statistical Guarantees of Generative Adversarial Networks for Distribution Estimation
This paper provides statistical guarantees of GANs for the estimation of data distributions which have densities in a H\"{o}lder space.
On Computation and Generalization of Generative Adversarial Imitation Learning
Generative Adversarial Imitation Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies.
Efficient Approximation of Deep ReLU Networks for Functions on Low Dimensional Manifolds
The network size scales exponentially in the approximation error, with an exponent depending on the intrinsic dimension of the data and the smoothness of the function.
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model.
Ranked #1 on
Semantic Textual Similarity
on MRPC
Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing
To overcome this limitation, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing.
On Generalization Bounds of a Family of Recurrent Neural Networks
We remark: (1) Our generalization bound for vanilla RNNs is significantly tighter than the best of existing results; (2) We are not aware of any other generalization bounds for MGU, LSTM, and Conv RNNs in the exiting literature; (3) We demonstrate the advantages of these variants in generalization.
Towards Understanding the Importance of Shortcut Connections in Residual Networks
We show, however, that gradient descent combined with proper normalization, avoids being trapped by the spurious local optimum, and converges to a global optimum in polynomial time, when the weight of the first layer is initialized at 0, and that of the second layer is initialized arbitrarily in a ball.
Towards Understanding the Importance of Noise in Training Neural Networks
Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks.
Meta Learning with Relational Information for Short Sequences
This paper proposes a new meta-learning method -- named HARMLESS (HAwkes Relational Meta LEarning method for Short Sequences) for learning heterogeneous point process models from short event sequence data along with a relational network.
Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recovery
It therefore demonstrates the adaptivity of deep ReLU networks to low-dimensional geometric structures of data, and partially explains the power of deep ReLU networks in tackling high-dimensional data with low-dimensional geometric structures.
Inductive Bias of Gradient Descent based Adversarial Training on Separable Data
Specifically, we show that when the adversarial perturbation during training has bounded $\ell_2$-norm, the classifier learned by gradient descent based adversarial training converges in direction to the maximum $\ell_2$-norm margin classifier at the rate of $\tilde{\mathcal{O}}(1/\sqrt{T})$, significantly faster than the rate $\mathcal{O}(1/\log T)$ of training with clean data.
On Computation and Generalization of Generative Adversarial Networks under Spectrum Control
Generative Adversarial Networks (GANs), though powerful, is hard to train.
On Scalable and Efficient Computation of Large Scale Optimal Transport
Optimal Transport (OT) naturally arises in many machine learning applications, yet the heavy computational burden limits its wide-spread uses.
On Tighter Generalization Bounds for Deep Neural Networks: CNNs, ResNets, and Beyond
We propose a generalization error bound for a general family of deep neural networks based on the depth and width of the networks, as well as the spectral norm of weight matrices.
Learning to Defense by Learning to Attack
From the perspective of generative learning, our proposed method can be viewed as learning a deep generative model for generating adversarial samples, which is adaptive to the robust classification.
On Computation and Generalization of GANs with Spectrum Control
Specifically, we propose a new reparameterization approach for the weight matrices of the discriminator in GANs, which allows us to directly manipulate the spectra of the weight matrices through various regularizers and constraints, without intensively computing singular value decompositions.
Learning to Defend by Learning to Attack
Adversarial training provides a principled approach for training robust neural networks.
Provable Gaussian Embedding with One Observation
In this paper, we study the Gaussian embedding model and develop the first theoretical results for exponential family embedding models.
On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond
We establish a margin based data dependent generalization error bound for a general family of deep neural networks in terms of the depth and width, as well as the Jacobian of the networks.
On Landscape of Lagrangian Functions and Stochastic Search for Constrained Nonconvex Optimization
However, due to the lack of convexity, their landscape is not well understood and how to find the stable equilibria of the Lagrangian function is still unknown.
Towards Understanding Acceleration Tradeoff between Momentum and Asynchrony in Nonconvex Stochastic Optimization
Asynchronous momentum stochastic gradient descent algorithms (Async-MSGD) is one of the most popular algorithms in distributed machine learning.
Detecting Nonlinear Causality in Multivariate Time Series with Sparse Additive Models
We propose a nonparametric method for detecting nonlinear causal relationship within a set of multidimensional discrete time series, by using sparse additive models (SpAMs).
Dimensionality Reduction for Stationary Time Series via Stochastic Nonconvex Optimization
Specifically, our goal is to estimate the principle component of time series data with respect to the covariance matrix of the stationary distribution.
A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization
Our theoretical discovery partially corroborates the empirical success of MSGD in training deep neural networks.
Misspecified Nonconvex Statistical Optimization for Phase Retrieval
Existing nonconvex statistical optimization theory and methods crucially rely on the correct specification of the underlying "true" statistical models.
Parametric Simplex Method for Sparse Learning
High dimensional sparse learning has imposed a great computational challenge to large scale data analysis.
On Quadratic Convergence of DC Proximal Newton Algorithm in Nonconvex Sparse Learning
We propose a DC proximal Newton algorithm for solving nonconvex regularized sparse learning problems in high dimensions.
Deep Hyperspherical Learning
In light of such challenges, we propose hyperspherical convolution (SphereConv), a novel learning framework that gives angular representations on hyperspheres.
Online Partial Least Square Optimization: Dropping Convexity for Better Efficiency and Scalability
Multiview representation learning is popular for latent factor analysis.
On Quadratic Convergence of DC Proximal Newton Algorithm for Nonconvex Sparse Learning in High Dimensions
We propose a DC proximal Newton algorithm for solving nonconvex regularized sparse learning problems in high dimensions.
Online Factorization and Partition of Complex Networks From Random Walks
We formulate this into a nonconvex stochastic factorization problem and propose an efficient and scalable stochastic generalized Hebbian algorithm.
Homotopy Parametric Simplex Method for Sparse Learning
High dimensional sparse learning has imposed a great computational challenge to large scale data analysis.
Dropping Convexity for More Efficient and Scalable Online Multiview Learning
Multiview representation learning is very popular for latent factor analysis.
Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization
We propose a general theory for studying the \xl{landscape} of nonconvex \xl{optimization} with underlying symmetric structures \tz{for a class of machine learning problems (e. g., low-rank matrix factorization, phase retrieval, and deep linear neural networks)}.
The Physical Systems Behind Optimization Algorithms
We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning.
On Faster Convergence of Cyclic Block Coordinate Descent-type Methods for Strongly Convex Minimization
In particular, we first show that for a family of quadratic minimization problems, the iteration complexity $\mathcal{O}(\log^2(p)\cdot\log(1/\epsilon))$ of the CBCD-type methods matches that of the GD methods in term of dependency on $p$, up to a $\log^2 p$ factor.
On Fast Convergence of Proximal Algorithms for SQRT-Lasso Optimization: Don't Worry About Its Nonsmooth Loss Function
Many machine learning techniques sacrifice convenient computational structures to gain estimation robustness and modeling flexibility.
NESTT: A Nonconvex Primal-Dual Splitting Method for Distributed and Stochastic Optimization
We study a stochastic and distributed algorithm for nonconvex problems whose objective consists of a sum of $N$ nonconvex $L_i/N$-smooth functions, plus a nonsmooth regularizer.
Nonconvex Sparse Learning via Stochastic Optimization with Progressive Variance Reduction
We propose a stochastic variance reduced optimization algorithm for solving sparse learning problems with cardinality constraints.
A Nonconvex Optimization Framework for Low Rank Matrix Estimation
We study the estimation of low rank matrices via nonconvex optimization.
Pathwise Coordinate Optimization for Sparse Learning: Algorithm and Theory
This is the first result on the computational and statistical guarantees of the pathwise coordinate optimization framework in high dimensions.
Accelerated Mini-batch Randomized Block Coordinate Descent Method
When the regularization function is block separable, we can solve the minimization problems in a randomized block coordinate descent (RBCD) manner.
Multivariate Regression with Calibration
We propose a new method named calibrated multivariate regression (CMR) for fitting high dimensional multivariate regression models.
Sparse Inverse Covariance Estimation with Calibration
We propose a semiparametric procedure for estimating high dimensional sparse inverse covariance matrix.
Calibrated Multivariate Regression with Application to Neural Semantic Basis Discovery
We propose a calibrated multivariate regression method named CMR for fitting high dimensional multivariate regression models.
