Search Results for author:
Found 35 papers, 4 papers with code
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation
In this work, we propose a novel approach for layerwise representation learning of a trained neural network.
Learning from Randomly Initialized Neural Network Features
We present the surprising result that randomly initialized neural networks are good feature extractors in expectation.
Step-size Adaptation Using Exponentiated Gradient Updates
In this paper, we update the step-size scale and the gain variables with exponentiated gradient updates instead.
LocoProp: Enhancing BackProp via Local Loss Optimization
Second-order methods have shown state-of-the-art performance for optimizing deep neural networks.
Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
Many learning tasks in machine learning can be viewed as taking a gradient step towards minimizing the average loss of a batch of examples in each training iteration.
Rank-smoothed Pairwise Learning In Perceptual Quality Assessment
Training a model on these pairwise preferences is a common deep learning approach.
A case where a spindly two-layer linear network whips any neural network with a fully connected input layer
It was conjectured that any neural network of any structure and arbitrary differentiable transfer functions at the nodes cannot learn the following problem sample efficiently when trained with gradient descent: The instances are the rows of a $d$-dimensional Hadamard matrix and the target is one of the features, i. e. very sparse.
Reparameterizing Mirror Descent as Gradient Descent
We present a general framework for casting a mirror descent update as a gradient descent update on a different set of parameters.
TriMap: Large-scale Dimensionality Reduction Using Triplets
We empirically show the excellent performance of TriMap on a large variety of datasets in terms of the quality of the embedding as well as the runtime.
An Implicit Form of Krasulina's k-PCA Update without the Orthonormality Constraint
We show that Krasulina's update corresponds to a projected gradient descent step on the Stiefel manifold of the orthonormal $k$-frames, while Oja's update amounts to a gradient descent step using the unprojected gradient.
Unbiased estimators for random design regression
We use them to show that for any input distribution and $\epsilon>0$ there is a random design consisting of $O(d\log d+ d/\epsilon)$ points from which an unbiased estimator can be constructed whose expected square loss over the entire distribution is bounded by $1+\epsilon$ times the loss of the optimum.
Robust Bi-Tempered Logistic Loss Based on Bregman Divergences
We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization.
Adaptive scale-invariant online algorithms for learning linear models
We consider online learning with linear models, where the algorithm predicts on sequentially revealed instances (feature vectors), and is compared against the best linear function (comparator) in hindsight.
Divergence-Based Motivation for Online EM and Combining Hidden Variable Models
Expectation-Maximization (EM) is a prominent approach for parameter estimation of hidden (aka latent) variable models.
Minimax experimental design: Bridging the gap between statistical and worst-case approaches to least squares regression
In the process, we develop a new algorithm for a joint sampling distribution called volume sampling, and we propose a new i. i. d.
Unlabeled sample compression schemes and corner peelings for ample and maximum classes
On the positive side we present a new construction of an unlabeled sample compression scheme for maximum classes.
Correcting the bias in least squares regression with volume-rescaled sampling
Without any assumptions on the noise, the linear least squares solution for any i. i. d.
Reverse iterative volume sampling for linear regression
We can only afford to attain the responses for a small subset of the points that are then used to construct linear predictions for all points in the dataset.
Online Non-Additive Path Learning under Full and Partial Information
We study the problem of online path learning with non-additive gains, which is a central problem appearing in several applications, including ensemble structured prediction.
Speech Recognition: Keyword Spotting Through Image Recognition
The problem of identifying voice commands has always been a challenge due to the presence of noise and variability in speed, pitch, etc.
A more globally accurate dimensionality reduction method using triplets
We first show that the commonly used dimensionality reduction (DR) methods such as t-SNE and LargeVis poorly capture the global structure of the data in the low dimensional embedding.
Leveraged volume sampling for linear regression
We then develop a new rescaled variant of volume sampling that produces an unbiased estimate which avoids this bad behavior and has at least as good a tail bound as leverage score sampling: sample size $k=O(d\log d + d/\epsilon)$ suffices to guarantee total loss at most $1+\epsilon$ times the minimum with high probability.
Subsampling for Ridge Regression via Regularized Volume Sampling
However, when labels are expensive, we are forced to select only a small subset of vectors $\mathbf{x}_i$ for which we obtain the labels $y_i$.
Online Dynamic Programming
We consider the problem of repeatedly solving a variant of the same dynamic programming problem in successive trials.
Unbiased estimates for linear regression via volume sampling
Pseudo inverse plays an important part in solving the linear least squares problem, where we try to predict a label for each column of $X$.
Two-temperature logistic regression based on the Tsallis divergence
We explain this by showing that $t_1 < 1$ caps the surrogate loss and $t_2 >1$ makes the predictive distribution have a heavy tail.
Low-dimensional Data Embedding via Robust Ranking
We describe a new method called t-ETE for finding a low-dimensional embedding of a set of objects in Euclidean space.
PCA with Gaussian perturbations
We develop a simple algorithm that needs $O(kn^2)$ per trial whose regret is off by a small factor of $O(n^{1/4})$.
Labeled compression schemes for extremal classes
We consider a generalization of maximum classes called extremal classes.
The limits of squared Euclidean distance regularization
We conjecture that our hardness results hold for any training algorithm that is based on the squared Euclidean distance regularization (i. e. Back-propagation with the Weight Decay heuristic).
A Bayesian Probability Calculus for Density Matrices
Finite probability distributions are a special case where the density matrix is restricted to be diagonal.
On-line PCA with Optimal Regrets
Furthermore, we show that when considering regret bounds as function of a loss budget, EG remains optimal and strictly outperforms GD.
Putting Bayes to sleep
If the nature of the data is changing over time in that different models predict well on different segments of the data, then adaptivity is typically achieved by mixing into the weights in each round a bit of the initial prior (kind of like a weak restart).
Learning Eigenvectors for Free
In this extension, the alphabet of $n$ outcomes is replaced by the set of all dyads, i. e. outer products $\u\u^\top$ where $\u$ is a vector in $\R^n$ of unit length.
Repeated Games against Budgeted Adversaries
We study repeated zero-sum games against an adversary on a budget.
