Search Results for author:
Found 41 papers, 1 papers with code
The Implicit Bias of Depth: How Incremental Learning Drives Generalization
A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity.
Learning Rules-First Classifiers
Complex classifiers may exhibit "embarassing" failures in cases where humans can easily provide a justified classification.
Planning and Learning with Stochastic Action Sets
From an RL perspective, we show that Q-learning with sampled action sets is sound.
Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning.
SGD Learns the Conjugate Kernel Class of the Network
We show that the standard stochastic gradient decent (SGD) algorithm is guaranteed to learn, in polynomial time, a function that is competitive with the best function in the conjugate kernel space of the network, as defined in Daniely, Frostig and Singer.
Random Features for Compositional Kernels
We describe and analyze a simple random feature scheme (RFS) from prescribed compositional kernels.
Depth Separation for Neural Networks
As many functions of the above form can be well approximated by poly-size depth three networks with poly-bounded weights, this establishes a separation between depth two and depth three networks w. r. t.\ the uniform distribution on $\mathbb{S}^{d-1}\times \mathbb{S}^{d-1}$.
Behavior-Based Machine-Learning: A Hybrid Approach for Predicting Human Decision Making
A large body of work in behavioral fields attempts to develop models that describe the way people, as opposed to rational agents, make decisions.
Sketching and Neural Networks
In stark contrast, our approach of using improper learning, using a larger hypothesis class allows the sketch size to have a logarithmic dependence on the degree.
Complexity Theoretic Limitations on Learning Halfspaces
We show that no efficient learning algorithm has non-trivial worst-case performance even under the guarantees that $\mathrm{Err}_{\mathrm{HALF}}(\mathcal{D}) \le \eta$ for arbitrarily small constant $\eta>0$, and that $\mathcal{D}$ is supported in $\{\pm 1\}^n\times \{\pm 1\}$.
Distribution Free Learning with Local Queries
The model of learning with \emph{local membership queries} interpolates between the PAC model and the membership queries model by allowing the learner to query the label of any example that is similar to an example in the training set.
A PTAS for Agnostically Learning Halfspaces
We present a PTAS for agnostically learning halfspaces w. r. t.
Strongly Adaptive Online Learning
Strongly adaptive algorithms are algorithms whose performance on every time interval is close to optimal.
Multiclass learnability and the ERM principle
We study the sample complexity of multiclass prediction in several learning settings.
Complexity theoretic limitations on learning DNF's
Using the recently developed framework of [Daniely et al, 2014], we show that under a natural assumption on the complexity of refuting random K-SAT formulas, learning DNF formulas is hard.
Learning Economic Parameters from Revealed Preferences
In this work we advance this line of work by providing sample complexity guarantees and efficient algorithms for a number of important classes.
Optimal Learners for Multiclass Problems
Furthermore, we show that the sample complexity of these learners is better than the sample complexity of the ERM rule, thus settling in negative an open question due to Collins (2005).
The complexity of learning halfspaces using generalized linear methods
The best approximation ratio achievable by an efficient algorithm is $O\left(\frac{1/\gamma}{\sqrt{\log(1/\gamma)}}\right)$ and is achieved using an algorithm from the above class.
From average case complexity to improper learning complexity
The biggest challenge in proving complexity results is to establish hardness of {\em improper learning} (a. k. a.
More data speeds up training time in learning halfspaces over sparse vectors
That is, if more data is available, beyond the sample complexity limit, is it possible to use the extra examples to speed up the computation time required to perform the learning task?
The price of bandit information in multiclass online classification
We consider two scenarios of multiclass online learning of a hypothesis class $H\subseteq Y^X$.
Locally Private Learning without Interaction Requires Separation
The only lower bound we are aware of is for PAC learning an artificial class of functions with respect to a uniform distribution (Kasiviswanathan et al. 2011).
Multiclass Learning Approaches: A Theoretical Comparison with Implications
We analyze both the estimation error and the approximation error of these methods.
Competitive ratio versus regret minimization: achieving the best of both worlds
Our end result is an online algorithm that can combine a "base" online algorithm, having a guaranteed competitive ratio, with a range of online algorithms that guarantee a small regret over any interval of time.
ID3 Learns Juntas for Smoothed Product Distributions
In recent years, there are many attempts to understand popular heuristics.
On the Optimality of Trees Generated by ID3
Since its inception in the 1980s, ID3 has become one of the most successful and widely used algorithms for learning decision trees.
Generalization Bounds for Neural Networks via Approximate Description Length
We show that for any depth $t$, if the inputs are in $[-1, 1]^d$, the sample complexity of $H$ is $\tilde O\left(\frac{dR^2}{\epsilon^2}\right)$.
Neural Networks Learning and Memorization with (almost) no Over-Parameterization
Many results in recent years established polynomial time learnability of various models via neural networks algorithms.
On the Complexity of Minimizing Convex Finite Sums Without Using the Indices of the Individual Functions
Recent advances in randomized incremental methods for minimizing $L$-smooth $\mu$-strongly convex finite sums have culminated in tight complexity of $\tilde{O}((n+\sqrt{n L/\mu})\log(1/\epsilon))$ and $O(n+\sqrt{nL/\epsilon})$, where $\mu>0$ and $\mu=0$, respectively, and $n$ denotes the number of individual functions.
Learning Parities with Neural Networks
On the other hand, under the same distributions, these parities cannot be learned efficiently by linear methods.
Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networks
We prove that a single step of gradient decent over depth two network, with $q$ hidden neurons, starting from orthogonal initialization, can memorize $\Omega\left(\frac{dq}{\log^4(d)}\right)$ independent and randomly labeled Gaussians in $\mathbb{R}^d$.
Hardness of Learning Neural Networks with Natural Weights
A natural approach to settle the discrepancy is to assume that the network's weights are "well-behaved" and posses some generic properties that may allow efficient learning.
Most ReLU Networks Suffer from $\ell^2$ Adversarial Perturbations
We consider ReLU networks with random weights, in which the dimension decreases at each layer.
From Local Pseudorandom Generators to Hardness of Learning
We also establish lower bounds on the complexity of learning intersections of a constant number of halfspaces, and ReLU networks with a constant number of hidden neurons.
An Exact Poly-Time Membership-Queries Algorithm for Extraction a three-Layer ReLU Network
In this work, we present a polynomial-time algorithm that can learn a depth-two ReLU network from queries under mild general position assumptions.
Asynchronous Stochastic Optimization Robust to Arbitrary Delays
In the general non-convex smooth optimization setting, we give a simple and efficient algorithm that requires $O( \sigma^2/\epsilon^4 + \tau/\epsilon^2 )$ steps for finding an $\epsilon$-stationary point $x$, where $\tau$ is the \emph{average} delay $\smash{\frac{1}{T}\sum_{t=1}^T d_t}$ and $\sigma^2$ is the variance of the stochastic gradients.
Monotone Learning
Our transformation readily implies monotone learners in a variety of contexts: for example it extends Pestov's result to classification tasks with an arbitrary number of labels.
Approximate Description Length, Covering Numbers, and VC Dimension
Recently, Daniely and Granot [arXiv:1910. 05697] introduced a new notion of complexity called Approximate Description Length (ADL).
On the Sample Complexity of Two-Layer Networks: Lipschitz vs. Element-Wise Lipschitz Activation
We investigate the sample complexity of bounded two-layer neural networks using different activation functions.
Locally Optimal Descent for Dynamic Stepsize Scheduling
We introduce a novel dynamic learning-rate scheduling scheme grounded in theory with the goal of simplifying the manual and time-consuming tuning of schedules in practice.
RedEx: Beyond Fixed Representation Methods via Convex Optimization
Optimizing Neural networks is a difficult task which is still not well understood.
