Search Results for author:
Found 57 papers, 9 papers with code
Provable Guarantees on Learning Hierarchical Generative Models with Deep CNNs
To show any positive theoretical results, one must make assumptions on the data distribution.
Jamba: A Hybrid Transformer-Mamba Language Model
We present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture.
Managing AI Risks in an Era of Rapid Progress
In this short consensus paper, we outline risks from upcoming, advanced AI systems.
Less is More: Selective Layer Finetuning with SubTuning
Finetuning a pretrained model has become a standard approach for training neural networks on novel tasks, resulting in fast convergence and improved performance.
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks.
Standing on the Shoulders of Giant Frozen Language Models
To demonstrate this, we introduce three novel methods for leveraging frozen models: input-dependent prompt tuning, frozen readers, and recursive LMs, each of which vastly improves on current frozen-model approaches.
Knowledge Distillation: Bad Models Can Be Good Role Models
We relate the notion of such samplers to knowledge distillation, where a student network imitates the outputs of a teacher on unlabeled data.
Provable Learning of Convolutional Neural Networks with Data Driven Features
Convolutional networks (CNN) are computationally hard to learn.
The Connection Between Approximation, Depth Separation and Learnability in Neural Networks
On the other hand, the fact that deep networks can efficiently express a target function does not mean that this target function can be learned efficiently by deep neural networks.
The Implications of Local Correlation on Learning Some Deep Functions
In fact, the proofs of such hardness results show that even weakly learning deep networks is hard.
Computational Separation Between Convolutional and Fully-Connected Networks
Convolutional neural networks (CNN) exhibit unmatched performance in a multitude of computer vision tasks.
When Hardness of Approximation Meets Hardness of Learning
A supervised learning algorithm has access to a distribution of labeled examples, and needs to return a function (hypothesis) that correctly labels the examples.
On the Ethics of Building AI in a Responsible Manner
The AI-alignment problem arises when there is a discrepancy between the goals that a human designer specifies to an AI learner and a potential catastrophic outcome that does not reflect what the human designer really wants.
Proving the Lottery Ticket Hypothesis: Pruning is All You Need
The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network.
Learning Boolean Circuits with Neural Networks
To separate hard from easy to learn distributions, we observe the property of local correlation: correlation between local patterns of the input and the target label.
The Implicit Bias of Depth: How Incremental Learning Drives Generalization
A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity.
SenseBERT: Driving Some Sense into BERT
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding.
Ranked #11 on
Word Sense Disambiguation
on Words in Context
Discriminative Active Learning
We propose a new batch mode active learning algorithm designed for neural networks and large query batch sizes.
Decoupling Gating from Linearity
Specifically, we show a memorization result for networks of size $\tilde{\Omega}(\frac{m}{d})$, and improved generalization bounds.
Is Deeper Better only when Shallow is Good?
Using this result we prove that, at least in some distributions, the success of learning deep networks depends on whether the distribution can be well approximated by shallower networks, and we conjecture that this property holds in general.
A Provably Correct Algorithm for Deep Learning that Actually Works
We describe a layer-by-layer algorithm for training deep convolutional networks, where each step involves gradient updates for a two layer network followed by a simple clustering algorithm.
SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
Neural networks exhibit good generalization behavior in the over-parameterized regime, where the number of network parameters exceeds the number of observations.
On a Formal Model of Safe and Scalable Self-driving Cars
In the second part we describe a design of a system that adheres to our safety assurance requirements and is scalable to millions of cars.
Decoupling "when to update" from "how to update"
Unfortunately, this approach often leads to noisy labels.
Weight Sharing is Crucial to Succesful Optimization
Exploiting the great expressive power of Deep Neural Network architectures, relies on the ability to train them.
Failures of Gradient-Based Deep Learning
In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art.
Fast Rates for Empirical Risk Minimization of Strict Saddle Problems
We derive bounds on the sample complexity of empirical risk minimization (ERM) in the context of minimizing non-convex risks that admit the strict saddle property.
Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
Second, the Markov Decision Process model often used in robotics is problematic in our case because of unpredictable behavior of other agents in this multi-agent scenario.
Learning a Metric Embedding for Face Recognition using the Multibatch Method
This work is motivated by the engineering task of achieving a near state-of-the-art face recognition on a minimal computing budget running on an embedded system.
On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training
We compare the end-to-end training approach to a modular approach in which a system is decomposed into semantically meaningful components.
Distribution Free Learning with Local Queries
The model of learning with \emph{local membership queries} interpolates between the PAC model and the membership queries model by allowing the learner to query the label of any example that is similar to an example in the training set.
Solving Ridge Regression using Sketched Preconditioned SVRG
We develop a novel preconditioning method for ridge regression, based on recent linear sketching methods.
SDCA without Duality, Regularization, and Individual Convexity
Stochastic Dual Coordinate Ascent is a popular method for solving regularized loss minimization for the case of convex losses.
Long-term Planning by Short-term Prediction
We argue that dual versions of the MDP framework (that depend on the value function and the $Q$ function) are problematic for autonomous driving applications due to the non Markovian of the natural state space representation, and due to the continuous state and action spaces.
Minimizing the Maximal Loss: How and Why?
Second, it is often argued that there is no sense to minimize the loss on the training set too much, as it will not be reflected in the generalization loss.
Average Stability is Invariant to Data Preconditioning. Implications to Exp-concave Empirical Risk Minimization
We show that the average stability notion introduced by \cite{kearns1999algorithmic, bousquet2002stability} is invariant to data preconditioning, for a wide class of generalized linear models that includes most of the known exp-concave losses.
Beyond Convexity: Stochastic Quasi-Convex Optimization
The Normalized Gradient Descent (NGD) algorithm, is an adaptation of Gradient Descent, which updates according to the direction of the gradients, rather than the gradients themselves.
Faster SGD Using Sketched Conditioning
We propose a novel method for speeding up stochastic optimization algorithms via sketching methods, which recently became a powerful tool for accelerating algorithms for numerical linear algebra.
On Lower and Upper Bounds for Smooth and Strongly Convex Optimization Problems
This, in turn, reveals a powerful connection between a class of optimization algorithms and the analytic theory of polynomials whereby new lower and upper bounds are derived.
On Graduated Optimization for Stochastic Non-Convex Problems
We extend our algorithm and analysis to the setting of stochastic non-convex optimization with noisy gradient feedback, attaining the same convergence rate.
Strongly Adaptive Online Learning
Strongly adaptive algorithms are algorithms whose performance on every time interval is close to optimal.
SDCA without Duality
Stochastic Dual Coordinate Ascent is a popular method for solving regularized loss minimization for the case of convex losses.
SelfieBoost: A Boosting Algorithm for Deep Learning
We describe and analyze a new boosting algorithm for deep learning called SelfieBoost.
On the Computational Efficiency of Training Neural Networks
It is well-known that neural networks are computationally hard to train.
Optimal Learners for Multiclass Problems
Furthermore, we show that the sample complexity of these learners is better than the sample complexity of the ERM rule, thus settling in negative an open question due to Collins (2005).
Subspace Learning with Partial Information
The goal of subspace learning is to find a $k$-dimensional subspace of $\mathbb{R}^d$, such that the expected squared distance between instance vectors and the subspace is as small as possible.
From average case complexity to improper learning complexity
The biggest challenge in proving complexity results is to establish hardness of {\em improper learning} (a. k. a.
Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization
We introduce a proximal version of the stochastic dual coordinate ascent method and show how to accelerate the method using an inner-outer iteration procedure.
Multiclass learnability and the ERM principle
We study the sample complexity of multiclass prediction in several learning settings.
Accelerated Mini-Batch Stochastic Dual Coordinate Ascent
Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning.
An Algorithm for Training Polynomial Networks
The main goal of this paper is the derivation of an efficient layer-by-layer algorithm for training such networks, which we denote as the \emph{Basis Learner}.
Learning Sparse Low-Threshold Linear Classifiers
We consider the problem of learning a non-negative linear classifier with a $1$-norm of at most $k$, and a fixed threshold, under the hinge-loss.
The complexity of learning halfspaces using generalized linear methods
The best approximation ratio achievable by an efficient algorithm is $O\left(\frac{1/\gamma}{\sqrt{\log(1/\gamma)}}\right)$ and is achieved using an algorithm from the above class.
Efficient Active Learning of Halfspaces: an Aggressive Approach
Our efficient aggressive active learner of half-spaces has formal approximation guarantees that hold when the pool is separable with a margin.
ShareBoost: Efficient multiclass learning with feature sharing
We consider the problem of learning a multiclass predictor that uses only few features, and in particular, the number of used features should increase sub-linearly with the number of possible classes.
Mind the Duality Gap: Logarithmic regret algorithms for online optimization
We describe a primal-dual framework for the design and analysis of online strongly convex optimization algorithms.
Fast Rates for Regularized Objectives
We show that the empirical minimizer of a stochastic strongly convex objective, where the stochastic component is linear, converges to the population minimizer with rate $O(1/n)$.
