Search Results for author:
Found 11 papers, 2 papers with code
FOSI: Hybrid First and Second Order Optimization
Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions.
Unsupervised Frequent Pattern Mining for CEP
We present REDEEMER (REinforcement baseD cEp pattErn MinER), a novel reinforcement and active learning approach aimed at mining CEP patterns that allow expansion of the knowledge extracted while reducing the human effort required.
Coin Flipping Neural Networks
We show that a CFNN can approximate the indicator of a $d$-dimensional ball to arbitrary accuracy with only 2 layers and $\mathcal{O}(1)$ neurons, where a 2-layer deterministic network was shown to require $\Omega(e^d)$ neurons, an exponential improvement (arXiv:1610. 09887).
Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism
Fine-tuning is an increasingly common technique that leverages transfer learning to dramatically expedite the training of huge, high-quality models.
Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delays
We consider stochastic convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory.
Gap-Aware Mitigation of Gradient Staleness
In this paper we define the Gap as a measure of gradient staleness and propose Gap-Aware (GA), a novel asynchronous-distributed method that penalizes stale gradients linearly to the Gap and performs well even when scaling to large numbers of workers.
It's Not What Machines Can Learn, It's What We Cannot Teach
Can deep neural networks learn to solve any task, and in particular problems of high complexity?
Adaptive Communication Bounds for Distributed Online Learning
The learning performance of such a protocol is intuitively optimal if approximately the same loss is incurred as in a hypothetical serial setting.
Gap Aware Mitigation of Gradient Staleness
In this paper we define the Gap as a measure of gradient staleness and propose Gap-Aware (GA), a novel asynchronous-distributed method that penalizes stale gradients linearly to the Gap and performs well even when scaling to large numbers of workers.
Taming Momentum in a Distributed Asynchronous Environment
We propose DANA: a novel technique for asynchronous distributed SGD with momentum that mitigates gradient staleness by computing the gradient on an estimated future position of the model's parameters.
DANA: Scalable Out-of-the-box Distributed ASGD Without Retuning
We propose DANA, a novel approach that scales out-of-the-box to large clusters using the same hyperparameters and learning schedule optimized for training on a single worker, while maintaining similar final accuracy without additional overhead.
