Stochastic Optimization
278 papers with code • 12 benchmarks • 11 datasets
Stochastic Optimization is the task of optimizing certain objective functional by generating and using stochastic random variables. Usually the Stochastic Optimization is an iterative process of generating random variables that progressively finds out the minima or the maxima of the objective functional. Stochastic Optimization is usually applied in the non-convex functional spaces where the usual deterministic optimization such as linear or quadratic programming or their variants cannot be used.
Source: ASOC: An Adaptive Parameter-free Stochastic Optimization Techinique for Continuous Variables
Benchmarks
These leaderboards are used to track progress in Stochastic Optimization
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Stochastic Optimization models and implementationsDatasets
CIFAR-10
CIFAR-100
MNIST
GLUE
Penn Treebank
AG News
CoLA
Amazon Product Data
MSRA Hand
Most implemented papers
Variational Inference: A Review for Statisticians
magister-informatica-uach/INFO3XX
•
•
One of the core problems of modern statistics is to approximate difficult-to-compute probability densities.
Training Deep Networks without Learning Rates Through Coin Betting
bremen79/cocob
•
•
NeurIPS 2017
Instead, we reduce the optimization process to a game of betting on a coin and propose a learning-rate-free optimal algorithm for this scenario.
Agnostic Federated Learning
A key learning scenario in large-scale applications is that of federated learning, where a centralized model is trained based on data originating from a large number of clients.
Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates
Adaptive gradient methods for stochastic optimization adjust the learning rate for each parameter locally.
Learning concise representations for regression by evolving networks of trees
We propose and study a method for learning interpretable representations for the task of regression.
Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods.
Second-Order Stochastic Optimization for Machine Learning in Linear Time
First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity.
Revisiting Distributed Synchronous SGD
Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony.
Kronecker Determinantal Point Processes
Determinantal Point Processes (DPPs) are probabilistic models over all subsets a ground set of $N$ items.
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
booydar/t5-experiments
•
•
ICML 2018
In several recently proposed stochastic optimization methods (e. g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients.
