Search Results for author:
Found 4 papers, 2 papers with code
AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
Modern deep learning (DL) architectures are trained using variants of the SGD algorithm that is run with a $\textit{manually}$ defined learning rate schedule, i. e., the learning rate is dropped at the pre-defined epochs, typically when the training loss is expected to saturate.
Overcoming Catastrophic Forgetting via Direction-Constrained Optimization
We study the principal directions of the trajectory of the optimizer after convergence and show that traveling along a few top principal directions can quickly bring the parameters outside the cone but this is not the case for the remaining directions.
Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
Finally, we implement an asynchronous version of our algorithm and extend it to the multi-leader setting, where we form groups of workers, each represented by its own local leader (the best performer in a group), and update each worker with a corrective direction comprised of two attractive forces: one to the local, and one to the global leader (the best performer among all workers).
Invertible Autoencoder for domain adaptation
However, it does not explicitly enforce $F_{BA}$ to be an inverse operation to $F_{AB}$.
