Search Results for author:
Found 15 papers, 7 papers with code
A Negative Result on Gradient Matching for Selective Backprop
With increasing scale in model and dataset size, the training of deep neural networks becomes a massive computational burden.
Continual Learning with Low Rank Adaptation
Recent work using pretrained transformers has shown impressive performance when fine-tuned with data from the downstream problem of interest.
Renate: A Library for Real-World Continual Learning
We discuss requirements for the use of continual learning algorithms in practice, from which we derive design principles for Renate.
PASHA: Efficient HPO and NAS with Progressive Resource Allocation
Hyperparameter optimization (HPO) and neural architecture search (NAS) are methods of choice to obtain the best-in-class machine learning models, but in practice they can be costly to run.
Gradient-Matching Coresets for Rehearsal-Based Continual Learning
Most widely-used CL methods rely on a rehearsal memory of data points to be reused while training on new data.
Gradient-matching coresets for continual learning
We devise a coreset selection method based on the idea of gradient matching: The gradients induced by the coreset should match, as closely as possible, those induced by the original training dataset.
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters.
The Geometry of Sign Gradient Descent
Sign-based optimization methods have become popular in machine learning due to their favorable communication cost in distributed optimization and their surprisingly good performance in neural network training.
Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information.
Holographic and other Point Set Distances for Machine Learning
We introduce an analytic distance function for moderately sized point sets of known cardinality that is shown to have very desirable properties, both as a loss function as well as a regularizer for machine learning applications.
DeepOBS: A Deep Learning Optimizer Benchmark Suite
We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization.
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation
We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions.
Ranked #66 on
Monocular Depth Estimation
on KITTI Eigen split
Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
The ADAM optimizer is exceedingly popular in the deep learning community.
Early Stopping without a Validation Set
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization.
Coupling Adaptive Batch Sizes with Learning Rates
The batch size significantly influences the behavior of the stochastic optimization algorithm, though, since it determines the variance of the gradient estimates.
