Search Results for author:
Found 61 papers, 22 papers with code
An SDE for Modeling SAM: Theory and Insights
We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent.
Mastering Spatial Graph Prediction of Road Networks
Accurately predicting road networks from satellite images requires a global understanding of the network topology.
On the Theoretical Properties of Noise Correlation in Stochastic Optimization
This generalizes processes based on Brownian motion, such as the Ornstein-Uhlenbeck process.
A Theoretical Analysis of the Learning Dynamics under Class Imbalance
We find that GD is not guaranteed to decrease the loss for each class but that this problem can be addressed by performing a per-class normalization of the gradient.
Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
First, we show that rank collapse of the tokens' representations hinders training by causing the gradients of the queries and keys to vanish at initialization.
Phenomenology of Double Descent in Finite-Width Neural Networks
`Double descent' delineates the generalization behaviour of models depending on the regime they belong to: under- or over-parameterized.
A Globally Convergent Evolutionary Strategy for Stochastic Constrained Optimization with Applications to Reinforcement Learning
Evolutionary strategies have recently been shown to achieve competing levels of performance for complex optimization problems in reinforcement learning.
Anticorrelated Noise Injection for Improved Generalization
Injecting artificial noise into gradient descent (GD) is commonly employed to improve the performance of machine learning models.
Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity
Gradient descent ascent (GDA), the simplest single-loop algorithm for nonconvex minimax optimization, is widely used in practical applications such as generative adversarial networks (GANs) and adversarial training.
On the Second-order Convergence Properties of Random Search Methods
We prove that this approach converges to a second-order stationary point at a much faster rate than vanilla methods: namely, the complexity in terms of the number of function evaluations is only linear in the problem dimension.
Neural Symbolic Regression that Scales
We procedurally generate an unbounded set of equations, and simultaneously pre-train a Transformer to predict the symbolic equation from a corresponding set of input-output-pairs.
Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
This paper revisits the so-called vanishing gradient phenomenon, which commonly occurs in deep randomly initialized neural networks.
Learning Generative Models of Textured 3D Meshes from Real-World Images
Recent advances in differentiable rendering have sparked an interest in learning generative models of textured 3D meshes from image collections.
Generative Minimization Networks: Training GANs Without Competition
Among known problems experienced by practitioners is the lack of convergence guarantees or convergence to a non-optimum cycle.
Direct-Search for a Class of Stochastic Min-Max Problems
Recent applications in machine learning have renewed the interest of the community in min-max optimization problems.
Batch normalization provably avoids ranks collapse for randomly initialised deep networks
Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used.
A Seq2Seq approach to Symbolic Regression
Deep neural networks have proved to be powerful function approximators.
Scalable Graph Networks for Particle Simulations
However, the dynamics of many real-world systems are challenging to learn due to the presence of nonlinear potentials and a number of interactions that scales quadratically with the number of particles $N$, as in the case of the N-body problem.
An Accelerated DFO Algorithm for Finite-sum Convex Functions
Derivative-free optimization (DFO) has recently gained a lot of momentum in machine learning, spawning interest in the community to design faster methods for problems where gradients are not accessible.
Randomized Block-Diagonal Preconditioning for Parallel Learning
We study preconditioned gradient-based optimization methods where the preconditioning matrix has block-diagonal form.
Convolutional Generation of Textured 3D Meshes
A key contribution of our work is the encoding of the mesh and texture as 2D representations, which are semantically aligned and can be easily modeled by a 2D convolutional GAN.
Emulation of cosmological mass maps with conditional generative adversarial networks
Weak gravitational lensing mass maps play a crucial role in understanding the evolution of structures in the universe and our ability to constrain cosmological models.
Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks
Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used.
Practical Accelerated Optimization on Riemannian Manifolds
We develop a new Riemannian descent algorithm with an accelerated rate of convergence.
Optimization and Control
Controlling Style and Semantics in Weakly-Supervised Image Generation
We propose a weakly-supervised approach for conditional image generation of complex scenes where a user has fine control over objects appearing in the scene.
A Stochastic Tensor Method for Non-convex Optimization
We present a stochastic optimization method that uses a fourth-order regularized model to find local minima of smooth and potentially non-convex objective functions with a finite-sum structure.
Shadowing Properties of Optimization Algorithms
Ordinary differential equation (ODE) models of gradient-based optimization methods can provide insights into the dynamics of learning and inspire the design of new algorithms.
A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization
We propose a novel second-order ODE as the continuous-time limit of a Riemannian accelerated gradient-based method on a manifold with curvature bounded from below.
Optimization and Control
Ellipsoidal Trust Region Methods for Neural Network Training
We investigate the use of ellipsoidal trust region constraints for second-order optimization of neural networks.
Cosmological N-body simulations: a challenge for scalable generative models
Our results show that the proposed model produces samples of high visual quality, although the statistical analysis reveals that capturing rare features in the data poses significant problems for the generative models.
The Role of Memory in Stochastic Optimization
We first derive a general continuous-time model that can incorporate arbitrary types of memory, for both deterministic and stochastic settings.
Cosmological constraints with deep learning from KiDS-450 weak lensing maps
We present the cosmological results with a CNN from the KiDS-450 tomographic weak lensing dataset, constraining the total matter density $\Omega_m$, the fluctuation amplitude $\sigma_8$, and the intrinsic alignment amplitude $A_{\rm{IA}}$.
Cosmology and Nongalactic Astrophysics
Adaptive norms for deep learning with regularized Newton methods
We investigate the use of regularized Newton methods with adaptive norms for optimizing neural networks.
Evaluating GANs via Duality
Generative Adversarial Networks (GANs) have shown great results in accurately modeling complex distributions, but their training is known to be difficult due to instabilities caused by a challenging minimax optimization problem.
A domain agnostic measure for monitoring and evaluating GANs
Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training.
Continuous-time Models for Stochastic Optimization Algorithms
We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods.
Cosmological constraints from noisy convergence maps through deep learning
We find that, for a shape noise level corresponding to 8. 53 galaxies/arcmin$^2$ and the smoothing scale of $\sigma_s = 2. 34$ arcmin, the network is able to generate 45% tighter constraints.
Cosmology and Nongalactic Astrophysics
A Distributed Second-Order Algorithm You Can Trust
Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years.
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks.
Adversarially Robust Training through Structured Gradient Regularization
We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations.
Local Saddle Point Optimization: A Curvature Exploitation Approach
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems.
Escaping Saddles with Stochastic Gradients
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions.
Fast cosmic web simulations with generative adversarial networks
Computational models of the underlying physical processes, such as classical N-body simulations, are extremely resource intensive, as they track the action of gravity in an expanding universe using billions of particles as tracers of the cosmic matter distribution.
Fast Point Spread Function Modeling with Deep Learning
We find that our approach is able to accurately reproduce the SDSS PSF at the pixel level, which, due to the speed of both the model evaluation and the parameter estimation, offers good prospects for incorporating our method into the $MCCL$ framework.
Semantic Interpolation in Implicit Models
In implicit models, one often interpolates between sampled points in latent space.
Flexible Prior Distributions for Deep Generative Models
We consider the problem of training generative models with deep neural networks as generators, i. e. to map latent codes to data points.
Learning Aerial Image Segmentation from Online Maps
We adapt a state-of-the-art CNN architecture for semantic segmentation of buildings and roads in aerial images, and compare its performance when using different training data sets, ranging from manually labeled, pixel-accurate ground truth of the same city to automatic training data derived from OpenStreetMap data from distant locations.
Cosmological model discrimination with Deep Learning
We find that our implementation of DCNN outperforms the skewness and kurtosis statistics, especially for high noise levels.
An Online Learning Approach to Generative Adversarial Networks
We consider the problem of training generative models with a Generative Adversarial Network (GAN).
Stabilizing Training of Generative Adversarial Networks through Regularization
Deep generative models based on Generative Adversarial Networks (GANs) have demonstrated impressive sample quality but in order to work they require a careful choice of architecture, parameter initialization, and selection of hyper-parameters.
Sub-sampled Cubic Regularization for Non-convex Optimization
This approach is particularly attractive because it escapes strict saddle points and it provides stronger convergence guarantees than first- and second-order as well as classical trust region methods.
Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classification
This paper presents a novel approach for multi-lingual sentiment classification in short texts.
A Semi-supervised Framework for Image Captioning
We here propose a novel way of using such textual data by artificially generating missing visual information.
Radio frequency interference mitigation using deep convolutional neural networks
We employ a special type of Convolutional Neural Network, the U-Net, that enables the classification of clean signal and RFI signatures in 2D time-ordered data acquired from a radio telescope.
Instrumentation and Methods for Astrophysics
DynaNewton - Accelerating Newton's Method for Machine Learning
Solutions on this path are tracked such that the minimizer of the previous objective is guaranteed to be within the quadratic convergence region of the next objective to be optimized.
Starting Small -- Learning with Adaptive Sample Sizes
For many machine learning problems, data is abundant and it may be prohibitive to make multiple passes through the full training set.
Probabilistic Bag-Of-Hyperlinks Model for Entity Linking
We demonstrate the accuracy of our approach on a wide range of benchmark datasets, showing that it matches, and in many cases outperforms, existing state-of-the-art methods.
Variance Reduced Stochastic Gradient Descent with Neighbors
As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms.
A Variance Reduced Stochastic Newton Method
Quasi-Newton methods are widely used in practise for convex loss minimization problems.
Learning for Structured Prediction Using Approximate Subgradient Descent with Working Sets
We propose a working set based approximate subgradient descent algorithm to minimize the margin-sensitive hinge loss arising from the soft constraints in max-margin learning frameworks, such as the structured SVM.
