Search Results for author:
Found 52 papers, 23 papers with code
IPBoost – Non-Convex Boosting via Integer Programming
Recently non-convex optimization approaches for solving machine learning problems have gained significant attention.
Neural Parameter Regression for Explicit Representations of PDE Solution Operators
We introduce Neural Parameter Regression (NPR), a novel framework specifically developed for learning solution operators in Partial Differential Equations (PDEs).
On the Byzantine-Resilience of Distillation-Based Federated Learning
In this work, we study the performance of such approaches in the byzantine setting, where a subset of the clients act in an adversarial manner aiming to disrupt the learning process.
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
Neural Networks can be efficiently compressed through pruning, significantly reducing storage and computational demands while maintaining predictive performance.
Group-wise Sparse and Explainable Adversarial Attacks
However, crafting such attacks poses an optimization challenge, as it involves computing norms for groups of pixels within a non-convex objective.
Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging
Model soups (Wortsman et al., 2022) enhance generalization and out-of-distribution (OOD) performance by averaging the parameters of multiple models into a single one, without increasing inference time.
Accelerated Methods for Riemannian Min-Max Optimization Ensuring Bounded Geometric Penalties
In this work, we study optimization problems of the form $\min_x \max_y f(x, y)$, where $f(x, y)$ is defined on a product Riemannian manifold $\mathcal{M} \times \mathcal{N}$ and is $\mu_x$-strongly geodesically convex (g-convex) in $x$ and $\mu_y$-strongly g-concave in $y$, for $\mu_x, \mu_y \geq 0$.
Online Learning for Scheduling MIP Heuristics
In this work, we propose an online learning approach that adapts the application of heuristics towards the single instance at hand.
Accelerated Riemannian Optimization: Handling Constraints with a Prox to Bound Geometric Penalties
For smooth functions, we show we can implement the prox step inexactly with first-order methods in Riemannian balls of certain diameter that is enough for global accelerated optimization.
Convex mixed-integer optimization with Frank-Wolfe methods
These relaxations are solved with a Frank-Wolfe algorithm over the convex hull of mixed-integer feasible points instead of the continuous relaxation via calls to a mixed-integer linear solver as the linear oracle.
Approximate Vanishing Ideal Computations at Scale
The vanishing ideal of a set of points $X = \{\mathbf{x}_1, \ldots, \mathbf{x}_m\}\subseteq \mathbb{R}^n$ is the set of polynomials that evaluate to $0$ over all points $\mathbf{x} \in X$ and admits an efficient representation by a finite subset of generators.
Interpretability Guarantees with Merlin-Arthur Classifiers
We propose an interactive multi-agent classifier that provides provable interpretability guarantees even for complex agents such as neural networks.
Compression-aware Training of Neural Networks using Frank-Wolfe
Many existing Neural Network pruning approaches rely on either retraining or inducing a strong bias in order to converge to a sparse solution throughout training.
The Machine Learning for Combinatorial Optimization Competition (ML4CO): Results and Insights
Combinatorial optimization is a well-established area in operations research and computer science.
Training Characteristic Functions with Reinforcement Learning: XAI-methods play Connect Four
Given only a standard classifier function, it is unclear how partial input should be realised.
Conditional Gradients for the Approximately Vanishing Ideal
To accommodate the noise in the data set, we introduce the Conditional Gradients Approximately Vanishing Ideal algorithm (CGAVI) for the construction of the set of generators of the approximately vanishing ideal.
How I Learned to Stop Worrying and Love Retraining
Many Neural Network Pruning approaches consist of several iterative training and pruning steps, seemingly losing a significant amount of their performance after pruning and then recovering it in the subsequent retraining phase.
Interpretable Neural Networks with Frank-Wolfe: Sparse Relevance Maps and Relevance Orderings
We study the effects of constrained optimization formulations and Frank-Wolfe algorithms for obtaining interpretable neural network predictions.
Scalable Frank-Wolfe on Generalized Self-concordant Functions via Simple Steps
Generalized self-concordance is a key property present in the objective function of many important learning problems.
Efficient Online-Bandit Strategies for Minimax Learning Problems
Several learning problems involve solving min-max problems, e. g., empirical distributional robust learning or learning with non-standard aggregated losses.
Learning to Schedule Heuristics in Branch and Bound
Compared to the default settings of a state-of-the-art academic MIP solver, we are able to reduce the average primal integral by up to 49% on two classes of challenging instances.
Learning to Schedule Heuristics in Branch-and-Bound
In this work, we propose the first data-driven framework for scheduling heuristics in an exact MIP solver.
Linear Bandits on Uniformly Convex Sets
Linear bandit algorithms yield $\tilde{\mathcal{O}}(n\sqrt{T})$ pseudo-regret bounds on compact convex action sets $\mathcal{K}\subset\mathbb{R}^n$ and two types of structural assumptions lead to better pseudo-regret bounds.
Parameter-free Locally Accelerated Conditional Gradients
Projection-free conditional gradient (CG) methods are the algorithms of choice for constrained optimization setups in which projections are often computationally prohibitive but linear optimization over the constraint set remains computationally feasible.
Local and Global Uniform Convexity Conditions
We review various characterizations of uniform convexity and smoothness on norm balls in finite-dimensional spaces and connect results stemming from the geometry of Banach spaces with \textit{scaling inequalities} used in analysing the convergence of optimization methods.
Adversaries in Online Learning Revisited: with applications in Robust Optimization and Adversarial training
We revisit the concept of "adversary" in online learning, motivated by solving robust optimization and adversarial training using online learning methods.
Complexity of Linear Minimization and Projection on Some Sets
The Frank-Wolfe algorithm is a method for constrained optimization that relies on linear minimizations, as opposed to projections.
CINDy: Conditional gradient-based Identification of Non-linear Dynamics -- Noise-robust recovery
Governing equations are essential to the study of nonlinear dynamics, often enabling the prediction of previously unseen behaviors as well as the inclusion into control strategies.
Dynamical Systems Applications
Dual Prices for Frank--Wolfe Algorithms
In this note we observe that for constrained convex minimization problems $\min_{x \in P}f(x)$ over a polytope $P$, dual prices for the linear program $\min_{z \in P} \nabla f(x) z$ obtained from linearization at approximately optimal solutions $x$ have a similar interpretation of rate of change in optimal value as for linear programming, providing a convex form of sensitivity analysis.
Optimization and Control
Deep Neural Network Training with Frank-Wolfe
In particular, we show the general feasibility of training Neural Networks whose parameters are constrained by a convex feasible region using Frank-Wolfe algorithms and compare different stochastic variants.
Projection-Free Adaptive Gradients for Large-Scale Optimization
The complexity in large-scale optimization can lie in both handling the objective function and handling the constraint set.
Walking in the Shadow: A New Perspective on Descent Directions for Constrained Minimization
We combine these insights into a novel Shadow-CG method that uses FW and shadow steps, while enjoying linear convergence, with a rate that depends on the number of breakpoints in its projection curve, rather than the pyramidal width.
Boosting Frank-Wolfe by Chasing Gradients
The Frank-Wolfe algorithm has become a popular first-order optimization algorithm for it is simple and projection-free, and it has been successfully applied to a variety of real-world problems.
Second-order Conditional Gradient Sliding
Constrained second-order convex optimization algorithms are the method of choice when a high accuracy solution to a problem is needed, due to their local quadratic convergence.
IPBoost -- Non-Convex Boosting via Integer Programming
Recently non-convex optimization approaches for solving machine learning problems have gained significant attention.
Revisiting the Approximate Carathéodory Problem via the Frank-Wolfe Algorithm
The approximate Carath\'eodory theorem states that given a compact convex set $\mathcal{C}\subset\mathbb{R}^n$ and $p\in\left[2,+\infty\right[$, each point $x^*\in\mathcal{C}$ can be approximated to $\epsilon$-accuracy in the $\ell_p$-norm as the convex combination of $\mathcal{O}(pD_p^2/\epsilon^2)$ vertices of $\mathcal{C}$, where $D_p$ is the diameter of $\mathcal{C}$ in the $\ell_p$-norm.
Self-Supervised Policy Adaptation
This generates pairs of state encodings, i. e., a new representation from the environment and a (biased) old representation from the forward model, that allow us to bootstrap a neural network model for state translation.
Locally Accelerated Conditional Gradients
As such, they are frequently used in solving smooth convex optimization problems over polytopes, for which the computational cost of orthogonal projections would be prohibitive.
Blended Matching Pursuit
Matching pursuit algorithms are an important class of algorithms in signal processing and machine learning.
An Online-Learning Approach to Inverse Optimization
We also introduce several generalizations, such as the approximate learning of non-linear objective functions, dynamically changing as well as parameterized objectives and the case of suboptimal observed decisions.
Principled Deep Neural Network Training through Linear Programming
Our results provide a new perspective on training problems through the lens of polyhedral theory and reveal a strong structure arising from these problems.
Efficient algorithms for robust submodular maximization under matroid constraints
In this work, we consider robust submodular maximization with matroid constraints.
Blended Conditional Gradients: the unconditioning of conditional gradients
We present a blended conditional gradient approach for minimizing a smooth convex function over a polytope P, combining the Frank--Wolfe algorithm (also called conditional gradient) with gradient-based steps, different from away steps and pairwise steps, but still achieving linear convergence for strongly convex functions, along with good practical performance.
Emulating the Expert: Inverse Optimization through Online Learning
In this paper, we demonstrate how to learn the objective function of a decision maker while only observing the problem input data and the decision maker’s corresponding decisions over multiple rounds.
Reinforcement Learning under Model Mismatch
We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it.
Conditional Accelerated Lazy Stochastic Gradient Descent
In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate $O\left(\frac{1}{\varepsilon^2}\right)$ improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan and Kale [2012] with convergence rate $O\left(\frac{1}{\varepsilon^4}\right)$.
Hierarchical Clustering via Spreading Metrics
We also prove that our algorithm returns an $O(\log{n})$-approximate hierarchical clustering for a generalization of this cost function also studied in [arXiv:1510. 05043].
Lazifying Conditional Gradient Algorithms
Conditional gradient algorithms (also often called Frank-Wolfe algorithms) are popular due to their simplicity of only requiring a linear optimization oracle and more recently they also gained significant traction for online learning.
An efficient high-probability algorithm for Linear Bandits
For the linear bandit problem, we extend the analysis of algorithm CombEXP from [R. Combes, M. S. Talebi Mazraeh Shahi, A. Proutiere, and M. Lelarge.
Sequential Information Guided Sensing
We study the value of information in sequential compressed sensing by characterizing the performance of sequential information guided sensing in practical scenarios when information is inaccurate.
Sequential Sensing with Model Mismatch
We characterize the performance of sequential information guided sensing, Info-Greedy Sensing, when there is a mismatch between the true signal model and the assumed model, which may be a sample estimate.
Info-Greedy sequential adaptive compressed sensing
We present an information-theoretic framework for sequential adaptive compressed sensing, Info-Greedy Sensing, where measurements are chosen to maximize the extracted information conditioned on the previous measurements.
