Linear Mode Connectivity
14 papers with code • 0 benchmarks • 0 datasets
Linear Mode Connectivity refers to the relationship between input and output variables in a linear regression model. In a linear regression model, input variables are combined with weights to predict output variables. Understanding the linear model connectivity can help interpret model results and identify which input variables are most important for predicting output variables.
Benchmarks
These leaderboards are used to track progress in Linear Mode Connectivity
Most implemented papers
Git Re-Basin: Merging Models modulo Permutation Symmetries
The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease.
Linear Mode Connectivity and the Lottery Ticket Hypothesis
facebookresearch/open_lth
•
•
ICML 2020
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e. g., random data order and augmentation).
Proving Linear Mode Connectivity of Neural Networks via Optimal Transport
The energy landscape of high-dimensional non-convex optimization problems is crucial to understanding the effectiveness of modern deep neural network architectures.
Vanishing Feature: Diagnosing Model Merging and Beyond
Model merging offers an efficient way to combine pre-trained neural networks but often suffers from inconsistent performance, especially when merging models with different initializations.
Federated Learning over Connected Modes
Statistical heterogeneity in federated learning poses two major challenges: slow global training due to conflicting gradient signals, and the need of personalization for local distributions.
Linear Mode Connectivity in Multitask and Continual Learning
Continual (sequential) training and multitask (simultaneous) training are often attempting to solve the same overall objective: to find a solution that performs well on all considered tasks.
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
rahimentezari/permutationinvariance
•
•
ICLR 2022
In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them.
Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks
In our framework, the fusion occurs in a layer-wise manner and builds on an interpretation of a node in a network as a function of the layer preceding it.
Re-basin via implicit Sinkhorn differentiation
fagp/sinkhorn-rebasin
•
•
CVPR 2023
The recent emergence of new algorithms for permuting models into functionally equivalent regions of the solution space has shed some light on the complexity of error surfaces, and some promising properties like mode connectivity.
Lottery Tickets in Evolutionary Optimization: On Sparse Backpropagation-Free Trainability
Is the lottery ticket phenomenon an idiosyncrasy of gradient-based training or does it generalize to evolutionary optimization?
