Search Results for author:
Found 24 papers, 13 papers with code
MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
We then propose the Mixture of Layer Experts (MoLEx), a novel sparse mixture of experts (SMoE) whose experts are layers in the pre-trained model.
Distance-Based Tree-Sliced Wasserstein Distance
However, projecting measures onto low-dimensional spaces can lead to a loss of topological information.
Spherical Tree-Sliced Wasserstein Distance
Inspired by this approach, in this paper, we present an adaptation of tree systems on OT problems for measures supported on a sphere.
CAMEx: Curvature-aware Merging of Experts
In this paper, we introduce CAMEx (\textbf{C}urvature-\textbf{A}ware \textbf{M}erging of \textbf{Ex}perts), a novel expert merging protocol that incorporates natural gradients to account for the non-Euclidean curvature of the parameter manifold.
Tight Clusters Make Specialized Experts
Our AC router enables the MoE model to obtain three connected benefits: 1) faster convergence, 2) better robustness to data corruption, and 3) overall performance improvement, as experts are specialized in semantically distinct regions of the input space.
An Attention-based Framework for Fair Contrastive Learning
Contrastive learning has proven instrumental in learning unbiased representations of data, especially in complex environments characterized by high-cardinality and high-dimensional sensitive information.
MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts
Sparse Mixture of Experts (SMoE) has become the key to unlocking unparalleled scalability in deep learning.
Monomial Matrix Group Equivariant Neural Functional Networks
We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN).
Tree-Sliced Wasserstein Distance on a System of Lines
While SW is prone to loss of topological information of input measures due to relying on one-dimensional projection, TSW is more flexible and has a higher degree of freedom by choosing a tree rather than a line to alleviate the curse of dimensionality in SW.
A Primal-Dual Framework for Transformers and Neural Networks
Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision.
Elliptical Attention
Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision.
Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis
In our work, we derive self-attention from kernel principal component analysis (kernel PCA) and show that self-attention projects its query vectors onto the principal component axes of its key matrix in a feature space.
PIDformer: Transformer Meets Control Theory
Motivated by this control framework, we derive a novel class of transformers, PID-controlled Transformer (PIDformer), aimed at improving robustness and mitigating the rank-collapse issue inherent in softmax transformers.
Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals
Transformers have achieved remarkable success in a wide range of natural language processing and computer vision applications.
p-Laplacian Transformer
$p$-Laplacian regularization, rooted in graph and image signal processing, introduces a parameter $p$ to control the regularization effect on these data.
From Coupled Oscillators to Graph Neural Networks: Reducing Over-smoothing via a Kuramoto Model-based Approach
We propose the Kuramoto Graph Neural Network (KuramotoGNN), a novel class of continuous-depth graph neural networks (GNNs) that employs the Kuramoto model to mitigate the over-smoothing phenomenon, in which node features in GNNs become indistinguishable as the number of layers increases.
ARIST: An Effective API Argument Recommendation Approach
To implement this idea in the recommendation process, ARIST combines program analysis (PA), language models (LMs), and several features specialized for the recommendation task which consider the functionality of formal parameters and the positional information of code elements (e. g., variables or method calls) in the given context.
Improving Transformers with Probabilistic Attention Keys
Inspired by this observation, we propose Transformer with a Mixture of Gaussian Keys (Transformer-MGK), a novel transformer architecture that replaces redundant heads in transformers with a mixture of keys at each head.
Heavy Ball Neural Ordinary Differential Equations
We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference.
FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
For instance, FMMformers achieve an average classification accuracy of $60. 74\%$ over the five Long Range Arena tasks, which is significantly better than the standard transformer's average accuracy of $58. 70\%$.
MomentumRNN: Integrating Momentum into Recurrent Neural Networks
Designing deep neural networks is an art that often involves an expensive search over candidate architectures.
Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent
Nesterov accelerated gradient (NAG) improves the convergence rate of gradient descent (GD) for convex optimization using a specially designed momentum; however, it accumulates error when an inexact gradient is used (such as in SGD), slowing convergence at best and diverging at worst.
InfoCNF: An Efficient Conditional Continuous Normalizing Flow with Adaptive Solvers
Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation.
InfoCNF: Efficient Conditional Continuous Normalizing Flow Using Adaptive Solvers
Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation.
