Search Results for author:
Found 7 papers, 3 papers with code
Functional Interpolation for Relative Positions Improves Long Context Transformers
Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models.
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs).
Is $L^2$ Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?
In particular, we leverage the concept of stability in the literature of partial differential equation to study the asymptotic behavior of the learned solution as the loss approaches zero.
Your Transformer May Not be as Powerful as You Expect
Extensive experiments covering typical architectures and tasks demonstrate that our model is parameter-efficient and can achieve superior performance to strong baselines in a wide range of applications.
Learning Physics-Informed Neural Networks without Stacked Back-propagation
In this work, we develop a novel approach that can significantly accelerate the training of Physics-Informed Neural Networks.
Can Vision Transformers Perform Convolution?
Several recent studies have demonstrated that attention-based networks, such as Vision Transformer (ViT), can outperform Convolutional Neural Networks (CNNs) on several computer vision tasks without using convolutional layers.
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Since in many state-of-the-art models, relative positional encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.
