Search Results for author:
Found 10 papers, 2 papers with code
The Hidden Attention of Mamba Models
The Mamba layer offers an efficient selective state space model (SSM) that is highly effective in modeling multiple domains, including NLP, long-range sequence processing, and computer vision.
On the Long Range Abilities of Transformers
Despite their dominance in modern DL and, especially, NLP domains, transformer architectures exhibit sub-optimal performance on long-range tasks compared to recent layers that are specifically designed for this purpose.
Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption
This innovation enables us to perform secure inference on LMs with WikiText-103.
Multi-Dimensional Hyena for Spatial Inductive Bias
Our empirical findings indicate that the proposed Hyena N-D layer boosts the performance of various Vision Transformer architectures, such as ViT, Swin, and DeiT across multiple datasets.
Efficient Skip Connections Realization for Secure Inference on Encrypted Data
Homomorphic Encryption (HE) is a cryptographic tool that allows performing computation under encryption, which is used by many privacy-preserving machine learning solutions, for example, to perform secure classification.
2-D SSM: A General Spatial Layer for Visual Transformers
For example, vision transformers equipped with our layer exhibit effective performance even without positional encoding
Decision S4: Efficient Sequence-Based RL via State Spaces Layers
Recently, sequence learning methods have been applied to the problem of off-policy Reinforcement Learning, including the seminal work on Decision Transformers, which employs transformers for this task.
Focus Your Attention (with Adaptive IIR Filters)
We present a new layer in which dynamic (i. e., input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention.
Ranked #1 on
Language Modelling
on Text8
Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption
Training large-scale CNNs that during inference can be run under Homomorphic Encryption (HE) is challenging due to the need to use only polynomial operations.
Recovering AES Keys with a Deep Cold Boot Attack
In this work, we combine a novel cryptographic variant of a deep error correcting code technique with a modified SAT solver scheme to apply the attack on AES keys.
