Search Results for author:
Found 34 papers, 29 papers with code
On the Benefits of Memory for Modeling Time-Dependent PDEs
Data-driven techniques have emerged as a promising alternative to traditional numerical methods for solving partial differential equations (PDEs).
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models
In this work, we present a method that is able to distill a pretrained Transformer architecture into alternative architectures such as state space models (SSMs).
Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers
We identify a key axis of matrix parameterizations termed sequence alignment, which increases the flexibility and performance of matrix mixers, providing insights into the strong performance of Transformers and recent SSMs such as Mamba.
An Empirical Study of Mamba-based Language Models
On an additional 23 long-context tasks, the hybrid model continues to closely match or exceed the Transformer on average.
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale.
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Large-scale sequence modeling has sparked rapid advances that now extend into biology and genomics.
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale.
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.
Ranked #25 on
Language Modelling
on LAMBADA
Augmenting conformers with structured state-space sequence models for online speech recognition
Online speech recognition, where the model only accesses context to the left, is an important and challenging use case for ASR systems.
Resurrecting Recurrent Neural Networks for Long Sequences
Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train.
Structured State Space Models for In-Context Reinforcement Learning
We propose a modification to a variant of S4 that enables us to initialise and reset the hidden state in parallel, allowing us to tackle reinforcement learning tasks.
Modelling Long Range Dependencies in $N$D: From Task-Specific to a General Purpose CNN
Performant Convolutional Neural Network (CNN) architectures must be tailored to specific tasks in order to consider the length, resolution, and dimensionality of the input data.
Pretraining Without Attention
Even so, BiGS is able to match BERT pretraining accuracy on GLUE and can be extended to long-form pretraining of 4096 tokens without approximation.
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
On ImageNet-1k, S4ND exceeds the performance of a Vision Transformer baseline by $1. 5\%$ when training with a $1$D sequence of patches, and matches ConvNeXt when modeling images in $2$D.
How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections
Linear time-invariant state space models (SSM) are a classical model from engineering and statistics, that have recently been shown to be very promising in machine learning through the Structured State Space sequence model (S4).
Ranked #2 on
Long-range modeling
on LRA
On the Parameterization and Initialization of Diagonal State Space Models
On the other hand, a recent variant of S4 called DSS showed that restricting the state matrix to be fully diagonal can still preserve the performance of the original model when using a specific initialization based on approximating S4's matrix.
Towards a General Purpose CNN for Long Range Dependencies in $N$D
The use of Convolutional Neural Networks (CNNs) is widespread in Deep Learning due to a range of desirable model properties which result in an efficient and effective machine learning framework.
Diagonal State Spaces are as Effective as Structured State Spaces
Modeling long range dependencies in sequential data is a fundamental step towards attaining human-level performance in many modalities such as text, vision, audio and video.
It's Raw! Audio Generation with State-Space Models
SaShiMi yields state-of-the-art performance for unconditional waveform generation in the autoregressive setting.
Efficiently Modeling Long Sequences with Structured State Spaces
A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly on long-range dependencies.
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency.
Ranked #2 on
Sequential Image Classification
on Sequential MNIST
HoroPCA: Hyperbolic Dimensionality Reduction via Horospherical Projections
Given directions, PCA relies on: (1) a parameterization of subspaces spanned by these directions, (2) a method of projection onto subspaces that preserves information in these directions, and (3) an objective to optimize, namely the variance explained by projections.
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers
Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency.
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps.
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems
As the subclass labels are frequently unavailable, models trained using only the coarser-grained class labels often exhibit highly variable performance across different subclasses.
From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering
Recently, Dasgupta reframed HC as a discrete optimization problem by introducing a global cost function measuring the quality of a given tree.
HiPPO: Recurrent Memory with Optimal Polynomial Projections
A central problem in learning from sequential data is representing cumulative history in an incremental fashion as more data is processed.
Ranked #8 on
Sequential Image Classification
on Sequential MNIST
Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
Particularly concerning are models with inconsistent performance on specific subgroups of a class, e. g., exhibiting disparities in skin cancer classification in the presence or absence of a spurious bandage.
Improving the Gating Mechanism of Recurrent Neural Networks
Gating mechanisms are widely used in neural network models, where they allow gradients to backpropagate more easily through depth or time.
Learning Mixed-Curvature Representations in Product Spaces
The quality of the representations achieved by embeddings is determined by how well the geometry of the embedding space matches the structure of the data.
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations
Fast linear transforms are ubiquitous in machine learning, including the discrete Fourier transform, discrete cosine transform, and other structured transformations such as convolutions.
Learning Compressed Transforms with Low Displacement Rank
The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual.
Representation Tradeoffs for Hyperbolic Embeddings
Given a tree, we give a combinatorial construction that embeds the tree in hyperbolic space with arbitrarily low distortion without using optimization.
A Kernel Theory of Modern Data Augmentation
Data augmentation, a technique in which a training set is expanded with class-preserving transformations, is ubiquitous in modern machine learning pipelines.
