Search Results for author:
Found 9 papers, 6 papers with code
Mechanistic Design and Scaling of Hybrid Architectures
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation.
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
FlashFFTConv uses a matrix decomposition that computes the FFT using matrix multiply units and enables kernel fusion for long sequences, reducing I/O.
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models.
Hyena Hierarchy: Towards Larger Convolutional Language Models
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale.
Ranked #37 on
Language Modelling
on WikiText-103
Simple Hardware-Efficient Long Convolutions for Sequence Modeling
We find that a key requirement to achieving high performance is keeping the convolution kernels smooth.
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
On ImageNet-1k, S4ND exceeds the performance of a Vision Transformer baseline by $1. 5\%$ when training with a $1$D sequence of patches, and matches ConvNeXt when modeling images in $2$D.
Continual Reinforcement Learning with TELLA
Training reinforcement learning agents that continually learn across multiple environments is a challenging problem.
Lifelong Learning Metrics
The DARPA Lifelong Learning Machines (L2M) program seeks to yield advances in artificial intelligence (AI) systems so that they are capable of learning (and improving) continuously, leveraging data on one task to improve performance on another, and doing so in a computationally sustainable way.
OSCAR-Net: Object-centric Scene Graph Attention for Image Attribution
Our key contribution is OSCAR-Net (Object-centric Scene Graph Attention for Image Attribution Network); a robust image hashing model inspired by recent successes of Transformers in the visual domain.
