Search Results for author:
Found 31 papers, 18 papers with code
STAR: Synthesis of Tailored Architectures
In this work, we propose a new approach for the synthesis of tailored architectures (STAR).
State-Free Inference of State-Space Models: The Transfer Function Approach
We approach designing a state-space model for deep learning applications through its dual representation, the transfer function, and uncover a highly efficient sequence parallel inference algorithm that is state-free: unlike other proposed algorithms, state-free inference does not incur any significant memory or computational cost with an increase in state size.
Mechanistic Design and Scaling of Hybrid Architectures
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation.
Zoology: Measuring and Improving Recall in Efficient Language Models
To close the gap between synthetics and real language, we develop a new formalization of the task called multi-query associative recall (MQAR) that better reflects actual language.
Learning Efficient Surrogate Dynamic Models with Graph Spline Networks
While complex simulations of physical systems have been widely used in engineering and scientific computing, lowering their often prohibitive computational requirements has only recently been tackled by deep learning approaches.
Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension?
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models.
Ideal Abstractions for Decision-Focused Learning
We present a methodology for formulating simplifying abstractions in machine learning systems by identifying and harnessing the utility structure of decisions.
Effectively Modeling Time Series with Simple Discrete State Spaces
For expressivity, we propose a new SSM parameterization based on the companion matrix -- a canonical representation for discrete-time processes -- which enables SpaceTime's SSM layers to learn desirable autoregressive processes.
Hyena Hierarchy: Towards Larger Convolutional Language Models
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale.
Ranked #37 on
Language Modelling
on WikiText-103
Deep Latent State Space Models for Time-Series Generation
Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series.
Transform Once: Efficient Operator Learning in Frequency Domain
Instead, this work introduces a blueprint for frequency domain learning through a single transform: transform once (T1).
Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations
We investigate how to leverage the representations produced by Neural Collages in various tasks, including data compression and generation.
Monarch: Expressive Structured Matrices for Efficient and Accurate Training
To address these issues, we propose a class of matrices (Monarch) that is hardware-efficient (they are parameterized as products of two block-diagonal matrices for better hardware utilization) and expressive (they can represent many commonly used transforms).
Neural Solvers for Fast and Accurate Numerical Optimal Control
Synthesizing optimal controllers for dynamical systems often involves solving optimization problems with hard real-time constraints.
Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
This phenomenon, also known as shortcut learning, is emerging as a key limitation of the current generation of machine learning models.
Continuous-Depth Neural Models for Dynamic Graph Prediction
We introduce the framework of continuous-depth graph neural networks (GNNs).
Neural Hybrid Automata: Learning Dynamics with Multiple Modes and Stochastic Transitions
Effective control and prediction of dynamical systems often require appropriate handling of continuous-time and discrete, event-triggered processes.
Learning Stochastic Optimal Policies via Gradient Descent
We systematically develop a learning-based treatment of stochastic optimal control (SOC), relying on direct optimization of parametric control policies.
Optimal Energy Shaping via Neural Approximators
We introduce optimal energy shaping as an enhancement of classical passivity-based control methods.
Neural Ordinary Differential Equations for Intervention Modeling
By interpreting the forward dynamics of the latent representation of neural networks as an ordinary differential equation, Neural Ordinary Differential Equation (Neural ODE) emerged as an effective framework for modeling a system dynamics in the continuous time domain.
TorchDyn: A Neural Differential Equations Library
Continuous-depth learning has recently emerged as a novel perspective on deep learning, improving performance in tasks related to dynamical systems and density estimation.
Hypersolvers: Toward Fast Continuous-Depth Models
The infinite-depth paradigm pioneered by Neural ODEs has launched a renaissance in the search for novel dynamical system-inspired deep learning primitives; however, their utilization in problems of non-trivial size has often proved impossible due to poor computational scalability.
Stable Neural Flows
We introduce a provably stable variant of neural ordinary differential equations (neural ODEs) whose trajectories evolve on an energy functional parametrised by a neural network.
Port-Hamiltonian Gradient Flows
In this paper we present a general framework for continuous--time gradient descent, often referred to as gradient flow.
Neural Ordinary Differential Equation Value Networks for Parametrized Action Spaces
Action spaces equipped with parameter sets are a common occurrence in reinforcement learning applications.
Hierarchical Reinforcement Learning
reinforcement-learning
+2
Dissecting Neural ODEs
Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs).
Graph Neural Ordinary Differential Equations
We introduce the framework of continuous--depth graph neural networks (GNNs).
WATTNet: Learning to Trade FX via Hierarchical Spatio-Temporal Representation of Highly Multivariate Time Series
Finance is a particularly challenging application area for deep learning models due to low noise-to-signal ratio, non-stationarity, and partial observability.
Port-Hamiltonian Approach to Neural Network Training
Neural networks are discrete entities: subdivided into discrete layers and parametrized by weights which are iteratively optimized via difference equations.
