Search Results for author:
Found 15 papers, 6 papers with code
SubGen: Token Generation in Sublinear Time and Memory
In this work, our focus is on developing an efficient compression technique for the KV cache.
HyperAttention: Long-context Attention in Near-Linear Time
Recent work suggests that in the worst-case scenario, quadratic time is necessary unless the entries of the attention matrix are bounded or the matrix has low stable rank.
KDEformer: Accelerating Transformers via Kernel Density Estimation
Dot-product attention mechanism plays a crucial role in modern deep architectures (e. g., Transformer) for sequence modeling, however, na\"ive exact computation of this model incurs quadratic time and memory complexities in sequence length, hindering the training of long-sequence models.
Fast Neural Kernel Embeddings for General Activations
Moreover, most prior works on neural kernels have focused on the ReLU activation, mainly due to its popularity but also due to the difficulty of computing such kernels for general activations.
Leverage Score Sampling for Tensor Product Matrices in Input Sparsity Time
We propose an input sparsity time sampling algorithm that can spectrally approximate the Gram matrix corresponding to the $q$-fold column-wise tensor product of $q$ matrices using a nearly optimal number of samples, improving upon all previously known methods by poly$(q)$ factors.
Random Gegenbauer Features for Scalable Kernel Methods
Our proposed GZK family, generalizes the zonal kernels (i. e., dot-product kernels on the unit sphere) by introducing radial factors in their Gegenbauer series expansion, and includes a wide range of ubiquitous kernel functions such as the entirety of dot-product kernels as well as the Gaussian and the recently introduced Neural Tangent kernels.
Scaling Neural Tangent Kernels via Sketching and Random Features
To accelerate learning with NTK, we design a near input-sparsity time approximation algorithm for NTK, by sketching the polynomial expansions of arc-cosine kernels: our sketch for the convolutional counterpart of NTK (CNTK) can transform any image using a linear runtime in the number of pixels.
Learning with Neural Tangent Kernels in Near Input Sparsity Time
The Neural Tangent Kernel (NTK) characterizes the behavior of infinitely wide neural nets trained under least squares loss by gradient descent.
Scaling up Kernel Ridge Regression via Locality Sensitive Hashing
Random binning features, introduced in the seminal paper of Rahimi and Recht (2007), are an efficient method for approximating a kernel matrix using locality sensitive hashing.
Efficiently Learning Fourier Sparse Set Functions
In this paper we consider the problem of efficiently learning set functions that are defined over a ground set of size $n$ and that are sparse (say $k$-sparse) in the Fourier domain.
Oblivious Sketching of High-Degree Polynomial Kernels
Oblivious sketching has emerged as a powerful approach to speeding up numerical linear algebra over the past decade, but our understanding of oblivious sketching solutions for kernel matrices has remained quite limited, suffering from the aforementioned exponential dependence on input parameters.
Data Structures and Algorithms
A Universal Sampling Method for Reconstructing Signals with Simple Fourier Transforms
We formalize this intuition by showing that, roughly, a continuous signal from a given class can be approximately reconstructed using a number of samples proportional to the *statistical dimension* of the allowed power spectrum of that class.
Beyond $1/2$-Approximation for Submodular Maximization on Massive Data Streams
It is the first low-memory, single-pass algorithm that improves the factor $0. 5$, under the natural assumption that elements arrive in a random order.
Beyond 1/2-Approximation for Submodular Maximization on Massive Data Streams
It is the first low-memory, singlepass algorithm that improves the factor 0. 5, under the natural assumption that elements arrive in a random order.
Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guarantees
Qualitatively, our results are twofold: on the one hand, we show that random Fourier feature approximation can provably speed up kernel ridge regression under reasonable assumptions.
