Search Results for author:
Found 15 papers, 12 papers with code
SubGen: Token Generation in Sublinear Time and Memory
In this work, our focus is on developing an efficient compression technique for the KV cache.
HyperAttention: Long-context Attention in Near-Linear Time
Recent work suggests that in the worst-case scenario, quadratic time is necessary unless the entries of the attention matrix are bounded or the matrix has low stable rank.
KDEformer: Accelerating Transformers via Kernel Density Estimation
Dot-product attention mechanism plays a crucial role in modern deep architectures (e. g., Transformer) for sequence modeling, however, na\"ive exact computation of this model incurs quadratic time and memory complexities in sequence length, hindering the training of long-sequence models.
Fast Neural Kernel Embeddings for General Activations
Moreover, most prior works on neural kernels have focused on the ReLU activation, mainly due to its popularity but also due to the difficulty of computing such kernels for general activations.
Scalable MCMC Sampling for Nonsymmetric Determinantal Point Processes
In this work, we develop a scalable MCMC sampling algorithm for $k$-NDPPs with low-rank kernels, thus enabling runtime that is sublinear in $n$.
Random Gegenbauer Features for Scalable Kernel Methods
Our proposed GZK family, generalizes the zonal kernels (i. e., dot-product kernels on the unit sphere) by introducing radial factors in their Gegenbauer series expansion, and includes a wide range of ubiquitous kernel functions such as the entirety of dot-product kernels as well as the Gaussian and the recently introduced Neural Tangent kernels.
Scalable Sampling for Nonsymmetric Determinantal Point Processes
However, existing work leaves open the question of scalable NDPP sampling.
Scaling Neural Tangent Kernels via Sketching and Random Features
To accelerate learning with NTK, we design a near input-sparsity time approximation algorithm for NTK, by sketching the polynomial expansions of arc-cosine kernels: our sketch for the convolutional counterpart of NTK (CNTK) can transform any image using a linear runtime in the number of pixels.
Random Features for the Neural Tangent Kernel
We combine random features of the arc-cosine kernels with a sketching-based algorithm which can run in linear with respect to both the number of data points and input dimension.
Scalable Learning and MAP Inference for Nonsymmetric Determinantal Point Processes
Determinantal point processes (DPPs) have attracted significant attention in machine learning for their ability to model subsets drawn from a large item collection.
Polynomial Tensor Sketch for Element-wise Function of Low-Rank Matrix
This paper studies how to sketch element-wise functions of low-rank matrices.
Stochastic Chebyshev Gradient Descent for Spectral Optimization
A large class of machine learning techniques requires the solution of optimization problems involving spectral functions of parametric matrices, e. g. log-determinant and nuclear norm.
Faster Greedy MAP Inference for Determinantal Point Processes
Determinantal point processes (DPPs) are popular probabilistic models that arise in many machine learning tasks, where distributions of diverse sets are characterized by matrix determinants.
Approximating the Spectral Sums of Large-scale Matrices using Chebyshev Approximations
Computation of the trace of a matrix function plays an important role in many scientific computing applications, including applications in machine learning, computational physics (e. g., lattice quantum chromodynamics), network analysis and computational biology (e. g., protein folding), just to name a few application areas.
Data Structures and Algorithms
Large-scale Log-determinant Computation through Stochastic Chebyshev Expansions
Logarithms of determinants of large positive definite matrices appear ubiquitously in machine learning applications including Gaussian graphical and Gaussian process models, partition functions of discrete graphical models, minimum-volume ellipsoids, metric learning and kernel learning.
