Search Results for author:
Found 23 papers, 10 papers with code
Fast Tree-Field Integrators: From Low Displacement Rank to Topological Transformers
We present a new class of fast polylog-linear algorithms based on the theory of structured matrices (in particular low displacement rank) for integrating tensor fields defined on weighted trees.
Hot PATE: Private Aggregation of Distributions for Diverse Task
Moreover, the knowledge of models is often encapsulated in the response distribution itself and preserving this diversity is critical for fluid and effective knowledge transfer from teachers to student.
SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment.
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs).
Efficient Graph Field Integrators Meet Point Clouds
We present two new classes of algorithms for efficient field integration on graphs encoding point clouds.
FAVOR#: Sharp Attention Kernel Approximations via New Classes of Positive Random Features
The problem of efficient approximation of a linear operator induced by the Gaussian or softmax kernel is often addressed using random features (RFs) which yield an unbiased approximation of the operator's result.
Chefs' Random Tables: Non-Trigonometric Random Features
We introduce chefs' random tables (CRTs), a new class of non-trigonometric random features (RFs) to approximate Gaussian and softmax kernels.
From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers
In this paper we provide, to the best of our knowledge, the first comprehensive approach for incorporating various masking mechanisms into Transformers architectures in a scalable way.
ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
In this paper, we approach the problem of optimizing blackbox functions over large hybrid search spaces consisting of both combinatorial and continuous parameters.
Differentially Private Weighted Sampling
A weighted sample of keys by (a function of) frequency is a highly versatile summary that provides a sparse set of representative keys and supports approximate evaluations of query statistics.
Rethinking Attention with Performers
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness.
Ranked #9 on
D4RL
on D4RL
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed.
Stochastic Flows and Geometric Optimization on the Orthogonal Group
We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group $O(d)$ and naturally reductive homogeneous manifolds obtained from the action of the rotation group $SO(d)$.
Tight Dimensionality Reduction for Sketching Low Degree Polynomial Kernels
We give a new analysis of this sketch, providing nearly optimal bounds.
Reinforcement Learning with Chromatic Networks
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.
Reinforcement Learning with Chromatic Networks for Compact Architecture Search
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.
Matrix-Free Preconditioning in Online Learning
We provide an online convex optimization algorithm with regret that interpolates between the regret of an algorithm using an optimal preconditioning matrix and one using a diagonal preconditioning matrix.
Orthogonal Estimation of Wasserstein Distances
Wasserstein distances are increasingly used in a wide variety of applications in machine learning.
Geometrically Coupled Monte Carlo Sampling
Monte Carlo sampling in high-dimensional, low-sample settings is important in many machine learning tasks.
Structured adaptive and random spinners for fast machine learning computations
We consider an efficient computational framework for speeding up several machine learning algorithms with almost no loss of accuracy.
TripleSpin - a generic compact paradigm for fast machine learning computations
In particular, as a byproduct of the presented techniques and by using relatively new Berry-Esseen-type CLT for random vectors, we give the first theoretical guarantees for one of the most efficient existing LSH algorithms based on the $\textbf{HD}_{3}\textbf{HD}_{2}\textbf{HD}_{1}$ structured matrix ("Practical and Optimal LSH for Angular Distance").
Fastfood: Approximate Kernel Expansions in Loglinear Time
These improvements, especially in terms of memory usage, make kernel methods more practical for applications that have large training sets and/or require real-time prediction.
