Search Results for author:
Found 72 papers, 23 papers with code
SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment.
Scalable Neural Network Kernels
We introduce the concept of scalable neural network kernels (SNNKs), the replacements of regular feedforward layers (FFLs), capable of approximating the latter, but with favorable computational properties.
Universal Graph Random Features
This includes many of the most popular examples of kernels defined on the nodes of a graph.
Repelling Random Walks
We present a novel quasi-Monte Carlo mechanism to improve graph-based sampling, coined repelling random walks.
Augmenting conformers with structured state-space sequence models for online speech recognition
Online speech recognition, where the model only accesses context to the left, is an important and challenging use case for ASR systems.
Robotic Table Tennis: A Case Study into a High Speed Learning System
We present a deep-dive into a real-world robotic learning system that, in previous work, was shown to be capable of hundreds of table tennis rallies with a human and has the ability to precisely return the ball to desired targets.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
Taming graph kernels with random features
We also introduce a (still unbiased) quasi Monte Carlo variant of GRFs, q-GRFs, relying on the so-called reinforced random walks, that might be used to optimize the variance of GRFs.
Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR
Conformer models maintain a large number of internal states, the vast majority of which are associated with self-attention layers.
Efficient Graph Field Integrators Meet Point Clouds
We present two new classes of algorithms for efficient field integration on graphs encoding point clouds.
FAVOR#: Sharp Attention Kernel Approximations via New Classes of Positive Random Features
The problem of efficient approximation of a linear operator induced by the Gaussian or softmax kernel is often addressed using random features (RFs) which yield an unbiased approximation of the operator's result.
Simplex Random Features
We present Simplex Random Features (SimRFs), a new random feature (RF) mechanism for unbiased approximation of the softmax and Gaussian kernels by geometrical correlation of random projection vectors.
Karyotype AI for Precision Oncology
These individual chromosomes were used to train and assess deep learning models for classifying the 24 human chromosomes and identifying chromosomal aberrations.
Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation
Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e. g., in cluttered home environments or in human-occupied public spaces.
Multiple View Performers for Shape Completion
We propose the Multiple View Performer (MVP) - a new architecture for 3D shape completion from a series of temporally sequential views.
Implicit Two-Tower Policies
We present a new class of structured reinforcement learning policy-architectures, Implicit Two-Tower (ITT) policies, where the actions are chosen based on the attention scores of their learnable latent representations with those of the input states.
Chefs' Random Tables: Non-Trigonometric Random Features
We introduce chefs' random tables (CRTs), a new class of non-trigonometric random features (RFs) to approximate Gaussian and softmax kernels.
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Large pretrained (e. g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on.
Ranked #21 on
Video Retrieval
on MSR-VTT-1kA
(video-to-text R@1 metric)
PolyViT: Co-training Vision Transformers on Images, Videos and Audio
Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters?
Hybrid Random Features
We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) that automatically adapt the quality of kernel estimation to provide most accurate approximation in the defined regions of interest.
From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers
In this paper we provide, to the best of our knowledge, the first comprehensive approach for incorporating various masking mechanisms into Transformers architectures in a scalable way.
On the Expressive Power of Self-Attention Matrices
Our proof is constructive, enabling us to propose an algorithm for finding adaptive inputs and fixed self-attention parameters in order to approximate a given matrix.
Debiasing a First-order Heuristic for Approximate Bi-level Optimization
Approximate bi-level optimization (ABLO) consists of (outer-level) optimization problems, involving numerical (inner-level) optimization loops.
ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
In this paper, we approach the problem of optimizing blackbox functions over large hybrid search spaces consisting of both combinatorial and continuous parameters.
MLGO: a Machine Learning Guided Compiler Optimizations Framework
Leveraging machine-learning (ML) techniques for compiler optimizations has been widely studied and explored in academia.
Sub-Linear Memory: How to Make Performers SLiM
Recent works proposed various linear self-attention mechanisms, scaling only as $O(L)$ for serial computation.
Rethinking Attention with Performers
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness.
Ranked #7 on
Offline RL
on D4RL
Towards Tractable Optimism in Model-Based Reinforcement Learning
The principle of optimism in the face of uncertainty is prevalent throughout sequential decision making problems such as multi-armed bandits and reinforcement learning (RL).
An Ode to an ODE
We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d).
Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
Off-policy learning algorithms have been known to be sensitive to the choice of hyper-parameters.
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed.
UFO-BLO: Unbiased First-Order Bilevel Optimization
Bilevel optimization (BLO) is a popular approach with many applications including hyperparameter optimization, neural architecture search, adversarial robustness and model-agnostic meta-learning.
Demystifying Orthogonal Monte Carlo and Beyond
Orthogonal Monte Carlo (OMC) is a very effective sampling algorithm imposing structural geometric conditions (orthogonality) on samples for variance reduction.
Time Dependence in Non-Autonomous Neural ODEs
Neural Ordinary Differential Equations (ODEs) are elegant reinterpretations of deep networks where continuous time can replace the discrete notion of depth, ODE solvers perform forward propagation, and the adjoint method enables efficient, constant memory backpropagation.
CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices
We introduce an efficient approach for optimization over orthogonal groups on highly parallel computation units such as GPUs or TPUs.
Robotic Table Tennis with Model-Free Reinforcement Learning
We propose a model-free algorithm for learning efficient policies capable of returning table tennis balls by controlling robot joints at a rate of 100Hz.
Stochastic Flows and Geometric Optimization on the Orthogonal Group
We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group $O(d)$ and naturally reductive homogeneous manifolds obtained from the action of the rotation group $SO(d)$.
Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
Learning adaptable policies is crucial for robots to operate autonomously in our complex and quickly changing world.
Ready Policy One: World Building Through Active Learning
Model-Based Reinforcement Learning (MBRL) offers a promising direction for sample efficient learning, often achieving state of the art results for continuous control tasks.
Effective Diversity in Population Based Reinforcement Learning
Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment.
ES-MAML: Simple Hessian-Free Meta Learning
We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES).
Behavior-Guided Reinforcement Learning
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space.
Reinforcement Learning with Chromatic Networks
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.
Reinforcement Learning with Chromatic Networks for Compact Architecture Search
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.
Learning to Score Behaviors for Guided Policy Optimization
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space.
Linear interpolation gives better gradients than Gaussian smoothing in derivative-free optimization
We then demonstrate via rigorous analysis of the variance and by numerical comparisons on reinforcement learning tasks that the Gaussian sampling method used in [Salimans et al. 2016] is significantly inferior to the orthogonal sampling used in [Choromaski et al. 2018] as well as more general interpolation methods.
Structured Monte Carlo Sampling for Nonisotropic Distributions via Determinantal Point Processes
We propose a new class of structured methods for Monte Carlo (MC) sampling, called DPPMC, designed for high-dimensional nonisotropic distributions where samples are correlated to reduce the variance of the estimator via determinantal point processes.
Variance Reduction for Evolution Strategies via Structured Control Variates
Evolution Strategies (ES) are a powerful class of blackbox optimization techniques that recently became a competitive alternative to state-of-the-art policy gradient (PG) algorithms for reinforcement learning (RL).
A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization
To this end, we use the results in [Berahas et al., 2019] and show how each method can satisfy the sufficient conditions, possibly only with some sufficiently large probability at each iteration, as happens to be the case with Gaussian smoothing and smoothing on a sphere.
Optimization and Control
Orthogonal Estimation of Wasserstein Distances
Wasserstein distances are increasingly used in a wide variety of applications in machine learning.
From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization
ASEBO adapts to the geometry of the function and learns optimal sets of sensing directions, which are used to probe it, on-the-fly.
Provably Robust Blackbox Optimization for Reinforcement Learning
Interest in derivative-free optimization (DFO) and "evolutionary strategies" (ES) has recently surged in the Reinforcement Learning (RL) community, with growing evidence that they can match state of the art methods for policy optimization problems in Robotics.
Structured Evolution with Compact Architectures for Scalable Policy Optimization
We present a new method of blackbox optimization via gradient approximation with the use of structured random orthogonal matrices, providing more accurate estimators than baselines and with provable theoretical guarantees.
On the Needs for Rotations in Hypercubic Quantization Hashing
The aim of this paper is to endow the well-known family of hypercubic quantization hashing methods with theoretical guarantees.
Initialization matters: Orthogonal Predictive State Recurrent Neural Networks
In this paper, we extend the theory of ORFs to Kernel Ridge Regression and show that ORFs can be used to obtain Orthogonal PSRNNs (OPSRNNs), which are smaller and faster than PSRNNs.
Manifold Regularization for Kernelized LSTD
Policy evaluation or value function or Q-function approximation is a key procedure in reinforcement learning (RL).
Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
This eliminates the need for human engineers to anticipate what is important in an image and foresee all the necessary rules for safe driving.
Graph sketching-based Space-efficient Data Clustering
In this paper, we address the problem of recovering arbitrary-shaped data clusters from datasets while facing \emph{high space constraints}, as this is for instance the case in many real-world applications when analysis algorithms are directly deployed on resources-limited mobile devices collecting the data.
The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings
We examine a class of embeddings based on structured random matrices with orthogonal rows which can be applied in many machine learning applications including dimensionality reduction and kernel approximation.
VisualBackProp: efficient visualization of CNNs
We furthermore justify our approach with theoretical arguments and theoretically confirm that the proposed method identifies sets of input pixels, rather than individual pixels, that collaboratively contribute to the prediction.
Orthogonal Random Features
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error.
Structured adaptive and random spinners for fast machine learning computations
We consider an efficient computational framework for speeding up several machine learning algorithms with almost no loss of accuracy.
Recycling Randomness with Structure for Sublinear time Kernel Expansions
We propose a scheme for recycling Gaussian random vectors into structured matrices to approximate various kernel functions in sublinear time via random embeddings.
TripleSpin - a generic compact paradigm for fast machine learning computations
In particular, as a byproduct of the presented techniques and by using relatively new Berry-Esseen-type CLT for random vectors, we give the first theoretical guarantees for one of the most efficient existing LSH algorithms based on the $\textbf{HD}_{3}\textbf{HD}_{2}\textbf{HD}_{1}$ structured matrix ("Practical and Optimal LSH for Angular Distance").
On the boosting ability of top-down decision tree learning algorithm for multiclass classification
We analyze the performance of the top-down multiclass classification algorithm for decision tree learning called LOMtree, recently proposed in the literature Choromanska and Langford (2014) for solving efficiently classification problems with very large number of classes.
Fast nonlinear embeddings via structured matrices
Our framework covers as special cases already known structured approaches such as the Fast Johnson-Lindenstrauss Transform, but is much more general since it can be applied also to highly nonlinear embeddings.
Binary embeddings with structured hashed projections
We prove several theoretical results showing that projections via various structured matrices followed by nonlinear mappings accurately preserve the angular distance between input high-dimensional vectors.
Quantization based Fast Inner Product Search
We propose a quantization based approach for fast approximate Maximum Inner Product Search (MIPS).
Fast Online Clustering with Randomized Skeleton Sets
To achieve fast clustering, we propose to represent each cluster by a skeleton set which is updated continuously as new data is seen.
Differentially- and non-differentially-private random decision trees
We consider supervised learning with random decision trees, where the tree construction is completely random.
Notes on using Determinantal Point Processes for Clustering with Applications to Text Clustering
In this paper, we compare three initialization schemes for the KMEANS clustering algorithm: 1) random initialization (KMEANSRAND), 2) KMEANS++, and 3) KMEANSD++.
On Learning from Label Proportions
Learning from Label Proportions (LLP) is a learning setting, where the training data is provided in groups, or "bags", and only the proportion of each class in each bag is known.
