Search Results for author:
Found 92 papers, 15 papers with code
TPU-KNN: K Nearest Neighbor Search at Peak FLOP/s
This paper presents a novel nearest neighbor search algorithm achieving TPU (Google Tensor Processing Unit) peak performance, outperforming state-of-the-art GPU algorithms with similar level of recall.
ELM: Embedding and Logit Margins for Long-Tail Learning
Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners.
Predicting on the Edge: Identifying Where a Larger Model Does Better
Much effort has been devoted to making large and more accurate models, but relatively little has been put into understanding which examples are benefiting from the added complexity.
Robust Training of Neural Networks Using Scale Invariant Architectures
In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models.
Efficient Training of Retrieval Models using Negative Cache
These models are typically trained by optimizing the model parameters to score relevant positive" pairs higher than the irrelevantnegative" ones.
When in Doubt, Summon the Titans: Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
Leveraging redundancy in attention with Reuse Transformers
Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications in language and vision.
Less data is more: Selecting informative and diverse subsets with balancing constraints
Deep learning has yielded extraordinary results in vision and natural language processing, but this achievement comes at a cost.
Model-Efficient Deep Learning with Kernelized Classification
We investigate the possibility of using the embeddings produced by a lightweight network more effectively with a nonlinear classification layer.
When in Doubt, Summon the Titans: A Framework for Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
In defense of dual-encoders for neural ranking
Transformer-based models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query.
Batch Active Learning at Scale
The ability to train complex and highly effective models often requires an abundance of training data, which can easily become a bottleneck in cost, time, and computational resources.
Teacher's pet: understanding and mitigating biases in distillation
Knowledge distillation is widely used as a means of improving the performance of a relatively simple student model using the predictions from a complex teacher model.
Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation
State-of-the-art transformer models use pairwise dot-product based self-attention, which comes at a computational cost quadratic in the input sequence length.
Scaling Hierarchical Agglomerative Clustering to Billion-sized Datasets
Hierarchical Agglomerative Clustering (HAC) is one of the oldest but still most widely used clustering methods.
Balancing Robustness and Sensitivity using Feature Contrastive Learning
It is generally believed that robust training of extremely large networks is critical to their success in real-world applications.
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all labels into account.
Less is more: Selecting informative and diverse subsets with balancing constraints
Deep learning has yielded extraordinary results in vision and natural language processing, but this achievement comes at a cost.
RankDistil: Knowledge Distillation for Ranking
Knowledge distillation is an approach to improve the performance of a student model by using the knowledge of a complex teacher. Despite its success in several deep learning applications, the study of distillation is mostly confined to classification settings.
On the Reproducibility of Neural Network Predictions
By analyzing the relationship between churn and prediction confidences, we pursue an approach with two components for churn reduction.
Overparameterisation and worst-case generalisation: friend or foe?
Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples, while still generalising to unseen test samples.
Kernelized Classification in Deep Networks
We propose a kernelized classification layer for deep networks.
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Modifying Memories in Transformer Models
In this paper, we propose a new task of \emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}.
Coping with Label Shift via Distributionally Robust Optimisation
The label shift problem refers to the supervised learning setting where the train and test label distributions do not match.
Semantic Label Smoothing for Sequence to Sequence Problems
Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising.
Learning discrete distributions: user vs item-level privacy
If each user has $m$ samples, we show that straightforward applications of Laplace or Gaussian mechanisms require the number of users to be $\mathcal{O}(k/(m\alpha^2) + k/\epsilon\alpha)$ to achieve an $\ell_1$ distance of $\alpha$ between the true and estimated distributions, with the privacy-induced penalty $k/\epsilon\alpha$ independent of the number of samples per user $m$.
Multi-Stage Influence Function
With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task.
Long-tail learning via logit adjustment
Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples.
Ranked #33 on
Long-tail Learning
on ImageNet-LT
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Evaluations and Methods for Explanation through Robustness Analysis
In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis.
Why distillation helps: a statistical perspective
In this paper, we present a statistical perspective on distillation which addresses this question, and provides a novel connection to extreme multiclass retrieval techniques.
Can gradient clipping mitigate label noise?
Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, it controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum.
Doubly-stochastic mining for heterogeneous retrieval
Modern retrieval problems are characterised by training sets with potentially billions of labels, and heterogeneous data distributions across subpopulations (e. g., users of a retrieval system may be from different countries), each of which poses a challenge.
Federated Learning with Only Positive Labels
We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class.
Robust Large-Margin Learning in Hyperbolic Space
In this paper, we present, to our knowledge, the first theoretical guarantees for learning a classifier in hyperbolic rather than Euclidean space.
Does label smoothing mitigate label noise?
Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors.
Ranked #8 on
Learning with noisy labels
on CIFAR-10N-Aggregate
Adaptive Federated Optimization
Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data.
Low-Rank Bottleneck in Multi-head Attention Models
Attention based Transformer architecture has enabled significant advances in the field of natural language processing.
Pre-training Tasks for Embedding-based Large-scale Retrieval
We consider the large-scale query-document retrieval problem: given a query (e. g., a question), return the set of relevant documents (e. g., paragraphs containing the answer) from a large document corpus.
New Loss Functions for Fast Maximum Inner Product Search
In this work, we focus directly on minimizing error in inner product approximation and derive a new class of quantization loss functions.
Are Transformers universal approximators of sequence-to-sequence functions?
In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models.
Why are Adaptive Methods Good for Attention Models?
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across important tasks, such as attention models.
Multilabel reductions: what is my loss optimising?
Multilabel classification is a challenging problem arising in applications ranging from information retrieval to image tagging.
Breaking the Glass Ceiling for Embedding-Based Classifiers for Large Output Spaces
In extreme classification settings, embedding-based neural network models are currently not competitive with sparse linear and tree-based methods in terms of accuracy.
Learning to Learn by Zeroth-Order Oracle
In the learning to learn (L2L) framework, we cast the design of optimization algorithms as a machine learning problem and use deep neural networks to learn the update rules.
LEARNING TO LEARN WITH BETTER CONVERGENCE
We consider the learning to learn problem, where the goal is to leverage deeplearning models to automatically learn (iterative) optimization algorithms for training machine learning models.
Why ADAM Beats SGD for Attention Models
While stochastic gradient descent (SGD) is still the de facto algorithm in deep learning, adaptive methods like Adam have been observed to outperform SGD across important tasks, such as attention models.
Concise Multi-head Attention Models
Attention based Transformer architecture has enabled significant advances in the field of natural language processing.
Online Hierarchical Clustering Approximations
The second algorithm, OHAC, is an online counterpart to offline HAC, which is known to yield a 1/3-approximation to the MW revenue, and produce good quality clusters in practice.
Accelerating Large-Scale Inference with Anisotropic Vector Quantization
Based on the observation that for a given query, the database points that have the largest inner products are more relevant, we develop a family of anisotropic quantization loss functions.
AdaCliP: Adaptive Clipping for Private SGD
Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed.
Sampled Softmax with Random Fourier Features
For the settings where a large number of classes are involved, a common method to speed up training is to sample a subset of classes and utilize an estimate of the loss gradient based on these classes, known as the sampled softmax method.
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection.
On the Convergence of Adam and Beyond
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients.
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Ranked #11 on
Question Answering
on SQuAD1.1 dev
(F1 metric)
Local Orthogonal Decomposition for Maximum Inner Product Search
More specifically, we decompose a residual vector locally into two orthogonal components and perform uniform quantization and multiscale quantization to each component respectively.
Efficient Inner Product Approximation in Hybrid Spaces
In this paper, we propose a technique that approximates the inner product computation in hybrid vectors, leading to substantial speedup in search while maintaining high accuracy.
Escaping Saddle Points with Adaptive Gradient Methods
Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood.
Adaptive Methods for Nonconvex Optimization
In this work, we provide a new analysis of such methods applied to nonconvex stochastic optimization problems, characterizing the effect of increasing minibatch size.
Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
The algorithm achieves an order of magnitude faster inference than the original softmax layer for predicting top-$k$ words in various tasks such as beam search in machine translation or next words prediction.
Stochastic Negative Mining for Learning with Large Output Spaces
Furthermore, we identify a particularly intuitive class of loss functions in the aforementioned family and show that they are amenable to practical implementation in the large output space setting (i. e. computation is possible without evaluating scores of all labels) by developing a technique called Stochastic Negative Mining.
Privacy and Utility Tradeoff in Approximate Differential Privacy
We show that the multiplicative gap of the lower bounds and upper bounds goes to zero in various high privacy regimes, proving the tightness of the lower and upper bounds and thus establishing the optimality of the truncated Laplacian mechanism.
Optimal Noise-Adding Mechanism in Additive Differential Privacy
We derive the optimal $(0, \delta)$-differentially private query-output independent noise-adding mechanism for single real-valued query function under a general cost-minimization framework.
Loss Decomposition for Fast Learning in Large Output Spaces
For problems with large output spaces, evaluation of the loss function and its gradient are expensive, typically taking linear time in the size of the output space.
Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling
Our experiments show that there is indeed additional structure beyond sparsity in the real datasets; our method is able to discover it and exploit it to create excellent reconstructions with fewer measurements (by a factor of 1. 1-3x) compared to the previous state-of-the-art methods.
Extreme Multi-Label Classification
Multi-Label Classification
+1
cpSGD: Communication-efficient and differentially-private distributed SGD
Distributed stochastic gradient descent is an important subroutine in distributed learning.
Nonlinear Online Learning with Adaptive Nyström Approximation
Use of nonlinear feature maps via kernel approximation has led to success in many online learning tasks.
Multiscale Quantization for Fast Similarity Search
We propose a multiscale quantization approach for fast similarity search on large, high-dimensional datasets.
Now Playing: Continuous low-power music recognition
To reduce battery consumption, a small music detector runs continuously on the mobile device's DSP chip and wakes up the main application processor only when it is confident that music is present.
Learning Spread-out Local Feature Descriptors
We propose a simple, yet powerful regularization technique that can be used to significantly improve both the pairwise and triplet losses in learning local feature descriptors.
Efficient Natural Language Response Suggestion for Smart Reply
This paper presents a computationally efficient machine-learned method for natural language response suggestion.
Stochastic Generative Hashing
Learning-based binary hashing has become a powerful paradigm for fast search and retrieval in massive databases.
Distributed Mean Estimation with Limited Communication
Motivated by the need for distributed learning and optimization algorithms with low communication cost, we study communication efficient algorithms for distributed mean estimation.
Orthogonal Random Features
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error.
Spherical Random Features for Polynomial Kernels
Among the commonly used kernels for nonlinear classification are polynomial kernels, for which low approximation error has thus far necessitated explicit feature maps of large dimensionality, especially for higher-order polynomials.
Fast Orthogonal Projection Based on Kronecker Product
We propose a family of structured matrices to speed up orthogonal projections for high-dimensional data commonly seen in computer vision applications.
On Binary Embedding using Circulant Matrices
To address this problem, we propose Circulant Binary Embedding (CBE) which generates binary codes by projecting the data with a circulant matrix.
Binary embeddings with structured hashed projections
We prove several theoretical results showing that projections via various structured matrices followed by nonlinear mappings accurately preserve the angular distance between input high-dimensional vectors.
Structured Transforms for Small-Footprint Deep Learning
We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices.
Learning to Hash for Indexing Big Data - A Survey
Such learning to hash methods exploit information such as data distributions or class labels when optimizing the hash codes or functions.
Quantization based Fast Inner Product Search
We propose a quantization based approach for fast approximate Maximum Inner Product Search (MIPS).
Fast Online Clustering with Randomized Skeleton Sets
To achieve fast clustering, we propose to represent each cluster by a skeleton set which is updated continuously as new data is seen.
Compact Nonlinear Maps and Circulant Extensions
This leads to much more compact maps without hurting the performance.
An exploration of parameter redundancy in deep networks with circulant projections
We explore the redundancy of parameters in deep neural networks by replacing the conventional linear projection in fully-connected layers with the circulant projection.
Discrete Graph Hashing
Hashing has emerged as a popular technique for fast nearest neighbor search in gigantic databases.
Circulant Binary Embedding
To address this problem, we propose Circulant Binary Embedding (CBE) which generates binary codes by projecting the data with a circulant matrix.
On Learning from Label Proportions
Learning from Label Proportions (LLP) is a learning setting, where the training data is provided in groups, or "bags", and only the proportion of each class in each bag is known.
$\propto$SVM for learning with label proportions
We study the problem of learning with label proportions in which the training data is provided in groups and only the proportion of each class in each group is known.
Learning Binary Codes for High-Dimensional Data Using Bilinear Projections
Recent advances in visual recognition indicate that to achieve good retrieval and classification accuracy on largescale datasets like ImageNet, extremely high-dimensional visual descriptors, e. g., Fisher Vectors, are needed.
Angular Quantization-based Binary Codes for Fast Similarity Search
Such data typically arises in a large number of vision and text applications where counts or frequencies are used as features.
Ensemble Nystrom Method
A crucial technique for scaling kernel methods to very large data sets reaching or exceeding millions of instances is based on low-rank approximation of kernel matrices.
