Search Results for author:
Found 53 papers, 4 papers with code
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
In particular, this paradigm relies on an SLM to both (1) provide soft labels as additional training supervision, and (2) select a small subset of valuable ("informative" and "hard") training examples.
A Statistical Framework for Data-dependent Retrieval-Augmented Models
We propose a statistical framework to study such models with two components: 1) a {\em retriever} to identify the relevant information out of a large corpus via a data-dependent metric; and 2) a {\em predictor} that consumes the input instances along with the retrieved information to make the final predictions.
Analysis of Plan-based Retrieval for Grounded Text Generation
In text generation, hallucinations refer to the generation of seemingly coherent text that contradicts established knowledge.
Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
Recent research has shown that Transformers with linear attention are capable of in-context learning (ICL) by implementing a linear estimator through gradient descent steps.
Efficient Document Ranking with Learnable Late Interactions
Cross-Encoder (CE) and Dual-Encoder (DE) models are two fundamental approaches for query-document relevance in information retrieval.
Faster Cascades via Speculative Decoding
Both approaches involve interleaving models of different sizes, but via fundamentally distinct mechanisms: cascades employ a deferral rule that invokes the larger model only for "hard" inputs, while speculative decoding uses speculative execution to primarily invoke the larger model in parallel verification mode.
Language Model Cascades: Token-level uncertainty and beyond
While the principles underpinning cascading are well-studied for classification tasks - with deferral based on predicted class uncertainty favored theoretically and practically - a similar understanding is lacking for generative LM tasks.
Mechanics of Next Token Prediction with Self-Attention
}$ We show that training self-attention with gradient descent learns an automaton which generates the next token in two distinct steps: $\textbf{(1)}$ $\textbf{Hard}$ $\textbf{retrieval:}$ Given input sequence, self-attention precisely selects the $\textit{high-priority}$ $\textit{input}$ $\textit{tokens}$ associated with the last input token.
From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation.
DistillSpec: Improving Speculative Decoding via Knowledge Distillation
Finally, in practical scenarios with models of varying sizes, first using distillation to boost the performance of the target model and then applying DistillSpec to train a well-aligned draft model can reduce decoding latency by 6-10x with minimal performance drop, compared to standard decoding without distillation.
What do larger image classifiers memorise?
The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to perfectly fit (memorise) completely random labels.
Think before you speak: Training Language Models With Pause Tokens
Language models generate responses by producing a series of tokens in immediate succession: the $(K+1)^{th}$ token is an outcome of manipulating $K$ hidden vectors per layer, one vector per preceding token.
When Does Confidence-Based Cascade Deferral Suffice?
Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn.
On the Role of Attention in Prompt-tuning
Despite its success in LLMs, there is limited theoretical understanding of the power of prompt-tuning and the role of the attention mechanism in prompting.
Supervision Complexity and its Role in Knowledge Distillation
Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps.
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR).
Large Language Models with Controllable Working Memory
By contrast, when the context is irrelevant to the task, the model should ignore it and fall back on its internal knowledge.
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse.
Generalization Properties of Retrieval-based Models
The second class of retrieval-based approaches we explore learns a global model using kernel methods to directly map an input instance and retrieved examples to a prediction, without explicitly solving a local learning task.
A Fourier Approach to Mixture Learning
Regev and Vijayaraghavan (2017) showed that with $\Delta = \Omega(\sqrt{\log k})$ separation, the means can be learned using $\mathrm{poly}(k, d)$ samples, whereas super-polynomially many samples are required if $\Delta = o(\sqrt{\log k})$ and $d = \Omega(\log k)$.
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data.
ELM: Embedding and Logit Margins for Long-Tail Learning
Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners.
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating server.
When in Doubt, Summon the Titans: Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
When in Doubt, Summon the Titans: A Framework for Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
An Empirical Study of Pre-trained Models on Out-of-distribution Generalization
Concretely, we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance.
In defense of dual-encoders for neural ranking
Transformer-based models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query.
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all labels into account.
Distilling Double Descent
Distillation is the technique of training a "student" model based on examples that are labeled by a separate "teacher" model, which itself is trained on a labeled dataset.
On the Reproducibility of Neural Network Predictions
By analyzing the relationship between churn and prediction confidences, we pursue an approach with two components for churn reduction.
Overparameterisation and worst-case generalisation: friend or foe?
Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples, while still generalising to unseen test samples.
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Modifying Memories in Transformer Models
In this paper, we propose a new task of \emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}.
Long-tail learning via logit adjustment
Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples.
Ranked #53 on
Long-tail Learning
on ImageNet-LT
Adversarial robustness via robust low rank representations
Adversarial robustness measures the susceptibility of a classifier to imperceptible perturbations made to the inputs at test time.
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Why distillation helps: a statistical perspective
In this paper, we present a statistical perspective on distillation which addresses this question, and provides a novel connection to extreme multiclass retrieval techniques.
Can gradient clipping mitigate label noise?
Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, it controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum.
Doubly-stochastic mining for heterogeneous retrieval
Modern retrieval problems are characterised by training sets with potentially billions of labels, and heterogeneous data distributions across subpopulations (e. g., users of a retrieval system may be from different countries), each of which poses a challenge.
Federated Learning with Only Positive Labels
We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class.
Robust Large-Margin Learning in Hyperbolic Space
In this paper, we present, to our knowledge, the first theoretical guarantees for learning a classifier in hyperbolic rather than Euclidean space.
Reliable Distributed Clustering with Redundant Data Assignment
In this paper, we present distributed generalized clustering algorithms that can handle large scale data across multiple machines in spite of straggling or unreliable machines.
Low-Rank Bottleneck in Multi-head Attention Models
Attention based Transformer architecture has enabled significant advances in the field of natural language processing.
Are Transformers universal approximators of sequence-to-sequence functions?
In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models.
Multilabel reductions: what is my loss optimising?
Multilabel classification is a challenging problem arising in applications ranging from information retrieval to image tagging.
Concise Multi-head Attention Models
Attention based Transformer architecture has enabled significant advances in the field of natural language processing.
Learning Network Parameters in the ReLU Model
Rectified linear units, or ReLUs, have become a preferred activation function for artificial neural networks.
Sampled Softmax with Random Fourier Features
For the settings where a large number of classes are involved, a common method to speed up training is to sample a subset of classes and utilize an estimate of the loss gradient based on these classes, known as the sampled softmax method.
Robust Gradient Descent via Moment Encoding with LDPC Codes
We, instead, propose to encode the second-moment of the data with a low density parity-check (LDPC) code.
Representation Learning and Recovery in the ReLU Model
Given a set of observation vectors $\mathbf{y}^i \in \mathbb{R}^d, i =1, 2, \dots , n$, we aim to recover $d\times k$ matrix $A$ and the latent vectors $\{\mathbf{c}^i\} \subset \mathbb{R}^k$ under the model $\mathbf{y}^i = \mathrm{ReLU}(A\mathbf{c}^i +\mathbf{b})$, where $\mathbf{b}\in \mathbb{R}^d$ is a random bias.
Lifting high-dimensional nonlinear models with Gaussian regressors
Unfortunately, both least-squares and the Lasso fail to recover $\mathbf{x}_0$ when $\mu_\ell=0$.
Associative Memory using Dictionary Learning and Expander Decoding
Designing an associative memory requires addressing two main tasks: 1) learning phase: given a dataset, learn a concise representation of the dataset in the form of a graphical model (or a neural network), 2) recall phase: given a noisy version of a message vector from the dataset, output the correct message vector via a neurally feasible algorithm over the network learnt during the learning phase.
Associative Memory via a Sparse Recovery Model
An associative memory is a structure learned from a dataset $\mathcal{M}$ of vectors (signals) in a way such that, given a noisy version of one of the vectors as input, the nearest valid vector from $\mathcal{M}$ (nearest neighbor) is provided as output, preferably via a fast iterative algorithm.
