Search Results for author:
Found 45 papers, 14 papers with code
ResMem: Learn what you can and memorize the rest
The impressive generalization performance of modern neural networks is attributed in part to their ability to implicitly memorize complex training patterns.
On student-teacher deviations in distillation: does it pay to disobey?
Knowledge distillation has been widely-used to improve the performance of a "student" network by hoping to mimic soft probabilities of a "teacher" network.
Learning to reject meets OOD detection: Are all abstentions created equal?
In this paper, we formally relate these problems, and show how they may be jointly solved.
Supervision Complexity and its Role in Knowledge Distillation
Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps.
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Our distillation approach is theoretically justified and applies to both dual encoder (DE) and cross-encoder (CE) models.
When does mixup promote local linearity in learned representations?
Towards this, we study two questions: (1) how does the Mixup loss that enforces linearity in the \emph{last} network layer propagate the linearity to the \emph{earlier} layers?
Robust Distillation for Worst-class Performance
We show empirically that our robust distillation techniques not only achieve better worst-class performance, but also lead to Pareto improvement in the tradeoff between overall performance and worst-class performance compared to other baseline methods.
ELM: Embedding and Logit Margins for Long-Tail Learning
Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners.
When in Doubt, Summon the Titans: Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
When in Doubt, Summon the Titans: A Framework for Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
In defense of dual-encoders for neural ranking
Transformer-based models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query.
Training Over-parameterized Models with Non-decomposable Objectives
Many modern machine learning applications come with complex and nuanced design goals such as minimizing the worst-case error, satisfying a given precision or recall target, or enforcing group-fairness constraints.
Teacher's pet: understanding and mitigating biases in distillation
Knowledge distillation is widely used as a means of improving the performance of a relatively simple student model using the predictions from a complex teacher model.
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all labels into account.
Interval-censored Hawkes processes
We propose the multi-impulse exogenous function - for when the exogenous events are observed as event time - and the latent homogeneous Poisson process exogenous function - for when the exogenous events are presented as interval-censored volumes.
RankDistil: Knowledge Distillation for Ranking
Knowledge distillation is an approach to improve the performance of a student model by using the knowledge of a complex teacher. Despite its success in several deep learning applications, the study of distillation is mostly confined to classification settings.
Distilling Double Descent
Distillation is the technique of training a "student" model based on examples that are labeled by a separate "teacher" model, which itself is trained on a labeled dataset.
Overparameterisation and worst-case generalisation: friend or foe?
Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples, while still generalising to unseen test samples.
Semantic Label Smoothing for Sequence to Sequence Problems
Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising.
SupMMD: A Sentence Importance Model for Extractive Summarization using Maximum Mean Discrepancy
Most work on multi-document summarization has focused on generic summarization of information present in each individual document set.
Long-tail learning via logit adjustment
Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples.
Ranked #41 on
Long-tail Learning
on ImageNet-LT
Why distillation helps: a statistical perspective
In this paper, we present a statistical perspective on distillation which addresses this question, and provides a novel connection to extreme multiclass retrieval techniques.
Can gradient clipping mitigate label noise?
Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, it controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum.
Doubly-stochastic mining for heterogeneous retrieval
Modern retrieval problems are characterised by training sets with potentially billions of labels, and heterogeneous data distributions across subpopulations (e. g., users of a retrieval system may be from different countries), each of which poses a challenge.
Federated Learning with Only Positive Labels
We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class.
Does label smoothing mitigate label noise?
Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors.
Ranked #10 on
Learning with noisy labels
on CIFAR-10N-Random3
Supervised Learning: No Loss No Cry
In detail, we cast {\sc SLIsotron} as learning a loss from a family of composite square losses.
Online Hierarchical Clustering Approximations
The second algorithm, OHAC, is an online counterpart to offline HAC, which is known to yield a 1/3-approximation to the MW revenue, and produce good quality clusters in practice.
Noise-tolerant fair classification
We finally show that our procedure is empirically effective on two case-studies involving sensitive feature censoring.
Fairness risk measures
In this paper, we propose a new definition of fairness that generalises some existing proposals, while allowing for generic sensitive features and resulting in a convex objective.
Cold-start Playlist Recommendation with Multitask Learning
Playlist recommendation involves producing a set of songs that a user might enjoy.
Comparative Document Summarisation via Classification
This paper considers extractive summarisation in a comparative setting: given two or more document groups (e. g., separated by publication time), the goal is to select a small number of documents that are representative of each group, and also maximally distinguishable from other groups.
Complementary-Label Learning for Arbitrary Losses and Models
In contrast to the standard classification paradigm where the true class is given to each training pattern, complementary-label learning only uses training patterns each equipped with a complementary label, which only specifies one of the classes that the pattern does not belong to.
Ranked #21 on
Image Classification
on Kuzushiji-MNIST
On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data
In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM.
Monge blunts Bayes: Hardness Results for Adversarial Training
A key feature of our result is that it holds for all proper losses, and for a popular subset of these, the optimisation of this central measure appears to be independent of the loss.
Anomaly Detection using One-Class Neural Networks
We propose a one-class neural network (OC-NN) model to detect anomalies in complex data sets.
Revisiting revisits in trajectory recommendation
Trajectory recommendation is the problem of recommending a sequence of places in a city for a tourist to visit.
f-GANs in an Information Geometric Nutshell
In this paper, we unveil a broad class of distributions for which such convergence happens --- namely, deformed exponential families, a wide superset of exponential families --- and show tight connections with the three other key GAN parameters: loss, game and architecture.
The cost of fairness in classification
We study the problem of learning classifiers with a fairness constraint, with three main contributions towards the goal of quantifying the problem's inherent tradeoffs.
Robust, Deep and Inductive Anomaly Detection
PCA is a classical statistical technique whose simplicity and maturity has seen it find widespread use as an anomaly detection technique.
A scaled Bregman theorem with applications
Experiments on each of these domains validate the analyses and suggest that the scaled Bregman theorem might be a worthy addition to the popular handful of Bregman divergence properties that have been pervasive in machine learning.
Learning from Binary Labels with Instance-Dependent Corruption
Suppose we have a sample of instances paired with binary labels corrupted by arbitrary instance- and label-dependent noise.
An Average Classification Algorithm
When working with a high or infinite dimensional kernel, it is imperative for speed of evaluation and storage issues that as few training samples as possible are used in the kernel expansion.
Learning with Symmetric Label Noise: The Importance of Being Unhinged
However, Long and Servedio [2010] proved that under symmetric label noise (SLN), minimisation of any convex potential over a linear function class can result in classification performance equivalent to random guessing.
AutoRec: Autoencoders Meet Collaborative Filtering
This paper proposes AutoRec, a novel autoencoder framework for collaborative filtering (CF).
Ranked #5 on
Recommendation Systems
on MovieLens 1M
