Search Results for author:
Found 5 papers, 1 papers with code
Investigating the Benefits of Projection Head for Representation Learning
An effective technique for obtaining high-quality representations is adding a projection head on top of the encoder during training, then discarding it and using the pre-projection representations.
Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent
Differentially private stochastic gradient descent (DP-SGD) is known to have poorer training and test performance on large neural networks, compared to ordinary stochastic gradient descent (SGD).
Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias
In this work, we provide the first theoretical analysis of the effect of simplicity bias on learning spurious correlations.
Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression
However, supervised CL is prone to collapsing representations of subclasses within a class by not capturing all their features, and unsupervised CL may suppress harder class-relevant features by focusing on learning easy class-irrelevant features; both significantly compromise representation quality.
Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability
We introduce Brain-Inspired Modular Training (BIMT), a method for making neural networks more modular and interpretable.
