Search Results for author:
Found 7 papers, 2 papers with code
What Matters In The Structured Pruning of Generative Language Models?
Auto-regressive large language models such as GPT-3 require enormous computational resources to use.
Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning
This paper further investigates the necessity of disentangled representation in downstream applications.
Convergence of End-to-End Training in Deep Unsupervised Contrastive Learning
Unsupervised contrastive learning has gained increasing attention in the latest research and has proven to be a powerful method for learning representations from unlabeled data.
Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning
We present an underlying principle called $\textbf{feature decoupling}$ to explain the effects of augmentations, where we theoretically characterize how augmentations can reduce the correlations of dense features between positive samples while keeping the correlations of sparse features intact, thereby forcing the neural networks to learn from the self-supervision of sparse features.
Improving Multi-Modal Learning with Uni-Modal Teachers
We name this problem Modality Failure, and hypothesize that the imbalance of modalities and the implicit bias of common objectives in fusion method prevent encoders of each modality from sufficient feature learning.
Ranked #60 on
Semantic Segmentation
on NYU Depth v2
The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning
The substitution effect happens when learning the stronger features in some neurons can substitute for learning these features in other neurons through updating the prediction head.
Transformers Provably Learn Feature-Position Correlations in Masked Image Modeling
Masked image modeling (MIM), which predicts randomly masked patches from unmasked ones, has emerged as a promising approach in self-supervised vision pretraining.
