Search Results for author:
Found 6 papers, 0 papers with code
Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck
In this paper, we find that such saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution.
On the Scaling Laws of Geographical Representation in Language Models
Language models have long been shown to embed geographical information in their hidden representations.
Anisotropy Is Inherent to Self-Attention in Transformers
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers.
Headless Language Models: Learning without Predicting with Contrastive Weight Tying
Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies.
Is Anisotropy Inherent to Transformers?
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers.
MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling
The resulting system offers a trade-off between the expressiveness of byte-level models and the speed of models trained using subword tokenization.
