Search Results for author:
Found 10 papers, 2 papers with code
Building A Corporate Corpus For Threads Constitution
In this paper we describe the process of build-ing a corporate corpus that will be used as a ref-erence for modelling and computing threadsfrom conversations generated using commu-nication and collaboration tools.
Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck
In this paper, we find that such saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution.
On the Scaling Laws of Geographical Representation in Language Models
Language models have long been shown to embed geographical information in their hidden representations.
Anisotropy Is Inherent to Self-Attention in Transformers
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers.
Headless Language Models: Learning without Predicting with Contrastive Weight Tying
Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies.
Is Anisotropy Inherent to Transformers?
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers.
MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling
The resulting system offers a trade-off between the expressiveness of byte-level models and the speed of models trained using subword tokenization.
Clustering-based Automatic Construction of Legal Entity Knowledge Base from Contracts
In contract analysis and contract automation, a knowledge base (KB) of legal entities is fundamental for performing tasks such as contract verification, contract generation and contract analytic.
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases
Progress in sentence simplification has been hindered by a lack of labeled parallel simplification data, particularly in languages other than English.
Ranked #2 on
Text Simplification
on ASSET
Controllable Sentence Simplification
Text simplification aims at making a text easier to read and understand by simplifying grammar and structure while keeping the underlying information identical.
Ranked #3 on
Text Simplification
on ASSET
