Search Results for author:
Found 10 papers, 7 papers with code
LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
To address these challenges, we propose Long Context Pre-training with Restoration Distillation (LongReD), a novel approach designed to mitigate short-text performance degradation through minimizing the distribution discrepancy between the extended and original models.
YuLan-Mini: An Open Data-efficient Language Model
Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved.
LLMBox: A Comprehensive Library for Large Language Models
To facilitate the research on large language models (LLMs), this paper presents a comprehensive and unified library, LLMBox, to ease the development, use, and evaluation of LLMs.
YuLan: An Open-source Large Language Model
This paper presents the development of YuLan, a series of open-source LLMs with $12$ billion parameters.
Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension.
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models
Recently, multiple studies have committed to extending the context length and enhancing the long text modeling capabilities of LLMs.
StructGPT: A General Framework for Large Language Model to Reason over Structured Data
Specially, we propose an \emph{invoking-linearization-generation} procedure to support LLMs in reasoning on the structured data with the help of the external interfaces.
A Survey of Large Language Models
To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size.
A Survey on Long Text Modeling with Transformers
In this paper, we provide an overview of the recent advances on long texts modeling based on Transformer models.
TextBox 2.0: A Text Generation Library with Pre-trained Language Models
To facilitate research on text generation, this paper presents a comprehensive and unified library, TextBox 2. 0, focusing on the use of pre-trained language models (PLMs).
Ranked #1 on
Abstractive Text Summarization
on CNN/Daily Mail
