Search Results for author:
Found 9 papers, 2 papers with code
InternLM2 Technical Report
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI).
Ranked #5 on
Long-Context Understanding
on Ada-LEval (BestAnswer)
WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset
To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb.
Balanced Data Sampling for Language Model Training with Clustering
Data plays a fundamental role in the training of Large Language Models (LLMs).
Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
To address the dropped tokens and padding, we propose the Rectify-Router, comprising the Intra-GPU Rectification and the Fill-in Rectification.
InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning
We further explore how to use LEAN to solve math problems and study its performance under the setting of multi-task learning which shows the possibility of using LEAN as a unified platform for solving and proving in math.
Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora
Large language models have demonstrated remarkable potential in various tasks, however, there remains a significant scarcity of open-source models and data for specific domains.
Pre-training for Information Retrieval: Are Hyperlinks Fully Explored?
Hyperlinks, which are commonly used in Web pages, have been leveraged for designing pre-training objectives.
Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding
Our experiments on 13 benchmark datasets across five natural language understanding tasks demonstrate the superiority of our method.
Towards More Effective and Economic Sparsely-Activated Model
To increase the number of activated experts without an increase in computational cost, we propose SAM (Switch and Mixture) routing, an efficient hierarchical routing mechanism that activates multiple experts in a same device (GPU).
