Search Results for author:
Found 6 papers, 3 papers with code
Effective Long-Context Scaling of Foundation Models
We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
Llama 2: Open Foundation and Fine-Tuned Chat Models
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
Ranked #2 on
Question Answering
on PubChemQA
HaSa: Hardness and Structure-Aware Contrastive Knowledge Graph Embedding
We consider a contrastive learning approach to knowledge graph embedding (KGE) via InfoNCE.
A Theory on Adam Instability in Large-Scale Machine Learning
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models.
Over-parametrization via Lifting for Low-rank Matrix Sensing: Conversion of Spurious Solutions to Strict Saddle Points
This paper studies the role of over-parametrization in solving non-convex optimization problems.
When Does MAML Objective Have Benign Landscape?
The paper studies the complexity of the optimization problem behind the Model-Agnostic Meta-Learning (MAML) algorithm.
