Long Context RAG Performance of Large Language Models

no code implementations • 5 Nov 2024 • Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, Michael Carbin

Can these new long context models improve RAG performance?

RAG

LoRA Learns Less and Forgets Less

1 code implementation • 15 May 2024 • Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, John P. Cunningham

In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics.

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

1 code implementation • NeurIPS 2023 • Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, Jonathan Frankle

Here, we introduce MosaicBERT, a BERT-style encoder architecture and training recipe that is empirically optimized for fast pretraining.

Language Modelling Masked Language Modeling

LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms

no code implementations • 22 Nov 2023 • Aditi Jha, Sam Havens, Jeremey Dohmann, Alex Trott, Jacob Portes

We find that subsets of 1k-6k instruction finetuning samples are sufficient to achieve good performance on both (1) traditional NLP benchmarks and (2) model-based evaluation.

Instruction Following