Search Results for author:
Found 42 papers, 21 papers with code
Expected Validation Performance and Estimation of a Random Variable’s Maximum
We find that the two biased estimators lead to the fewest incorrect conclusions, which hints at the importance of minimizing variance and MSE.
Towards Reproducible Machine Learning Research in Natural Language Processing
While recent progress in the field of ML has been significant, the reproducibility of these cutting-edge results is often lacking, with many submissions lacking the necessary information in order to ensure subsequent reproducibility.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Language models have become a critical technology to tackling a wide range of natural language processing tasks, yet many details about how the best-performing language models were developed are not reported.
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
Large language models' (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation.
Paloma: A Benchmark for Evaluating Language Model Fit
We invite submissions to our benchmark and organize results by comparability based on compliance with guidelines such as removal of benchmark contamination from pretraining.
Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
The success of large language models has shifted the evaluation paradigms in natural language processing (NLP).
What's In My Big Data?
We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.
Language Models Hallucinate, but May Excel at Fact Verification
Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs).
The Rise of Open Science: Tracking the Evolution and Perceived Value of Data and Methods Link-Sharing Practices
In recent years, funding agencies and journals increasingly advocate for open science practices (e. g. data and method sharing) to improve the transparency, access, and reproducibility of science.
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
In response, we introduce Pentathlon, a benchmark for holistic and realistic evaluation of model efficiency.
Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research
Many recent improvements in NLP stem from the development and use of large pre-trained language models (PLMs) with billions of parameters.
Reproducibility in NLP: What Have We Learned from the Checklist?
Scientific progress in NLP rests on the reproducibility of researchers' claims.
Evaluating the Social Impact of Generative AI Systems in Systems and Society
We move toward a standard approach in evaluating a generative AI system for any modality, in two overarching categories: what is able to be evaluated in a base system that has no predetermined application and what is able to be evaluated in society.
Stubborn Lexical Bias in Data and Models
Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data.
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
We release Multimodal C4, an augmentation of the popular text-only C4 corpus with images interleaved.
AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models
We also explore weight averaging of adapters trained on the same domain with different hyper-parameters, and show that it preserves the performance of a PLM on new domains while obtaining strong in-domain results.
Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publications
Scholarly text is often laden with jargon, or specialized language that can facilitate efficient in-group communication within fields but hinder understanding for out-groups.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Efficient Methods for Natural Language Processing: A Survey
Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows.
Modeling the Machine Learning Multiverse
Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis.
Measuring the Carbon Intensity of AI in Cloud Instances
By providing unprecedented access to computational resources, cloud computing has enabled rapid growth in technologies such as machine learning, the computational demands of which incur a high energy cost and a commensurate carbon footprint.
Data Governance in the Age of Large-Scale Data-Driven Language Technology
The recent emergence and adoption of Machine Learning technology, and specifically of Large Language Models, has drawn attention to the need for systematic and transparent management of language data.
Staged Training for Transformer Language Models
As an alternative, we consider a staged training setup that begins with a small model and incrementally increases the amount of compute used for training by applying a "growth operator" to increase the model depth and width.
Efficient Hierarchical Domain Adaptation for Pretrained Language Models
The remarkable success of large language models has been driven by dense models trained on massive unlabeled, unstructured corpora.
Can Machines Learn Morality? The Delphi Experiment
As AI systems become increasingly powerful and pervasive, there are growing concerns about machines' morality or a lack thereof.
Expected Validation Performance and Estimation of a Random Variable's Maximum
We find that the two biased estimators lead to the fewest incorrect conclusions, which hints at the importance of minimizing variance and MSE.
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.
Competency Problems: On Finding and Removing Artifacts in Language Data
In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.
The Right Tool for the Job: Matching Model and Instance Complexities
Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy.
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
We publicly release all of our experimental data, including training and validation scores for 2, 100 trials, to encourage further analysis of training dynamics during fine-tuning.
Show Your Work: Improved Reporting of Experimental Results
Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e. g., accuracy) on held-out test data, compared to previous results.
RNN Architecture Learning with Sparse Regularization
Our method also highlights the interpretable properties of rational RNNs.
Open Loop Hyperparameter Optimization and Determinantal Point Processes
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated.
Key-Value Memory Networks for Directly Reading Documents
Directly reading documents and being able to answer questions from them is an unsolved challenge.
Ranked #12 on
Question Answering
on WikiQA
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
A long-term goal of machine learning is to build intelligent conversational agents.
Retrofitting Word Vectors to Semantic Lexicons
Vector space word representations are learned from distributional information of words in large corpora.
