Search Results for author:
Found 35 papers, 19 papers with code
Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
Large language models (LLMs) have achieved widespread success on a variety of in-context few-shot tasks, but this success is typically evaluated via correctness rather than consistency.
Tool Learning with Foundation Models
Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 17 representative tools and show the potential of current foundation models in skillfully utilizing tools.
Pretraining Language Models with Human Preferences
Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more.
HyperTuning: Toward Adapting Large Language Models without Back-propagation
Fine-tuning large language models for different tasks can be costly and inefficient, and even methods that reduce the number of tuned parameters still require full gradient-based optimization.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
What Language Model to Train if You Have One Million GPU Hours?
All our models and code are open-sourced at https://huggingface. co/bigscience .
Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions
These findings suggest that, in the case of answering reading comprehension questions, debate is not a helpful format.
EleutherAI: Going Beyond "Open Science" to "Science in the Open"
Over the past two years, EleutherAI has established itself as a radically novel initiative aimed at both promoting open-source research and conducting research in a transparent, openly accessible and collaborative manner.
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
We present the results of the NLP Community Metasurvey.
Investigating Efficiently Extending Transformers for Long Input Summarization
While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge.
Ranked #2 on
Long-range modeling
on SCROLLS
(GovRep metric)
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way
Summarization datasets are often assembled either by scraping naturally occurring public-domain summaries -- which are nearly always in difficult-to-work-with technical domains -- or by using approximate heuristics to extract them from everyday text -- which frequently yields unfaithful summaries.
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license.
Ranked #48 on
Multi-task Language Understanding
on MMLU
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions
We use long contexts -- humans familiar with the context write convincing explanations for pre-selected correct and incorrect answers, and we test if those explanations allow humans who have not read the full context to more accurately determine the correct answer.
QuALITY: Question Answering with Long Input Texts, Yes!
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5, 000 tokens, much longer than typical current models can process.
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
We find that AFLite indeed selects more challenging examples, lowering the performance of evaluated models more as stronger adversary models are used.
BBQ: A Hand-Built Bias Benchmark for Question Answering
It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA).
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
Despite the success of fine-tuning pretrained language encoders like BERT for downstream natural language understanding (NLU) tasks, it is still poorly understood how neural networks change after fine-tuning.
Comparing Test Sets with Item Response Theory
Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks.
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models.
Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
We find that the most important ingredients for high quality saliency map generation are (1) using both masked-in and masked-out objectives and (2) training the classifier alongside the masking model.
Reducing false-positive biopsies with deep neural networks that utilize local and global information in screening mammograms
Breast cancer is the most common cancer in women, and hundreds of thousands of unnecessary biopsies are done around the world at a tremendous cost.
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
However, we fail to observe more granular correlations between probing and target task performance, highlighting the need for further work on broad-coverage probing benchmarks.
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings.
Ranked #20 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
However, we fail to observe more granular correlations between probing and target task performance, highlighting the need for further work on broad-coverage probing benchmarks.
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
We introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks.
An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization
In this work, we extend the globally-aware multiple instance classifier, a framework we proposed to address these unique properties of medical images.
Do Attention Heads in BERT Track Syntactic Dependencies?
We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations.
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.
Improving localization-based approaches for breast cancer screening exam classification
We trained and evaluated a localization-based deep CNN for breast cancer screening exam classification on over 200, 000 exams (over 1, 000, 000 images).
Screening Mammogram Classification with Prior Exams
Radiologists typically compare a patient's most recent breast cancer screening exam to their previous ones in making informed diagnoses.
Globally-Aware Multiple Instance Classifier for Breast Cancer Screening
Moreover, both the global structure and local details play important roles in medical image analysis tasks.
Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening
We present a deep convolutional neural network for breast cancer screening exam classification, trained and evaluated on over 200, 000 exams (over 1, 000, 000 images).
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
Pretraining sentence encoders with language modeling and related unsupervised tasks has recently been shown to be very effective for language understanding tasks.
Unsupervised Sentence Compression using Denoising Auto-Encoders
In sentence compression, the task of shortening sentences while retaining the original meaning, models tend to be trained on large corpora containing pairs of verbose and compressed sentences.
