Search Results for author:
Found 33 papers, 14 papers with code
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Evaluating bias, fairness, and social impact in monolingual language models is a difficult task.
Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?
New capabilities in foundation models are owed in large part to massive, widely-sourced, and under-documented training data collections.
A Safe Harbor for AI Evaluation and Red Teaming
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems.
On the Societal Impact of Open Foundation Models
To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk.
A Survey on Data Selection for Language Models
A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training.
Foundation Model Transparency Reports
Foundation models are critical digital technologies with sweeping societal impact that necessitates transparency.
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages.
The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI
The race to train language models on vast, diverse, and inconsistently documented datasets has raised pressing concerns about the legal and ethical risks for practitioners.
The Foundation Model Transparency Index
We score 10 major foundation model developers (e. g. OpenAI, Google, Meta) against the 100 indicators to assess their transparency.
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
We first construct the Feedback Collection, a new dataset that consists of 1K fine-grained score rubrics, 20K instructions, and 100K responses and language feedback generated by GPT-4.
OctoPack: Instruction Tuning Code Large Language Models
We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46. 2% pass@1).
Ranked #4 on
Code Generation
on HumanEval
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
Second, we explore the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations.
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings.
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022).
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Multi-task Language Understanding
on BBH-nlp
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
Our work quantitatively demonstrates that combining compression methods can synergistically reduce model size, and that practitioners should prioritize (1) quantization, (2) knowledge distillation, and (3) pruning to maximize accuracy vs. model size tradeoffs.
MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages
We present the results of the Workshop on Multilingual Information Access (MIA) 2022 Shared Task, evaluating cross-lingual open-retrieval question answering (QA) systems in 16 typologically diverse languages.
Active Learning Over Multiple Domains in Natural Language Tasks
Studies of active learning traditionally assume the target and source data stem from a single domain.
Entity-Based Knowledge Conflicts in Question Answering
To understand how models use these sources together, we formalize the problem of knowledge conflicts, where the contextual information contradicts the learned information.
Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP
These experiments on AmbER sets show their utility as an evaluation tool and highlight the weaknesses of popular retrieval systems.
Leveraging Query Resolution and Reading Comprehension for Conversational Passage Retrieval
This paper describes the participation of UvA. ILPS group at the TREC CAsT 2020 track.
A Comparison of Question Rewriting Methods for Conversational Passage Retrieval
Conversational passage retrieval relies on question rewriting to modify the original question so that it no longer depends on the conversation history.
Pivot Through English: Reliably Answering Multilingual Questions without Document Retrieval
Existing methods for open-retrieval question answering in lower resource languages (LRLs) lag significantly behind English.
A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering
The dependency between an adequate question formulation and correct answer selection is a very intriguing but still underexplored area.
Open-Domain Question Answering Goes Conversational via Question Rewriting
We introduce a new dataset for Question Rewriting in Conversational Context (QReCC), which contains 14K conversations with 80K question-answer pairs.
How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?
Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models.
On the Transferability of Minimal Prediction Preserving Inputs in Question Answering
In the context of question answering, we investigate competing hypotheses for the existence of MPPIs, including poor posterior calibration of neural models, lack of pretraining, and "dataset bias" (where a model learns to attend to spurious, non-generalizable cues in the training data).
MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets.
Question Rewriting for Conversational Question Answering
Conversational question answering (QA) requires the ability to correctly interpret a question in the context of previous conversation turns.
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
To produce a domain-agnostic question answering model for the Machine Reading Question Answering (MRQA) 2019 Shared Task, we investigate the relative benefits of large pre-trained language models, various data sampling strategies, as well as query and context paraphrases generated by back-translation.
A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs
LSTMs have become a basic building block for many deep NLP models.
