Search Results for author:
Found 26 papers, 7 papers with code
Intrinsic Evaluation of Summarization Datasets
High quality data forms the bedrock for building meaningful statistical models in NLP.
The Reality of AI and Biorisk
To accurately and confidently answer the question 'could an AI model or system increase biorisk', it is necessary to have both a sound theoretical threat model for how AI models or systems could increase biorisk and a robust method for testing that threat model.
Effective Mitigations for Systemic Risks from General-Purpose AI
The systemic risks posed by general-purpose AI models are a growing concern, yet the effectiveness of mitigations remains underexplored.
Language model developers should report train-test overlap
Overall, we take the position that language model developers should publish train-test overlap statistics and/or training data whenever they report evaluation results on public test sets.
The Foundation Model Transparency Index v1.1: May 2024
To characterize the status quo, the Foundation Model Transparency Index was launched in October 2023 to measure the transparency of leading foundation model developers.
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications.
A Safe Harbor for AI Evaluation and Red Teaming
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems.
On the Societal Impact of Open Foundation Models
To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk.
Foundation Model Transparency Reports
Foundation models are critical digital technologies with sweeping societal impact that necessitates transparency.
The Foundation Model Transparency Index
We score 10 major foundation model developers (e. g. OpenAI, Google, Meta) against the 100 indicators to assess their transparency.
Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
In practice, the societal impact of machine learning is determined by the surrounding context of machine learning deployments.
Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs
Large language models (LLMs) power many state-of-the-art systems in natural language processing.
Ecosystem Graphs: The Social Footprint of Foundation Models
Foundation models (e. g. ChatGPT, StableDiffusion) pervasively influence society, warranting immediate social attention.
Evaluation for Change
Under our analysis, we conjecture that the current trajectory of NLP suggests evaluation's power is waning, in spite of its potential for realizing more pluralistic ambitions in the field.
Evaluating Human-Language Model Interaction
To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics.
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e. g. training data), are deployed by multiple decision-makers.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Data Governance in the Age of Large-Scale Data-Driven Language Technology
The recent emergence and adoption of Machine Learning technology, and specifically of Large Language Models, has drawn attention to the need for systematic and transparent management of language data.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Generalized Optimal Linear Orders
Consequently, in designing computational models of language, the de facto approach is to present sentences to machines with the words ordered in the same order as in the original human-authored sentence.
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Contextualized representations (e. g. ELMo, BERT) have become the default pretrained representations for downstream NLP applications.
Long-Distance Dependencies Don't Have to Be Long: Simplifying through Provably (Approximately) Optimal Permutations
Neural models at the sentence level often operate on the constituent words/tokens in a way that encodes the inductive bias of processing the input in a similar fashion to how humans do.
SPARSE: Structured Prediction using Argument-Relative Structured Encoding
We apply our approach to the second-order structured prediction task studied in the 2016/2017 Belief and Sentiment analysis evaluations (BeSt): given a document and its entities, relations, and events (including metadata and mentions), determine the sentiment of each entity towards every relation and event in the document.
