Search Results for author:
Found 21 papers, 7 papers with code
Intrinsic Evaluation of Summarization Datasets
High quality data forms the bedrock for building meaningful statistical models in NLP.
A Safe Harbor for AI Evaluation and Red Teaming
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems.
On the Societal Impact of Open Foundation Models
To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk.
Foundation Model Transparency Reports
Foundation models are critical digital technologies with sweeping societal impact that necessitates transparency.
The Foundation Model Transparency Index
We score 10 major foundation model developers (e. g. OpenAI, Google, Meta) against the 100 indicators to assess their transparency.
Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
In practice, the societal impact of machine learning is determined by the surrounding context of machine learning deployments.
Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs
Large language models (LLMs) power many state-of-the-art systems in natural language processing.
Ecosystem Graphs: The Social Footprint of Foundation Models
Foundation models (e. g. ChatGPT, StableDiffusion) pervasively influence society, warranting immediate social attention.
Evaluation for Change
Under our analysis, we conjecture that the current trajectory of NLP suggests evaluation's power is waning, in spite of its potential for realizing more pluralistic ambitions in the field.
Evaluating Human-Language Model Interaction
To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics.
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e. g. training data), are deployed by multiple decision-makers.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Data Governance in the Age of Large-Scale Data-Driven Language Technology
The recent emergence and adoption of Machine Learning technology, and specifically of Large Language Models, has drawn attention to the need for systematic and transparent management of language data.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Generalized Optimal Linear Orders
Consequently, in designing computational models of language, the de facto approach is to present sentences to machines with the words ordered in the same order as in the original human-authored sentence.
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Contextualized representations (e. g. ELMo, BERT) have become the default pretrained representations for downstream NLP applications.
Long-Distance Dependencies Don't Have to Be Long: Simplifying through Provably (Approximately) Optimal Permutations
Neural models at the sentence level often operate on the constituent words/tokens in a way that encodes the inductive bias of processing the input in a similar fashion to how humans do.
SPARSE: Structured Prediction using Argument-Relative Structured Encoding
We apply our approach to the second-order structured prediction task studied in the 2016/2017 Belief and Sentiment analysis evaluations (BeSt): given a document and its entities, relations, and events (including metadata and mentions), determine the sentiment of each entity towards every relation and event in the document.
