Search Results for author:
Found 45 papers, 16 papers with code
The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank
While previous work has tackled this problem with models trained specifically to do fill in the blank, a more useful model is one that can effectively perform _both_ FitB and continuation tasks.
Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards
Our attack consists of two steps: first, we show how an attacker can determine which model was used to generate a given reply with more than $95\%$ accuracy; and then, the attacker can use this information to consistently vote for (or against) a target model.
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks.
Measuring Non-Adversarial Reproduction of Training Data in Large Language Models
For a variety of innocuous prompt categories (e. g., writing a letter or a tutorial), we show that up to 15% of the text output by popular conversational language models overlaps with snippets from the Internet.
Chasing Random: Instruction Selection Strategies Fail to Generalize
Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions.
Persistent Pre-Training Poisoning of LLMs
Our work evaluates for the first time whether language models can also be compromised during pre-training, with a focus on the persistence of pre-training attacks after models are fine-tuned as helpful and harmless chatbots (i. e., after SFT and DPO).
Human-aligned Chess with a Bit of Search
Chess has long been a testbed for AI's quest to match human intelligence, and in recent years, chess AI systems have surpassed the strongest humans at the game.
Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning
One-size-fits-all large language models (LLMs) are increasingly being used to help people with their writing.
Consent in Crisis: The Rapid Decline of the AI Data Commons
To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora.
RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Many commercial and open-source models claim to detect machine-generated text with extremely high accuracy (99% or more).
LMD3: Language Model Data Density Dependence
We develop a methodology for analyzing language model task performance at the individual example level based on training data density estimation.
Forcing Diffuse Distributions out of Language Models
Despite being trained specifically to follow user instructions, today's instructiontuned language models perform poorly when instructed to produce random outputs.
Gemma: Open Models Based on Gemini Research and Technology
This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models.
Scalable Extraction of Training Data from (Production) Language Models
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset.
Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System
Neural language models are increasingly deployed into APIs and websites that allow a user to pass in a prompt and receive generated text.
Effective Prompt Extraction from Language Models
In experiments with 3 different sources of prompts and 11 underlying large language models, we find that simple text-based attacks can in fact reveal prompts with high probability.
Are aligned neural networks adversarially aligned?
We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force.
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
Second, we explore the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations.
Extracting Training Data from Diffusion Models
Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images.
Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text
As text generated by large language models proliferates, it becomes vital to understand how humans engage with such text, and whether or not they are able to detect when the text they are reading did not originate with a human writer.
Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers
Recent developments in natural language generation (NLG) using neural language models have brought us closer than ever to the goal of building AI-powered creative writing tools.
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
Studying data memorization in neural language models helps us understand the risks (e. g., to privacy or copyright) associated with models regurgitating training data and aids in the development of countermeasures.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In this paper, we frame D&D specifically as a dialogue system challenge, where the tasks are to both generate the next conversational turn in the game and predict the state of the game given the dialogue history.
Measuring Forgetting of Memorized Training Examples
In memorization, models overfit specific training examples and become susceptible to privacy attacks.
The Case for a Single Model that can Both Generate Continuations and Fill in the Blank
The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Coreference Resolution
on Winograd Schema Challenge
Creating Multimedia Summaries Using Tweets and Videos
While popular televised events such as presidential debates or TV shows are airing, people provide commentary on them in real-time.
Quantifying Memorization Across Neural Language Models
Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim.
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets
We use our method to curate SynthBio - a new evaluation set for WikiBio - composed of structured attribute lists describing fictional individuals, mapped to natural language biographies.
A Recipe For Arbitrary Text Style Transfer with Large Language Models
In this paper, we leverage large language models (LMs) to perform zero-shot text style transfer.
Wordcraft: a Human-AI Collaborative Editor for Story Writing
As neural language models grow in effectiveness, they are increasingly being applied in real-world settings.
Deduplicating Training Data Makes Language Models Better
As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data.
RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text
In recent years, large neural networks for natural language generation (NLG) have made leaps and bounds in their ability to generate fluent text.
COVI White Paper
Manual contact tracing of Covid-19 cases has significant challenges that limit the ability of public health authorities to minimize community infections.
Toward Better Storylines with Sentence-Level Language Models
We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives.
Trading Off Diversity and Quality in Natural Language Generation
For open-ended language generation tasks such as storytelling and dialogue, choosing the right decoding algorithm is critical to controlling the tradeoff between generation quality and diversity.
Contact Tracing Mobile Apps for COVID-19: Privacy Considerations and Related Trade-offs
Importantly, though we discuss potential modifications, this document is not meant as a formal research paper, but instead is a response to some of the privacy characteristics of direct contact tracing apps like TraceTogether and an early-stage Request for Comments to the community.
Cryptography and Security
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text.
Comparison of Diverse Decoding Methods from Conditional Language Models
While conditional language models have greatly improved in their ability to output high-quality natural language, many NLP applications benefit from being able to generate a diverse set of candidate sequences.
Unsupervised Hierarchical Story Infilling
Story infilling involves predicting words to go into a missing span from a story.
ChatEval: A Tool for Chatbot Evaluation
We introduce a unified framework for human evaluation of chatbots that augments existing tools and provides a web-based hub for researchers to share and compare their dialog systems.
Learning Translations via Images with a Massively Multilingual Image Dataset
To facilitate research on the task, we introduce a large-scale multilingual corpus of images, each labeled with the word it represents.
Understanding image motion with group representations
Our results demonstrate that this representation is useful for learning motion in the general setting where explicit labels are difficult to obtain.
