Search Results for author:
Found 49 papers, 30 papers with code
Consistency Checks for Language Model Forecasters
Forecasting is a task that is difficult to evaluate: the ground truth can only be known in the future.
Gradient Masking All-at-Once: Ensemble Everything Everywhere Is Not Robust
Ensemble everything everywhere is a defense to adversarial examples that was recently proposed to make image classifiers robust.
Measuring Non-Adversarial Reproduction of Training Data in Large Language Models
For a variety of innocuous prompt categories (e. g., writing a letter or a tutorial), we show that up to 15% of the text output by popular conversational language models overlaps with snippets from the Internet.
Persistent Pre-Training Poisoning of LLMs
Our work evaluates for the first time whether language models can also be compromised during pre-training, with a focus on the persistence of pre-training attacks after models are fine-tuned as helpful and harmless chatbots (i. e., after SFT and DPO).
Gradient-based Jailbreak Images for Multimodal Fusion Models
Augmenting language models with image inputs may enable more effective jailbreak attacks through continuous optimization, unlike text inputs that require discrete optimization.
Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data
We argue that this approach is fundamentally unsound: to provide convincing evidence, the data creator needs to demonstrate that their attack has a low false positive rate, i. e., that the attack's output is unlikely under the null hypothesis that the model was not trained on the target data.
An Adversarial Perspective on Machine Unlearning for AI Safety
Large language models are finetuned to refuse questions about hazardous knowledge, but these protections can often be bypassed.
Extracting Training Data from Document-Based VQA Models
Vision-Language Models (VLMs) have made remarkable progress in document-based Visual Question Answering (i. e., responding to queries about the contents of an input document provided as an image).
Adversarial Search Engine Optimization for Large Language Models
As LLMs are increasingly used to rank third-party content, we expect Preference Manipulation Attacks to emerge as a significant threat.
Blind Baselines Beat Membership Inference Attacks for Foundation Models
Membership inference (MI) attacks try to determine if a data sample was used to train a machine learning model.
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks.
Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition
To study this problem, we organized a capture-the-flag competition at IEEE SaTML 2024, where the flag is a secret string in the LLM system prompt.
Evaluations of Machine Learning Privacy Defenses are Misleading
Empirical defenses for machine learning privacy forgo the provable guarantees of differential privacy in the hope of achieving higher utility while resisting realistic adversaries.
Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs
Large language models are aligned to be safe, preventing users from generating harmful content like misinformation or instructions for illegal activities.
Privacy Backdoors: Stealing Data with Corrupted Pretrained Models
We show that this practice introduces a new risk of privacy backdoors.
Query-Based Adversarial Prompt Generation
Recent work has shown it is possible to construct adversarial examples that cause an aligned language model to emit harmful strings or perform harmful behavior.
Scalable Extraction of Training Data from (Production) Language Models
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset.
Universal Jailbreak Backdoors from Poisoned Human Feedback
Reinforcement Learning from Human Feedback (RLHF) is used to align large language models to produce helpful and harmless responses.
Privacy Side Channels in Machine Learning Systems
Most current approaches for protecting privacy in machine learning (ML) assume that models exist in a vacuum.
Evaluating Superhuman Models with Consistency Checks
If machine learning models were to achieve superhuman abilities at various reasoning or decision-making tasks, how would we go about evaluating such models, given that humans would necessarily be poor proxies for ground truth?
Evading Black-box Classifiers Without Breaking Eggs
We then design new attacks that reduce the number of bad queries by $1. 5$-$7. 3\times$, but often at a significant increase in total (non-bad) queries.
Randomness in ML Defenses Helps Persistent Attackers and Hinders Evaluators
It is becoming increasingly imperative to design robust ML defenses.
Poisoning Web-Scale Training Datasets is Practical
Deep learning models are often trained on distributed, web-scale datasets crawled from the internet.
Tight Auditing of Differentially Private Machine Learning
Moreover, our auditing scheme requires only two training runs (instead of thousands) to produce tight privacy estimates, by adapting recent advances in tight composition theorems for differential privacy.
Extracting Training Data from Diffusion Models
Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images.
Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
The performance of differentially private machine learning can be boosted significantly by leveraging the transfer learning capabilities of non-private models pretrained on large public datasets.
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
Studying data memorization in neural language models helps us understand the risks (e. g., to privacy or copyright) associated with models regurgitating training data and aids in the development of countermeasures.
Preprocessors Matter! Realistic Decision-Based Attacks on Machine Learning Systems
Decision-based attacks construct adversarial examples against a machine learning (ML) model by making only hard-label queries.
Red-Teaming the Stable Diffusion Safety Filter
We then reverse-engineer the filter and find that while it aims to prevent sexual content, it ignores violence, gore, and other similarly disturbing content.
SNAP: Efficient Extraction of Private Properties with Poisoning
Property inference attacks allow an adversary to extract global properties of the training dataset from a machine learning model.
Measuring Forgetting of Memorized Training Examples
In memorization, models overfit specific training examples and become susceptible to privacy attacks.
Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets
We show that an adversary who can poison a training dataset can cause models trained on this dataset to leak significant private details of training points belonging to other parties.
What Does it Mean for a Language Model to Preserve Privacy?
Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their training sets.
Large Language Models Can Be Strong Differentially Private Learners
Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them
We prove a general hardness reduction between detection and classification of adversarial examples: given a robust detector for attacks at distance {\epsilon} (in some metric), we can build a similarly robust (but inefficient) classifier for attacks at distance {\epsilon}/2.
Data Poisoning Won't Save You From Facial Recognition
We demonstrate that this strategy provides a false sense of security, as it ignores an inherent asymmetry between the parties: users' pictures are perturbed once and for all before being published (at which point they are scraped) and must thereafter fool all future models -- including models trained adaptively against the users' past attacks, or models that use technologies discovered after the attack.
Antipodes of Label Differential Privacy: PATE and ALIBI
We propose two novel approaches based on, respectively, the Laplace mechanism and the PATE framework, and demonstrate their effectiveness on standard benchmarks.
Differentially Private Learning Needs Better Features (or Much More Data)
We demonstrate that differentially private machine learning has not yet reached its "AlexNet moment" on many canonical vision tasks: linear models trained on handcrafted features significantly outperform end-to-end deep neural networks for moderate privacy budgets.
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations
Adversarial examples are malicious inputs crafted to induce misclassification.
Advances and Open Problems in Federated Learning
FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches.
Adversarial Training and Robustness for Multiple Perturbations
Defenses against adversarial examples, such as adversarial training, are typically tailored to a single perturbation type (e. g., small $\ell_\infty$-noise).
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
Excessive invariance is not limited to models trained to be robust to perturbation-based $\ell_p$-norm adversaries.
SentiNet: Detecting Physical Attacks Against Deep Learning Systems
By leveraging the neural network's susceptibility to attacks and by using techniques from model interpretability and object detection as detection mechanisms, SentiNet turns a weakness of a model into a strength.
Cryptography and Security
AdVersarial: Perceptual Ad Blocking meets Adversarial Machine Learning
On the other, we present a concrete set of attacks on visual ad-blockers by constructing adversarial examples in a real web page context.
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
As Machine Learning (ML) gets applied to security-critical or sensitive domains, there is a growing need for integrity and privacy for outsourced ML computations.
Ensemble Adversarial Training: Attacks and Defenses
We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss.
The Space of Transferable Adversarial Examples
Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time.
Stealing Machine Learning Models via Prediction APIs
In such attacks, an adversary with black-box access, but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i. e., "steal") the model.
