Search Results for author:
Found 40 papers, 33 papers with code
The Mismeasure of Man and Models: Evaluating Allocational Harms in Large Language Models
Large language models (LLMs) are now being considered and even deployed for applications that support high-stakes decision-making, such as recruitment and clinical decisions.
The OPS-SAT benchmark for detecting anomalies in satellite telemetry
Detecting anomalous events in satellite telemetry is a critical task in space operations.
Do Parameters Reveal More than Loss for Membership Inference?
Membership inference attacks are used as a key tool for disclosure auditing.
Addressing Both Statistical and Causal Gender Fairness in NLP Models
Statistical fairness stipulates equivalent outcomes for every protected group, whereas causal fairness prescribes that a model makes the same prediction for an individual regardless of their protected characteristics.
Do Membership Inference Attacks Work on Large Language Models?
Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data.
Understanding Variation in Subpopulation Susceptibility to Poisoning Attacks
Machine learning is susceptible to poisoning attacks, in which an attacker controls a small fraction of the training data and chooses that data with the goal of inducing some behavior unintended by the model developer in the trained model.
SoK: Pitfalls in Evaluating Black-Box Attacks
However, these works make different assumptions on the adversary's knowledge and current literature lacks a cohesive organization centered around the threat model.
SoK: Memorization in General-Purpose Large Language Models
A major part of this success is due to their huge training datasets and the unprecedented number of model parameters, which allow them to memorize large amounts of information contained in the training data.
Manipulating Transfer Learning for Property Inference
We demonstrate attacks in which an adversary can manipulate the upstream model to conduct highly effective and specific property inference attacks (AUC score $> 0. 9$), without incurring significant performance loss on the main task.
TrojanPuzzle: Covertly Poisoning Code-Suggestion Models
To achieve this, prior attacks explicitly inject the insecure code payload into the training data, making the poison data detectable by static analysis tools that can remove such malicious data from the training set.
SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning
Deploying machine learning models in production may allow adversaries to infer sensitive information about training data.
Dissecting Distribution Inference
A distribution inference attack aims to infer statistical properties of data used to train machine learning models.
Balanced Adversarial Training: Balancing Tradeoffs between Fickleness and Obstinacy in NLP Models
Traditional (fickle) adversarial examples involve finding a small perturbation that does not change an input's true label but confuses the classifier into outputting a different prediction.
Are Attribute Inference Attacks Just Imputation?
Our main conclusions are: (1) previous attribute inference methods do not reveal more about the training data from the model than can be inferred by an adversary without access to the trained model, but with the same knowledge of the underlying distribution as needed to train the attribute inference attack; (2) black-box attribute inference attacks rarely learn anything that cannot be learned without the model; but (3) white-box attacks, which we introduce and evaluate in the paper, can reliably identify some records with the sensitive value attribute that would not be predicted without having access to the model.
Combing for Credentials: Active Pattern Extraction from Smart Reply
We show experimentally that it is possible for an adversary to extract sensitive user information present in the training data, even in realistic settings where all interactions with the model must go through a front-end that limits the types of queries.
Memorization in NLP Fine-tuning Methods
Large language models are shown to present privacy risks through memorization of training data, and several recent works have studied such risks for the pre-training phase.
Formalizing and Estimating Distribution Inference Risks
Distribution inference attacks can pose serious risks when models are trained on private data, but are difficult to distinguish from the intrinsic purpose of statistical machine learning -- namely, to produce models that capture statistical properties about a distribution.
Understanding Intrinsic Robustness Using Label Uncertainty
A fundamental question in adversarial machine learning is whether a robust classifier exists for a given task.
Formalizing Distribution Inference Risks
Property inference attacks reveal statistical properties about a training set but are difficult to distinguish from the primary purposes of statistical machine learning, which is to produce models that capture statistical properties about a distribution.
Stealthy Backdoors as Compression Artifacts
In a backdoor attack on a machine learning model, an adversary produces a model that performs well on normal inputs but outputs targeted misclassifications on inputs containing a small trigger pattern.
Improved Estimation of Concentration Under $\ell_p$-Norm Distance Metrics Using Half Spaces
Mahloujifar et al. presented an empirical way to measure the concentration of a data distribution using samples, and employed it to find lower bounds on intrinsic robustness for several benchmark datasets.
Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory
Most NLP datasets are manually labeled, so suffer from inconsistent labeling or limited size.
Model-Targeted Poisoning Attacks with Provable Convergence
Our attack is the first model-targeted poisoning attack that provides provable convergence for convex models, and in our experiments, it either exceeds or matches state-of-the-art attacks in terms of attack success rate and distance to the target model.
Pointwise Paraphrase Appraisal is Potentially Problematic
The prevailing approach for training and evaluating paraphrase identification models is constructed as a binary classification problem: the model is given a pair of sentences, and is judged by how accurately it classifies pairs as either paraphrases or non-paraphrases.
Revisiting Membership Inference Under Realistic Assumptions
Since previous inference attacks fail in imbalanced prior setting, we develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function, and show that an attack that combines this with thresholds on the per-instance loss can achieve high PPV even in settings where other attacks appear to be ineffective.
Certifying Joint Adversarial Robustness for Model Ensembles
Deep Neural Networks (DNNs) are often vulnerable to adversarial examples. Several proposed defenses deploy an ensemble of models with the hope that, although the individual models may be vulnerable, an adversary will not be able to find an adversarial example that succeeds against the ensemble.
One Neuron to Fool Them All
Despite vast research in adversarial examples, the root causes of model susceptibility are not well understood.
Understanding the Intrinsic Robustness of Image Distributions using Conditional Generative Models
Starting with Gilmer et al. (2018), several works have demonstrated the inevitability of adversarial examples based on different assumptions about the underlying input probability space.
Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization
We develop a notion of representation vulnerability that captures the maximum change of mutual information between the input and output distributions, under the worst-case input perturbation.
Advances and Open Problems in Federated Learning
FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches.
Efficient Privacy-Preserving Stochastic Nonconvex Optimization
While many solutions for privacy-preserving convex empirical risk minimization (ERM) have been developed, privacy-preserving nonconvex ERM remains a challenge.
Hybrid Batch Attacks: Finding Black-box Adversarial Examples with Limited Queries
In a black-box setting, the adversary only has API access to the target model and each query is expensive.
Cryptography and Security
Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness
Many recent works have shown that adversarial examples that fool classifiers can be found by minimally perturbing a normal input.
Evaluating Differentially Private Machine Learning in Practice
Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, $\epsilon$, about how much information is leaked by a mechanism.
Distributed Learning without Distress: Privacy-Preserving Empirical Risk Minimization
We explore two popular methods of differential privacy, output perturbation and gradient perturbation, and advance the state-of-the-art for both methods in the distributed learning setting.
Cost-Sensitive Robustness against Adversarial Examples
Several recent works have developed methods for training classifiers that are certifiably robust against norm-bounded adversarial perturbations.
Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning
We show in experiments that our method can attack a gradient-boosted machine learning model with evasion rates that are substantial and appear to be strongly dependent on the dataset.
Cryptography and Security
Query-limited Black-box Attacks to Classifiers
Specifically, we consider the problem of attacking machine learning classifiers subject to a budget of feature modification cost while minimizing the number of queries, where each query returns only a class and confidence score.
Feature Squeezing Mitigates and Detects Carlini/Wagner Adversarial Examples
Feature squeezing is a recently-introduced framework for mitigating and detecting adversarial examples.
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Although deep neural networks (DNNs) have achieved great success in many tasks, they can often be fooled by \emph{adversarial examples} that are generated by adding small but purposeful distortions to natural examples.
