Search Results for author:
Found 113 papers, 51 papers with code
On the Privacy Risk of In-context Learning
We show that deploying prompted models presents a significant privacy risk for the data used within the prompt by instantiating a highly effective membership inference attack.
Temporal-Difference Learning Using Distributed Error Signals
We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks.
Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses
To quantify these risks, we introduce a new safety evaluation framework based on impermissible information leakage of model outputs and demonstrate how our proposed question-decomposition attack can extract dangerous knowledge from a censored LLM more effectively than traditional jailbreaking.
UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI
The promise is that if the model does not have a certain malicious capability, then it cannot be used for the associated malicious purpose.
LLM Dataset Inference: Did you train on my dataset?
Instead, we propose a new dataset inference method to accurately identify the datasets used to train large language models.
Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model
Machine learning models can be trained with formal privacy guarantees via differentially private optimizers such as DP-SGD.
Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias
We simulate AR interventions by curating representative training batches for stochastic gradient descent to demonstrate how AR can improve upon the unfairnesses of models and data ecosystems subject to other MIDS.
Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy
In the privacy literature, this is known as membership inference.
The Fundamental Limits of Least-Privilege Learning
The promise of least-privilege learning -- to find feature representations that are useful for a learning task but prevent inference of any sensitive information unrelated to this task -- is highly appealing.
Architectural Neural Backdoors from First Principles
In this work we construct an arbitrary trigger detector which can be used to backdoor an architecture with no human supervision.
Regulation Games for Trustworthy Machine Learning
Existing work on trustworthy machine learning (ML) often concentrates on individual aspects of trust, such as fairness or privacy.
Unlearnable Algorithms for In-context Learning
Machine unlearning is a desirable operation as models get increasingly deployed on data with unknown provenance.
Decentralised, Collaborative, and Privacy-preserving Machine Learning for Multi-Hospital Data
In addition, the ML models trained with DeCaPH framework in general outperform those trained solely with the private datasets from individual parties, showing that DeCaPH enhances the model generalizability.
Memorization in Self-Supervised Learning Improves Downstream Generalization
Our definition compares the difference in alignment of representations for data points and their augmented views returned by both encoders that were trained on these data points and encoders that were not.
Beyond Labeling Oracles: What does it mean to steal ML models?
We thoroughly research factors influencing the success of model extraction.
The Adversarial Implications of Variable-Time Inference
In our examination of the timing side-channel vulnerabilities associated with this algorithm, we identified the potential to enhance decision-based attacks.
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?
Specifically, we demonstrate that semantic censorship can be perceived as an undecidable problem, highlighting the inherent challenges in censorship that arise due to LLMs' programmatic and instruction-following capabilities.
Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGD
Put all together, our evaluation shows that this novel DP-SGD analysis allows us to now formally show that DP-SGD leaks significantly less privacy for many datapoints (when trained on common benchmarks) than the current data-independent guarantee.
Augment then Smooth: Reconciling Differential Privacy with Certified Robustness
However, we show that standard differentially private model training is insufficient for providing strong certified robustness guarantees.
When Vision Fails: Text Attacks Against ViT and OCR
While text-based machine learning models that operate on visual inputs of rendered text have become robust against a wide range of existing attacks, we show that they are still vulnerable to visual adversarial examples encoded as text.
Training Private Models That Know What They Don't Know
Training reliable deep learning models which avoid making overconfident but incorrect predictions is a longstanding challenge.
The Curse of Recursion: Training on Generated Data Makes Models Forget
It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images.
Have it your way: Individualized Privacy Assignment for DP-SGD
DP-SGD is the canonical approach to training models with differential privacy.
Learning with Impartiality to Walk on the Pareto Frontier of Fairness, Privacy, and Utility
Deploying machine learning (ML) models often requires both fairness and privacy guarantees.
Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregation
FL is promoted as a privacy-enhancing technology (PET) that provides data minimization: data never "leaves" personal devices and users share only model updates with a server (e. g., a company) coordinating the distributed training.
Learned Systems Security
We analyze the root causes of potentially-increased attack surface in learned systems and develop a framework for identifying vulnerabilities that stem from the use of ML.
Private Multi-Winner Voting for Machine Learning
We use our mechanisms to enable privacy-preserving multi-label learning in the central setting by extending the canonical single-label technique: PATE.
Verifiable and Provably Secure Machine Unlearning
In this framework, the server first computes a proof that the model was trained on a dataset $D$.
Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search
For instance, we achieve 77. 9% accuracy for $(\varepsilon, \delta)=(2, 10^{-5})$ on CIFAR-100 for a model pretrained on ImageNet.
In Differential Privacy, There is Truth: On Vote Leakage in Ensemble Private Learning
When learning from sensitive data, care must be taken to ensure that training algorithms address privacy concerns.
Dataset Inference for Self-Supervised Models
We introduce a new dataset inference defense, which uses the private training set of the victim encoder model to attribute its ownership in the event of stealing.
Proof-of-Learning is Currently More Broken Than You Think
They empirically argued the benefit of this approach by showing how spoofing--computing a proof for a stolen model--is as expensive as obtaining the proof honestly by training the model.
$p$-DkNN: Out-of-Distribution Detection Through Statistical Testing of Deep Representations
We introduce $p$-DkNN, a novel inference procedure that takes a trained deep neural network and analyzes the similarity structures of its intermediate hidden representations to compute $p$-values associated with the end-to-end model prediction.
Efficient Adversarial Training With Data Pruning
In this paper we demonstrate data pruning-a method for increasing adversarial training efficiency through data sub-sampling. We empirically show that data pruning leads to improvements in convergence and reliability of adversarial training, albeit with different levels of utility degradation.
Measuring Forgetting of Memorized Training Examples
In memorization, models overfit specific training examples and become susceptible to privacy attacks.
Intrinsic Anomaly Detection for Multi-Variate Time Series
We introduce a novel, practically relevant variation of the anomaly detection problem in multi-variate time series: intrinsic anomaly detection.
The Privacy Onion Effect: Memorization is Relative
Machine learning models trained on private datasets have been shown to leak their private data.
On the Limitations of Stochastic Pre-processing Defenses
An example of such a defense is to apply a random transformation to inputs prior to feeding them to the model.
Architectural Backdoors in Neural Networks
Machine learning is vulnerable to adversarial manipulation.
Selective Classification Via Neural Network Training Dynamics
Selective classification is the task of rejecting inputs a model would predict incorrectly on through a trade-off between input space coverage and model accuracy.
On the Difficulty of Defending Self-Supervised Learning against Model Extraction
We construct several novel attacks and find that approaches that train directly on a victim's stolen representations are query efficient and enable high accuracy for downstream models.
Differentially Private Speaker Anonymization
We remove speaker information from these attributes by introducing differentially private feature extractors based on an autoencoder and an automatic speech recognizer, respectively, trained using noise layers.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Tubes Among Us: Analog Attack on Automatic Speaker Identification
Recent years have seen a surge in the popularity of acoustics-enabled personal devices powered by machine learning.
Increasing the Cost of Model Extraction with Calibrated Proof of Work
Since we calibrate the effort required to complete the proof-of-work to each query, this only introduces a slight overhead for regular users (up to 2x).
When the Curious Abandon Honesty: Federated Learning Is Not Private
Instead, these devices share gradients, parameters, or other model updates, with a central party (e. g., a company) coordinating the training.
On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning
Machine unlearning, i. e. having a model forget about some of its training data, has become increasingly more important as privacy legislation promotes variants of the right-to-be-forgotten.
Hyperparameter Tuning with Renyi Differential Privacy
For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood.
A Zest of LIME: Towards Architecture-Independent Model Distances
In this paper, we instead propose to compute distance between black-box models by comparing their Local Interpretable Model-Agnostic Explanations (LIME).
Losing Less: A Loss for Differentially Private Deep Learning
In this paper, we are the first to observe that some of this performance can be recovered when training with a loss tailored to DP-SGD; we challenge cross-entropy as the de facto loss for deep learning with DP.
Context-invariant, multi-variate time series representations
Modern time series corpora, in particular those coming from sensor-based data, exhibit characteristics that have so far not been adequately addressed in the literature on representation learning for time series.
Unrolling SGD: Understanding Factors Influencing Machine Unlearning
In this work, we first taxonomize approaches and metrics of approximate unlearning.
Interpretability in Safety-Critical FinancialTrading Systems
Indeed, we show in our evaluation that errors in the forecasting model's predictions alone are not sufficient for trading decisions made based on these forecasts to yield a negative return.
SoK: Machine Learning Governance
The application of machine learning (ML) in computer systems introduces not only many benefits but also risks to society.
On the Exploitability of Audio Machine Learning Pipelines to Surreptitious Adversarial Examples
In the white-box setting, we instantiate this class with a joint, multi-stage optimization attack.
Bad Characters: Imperceptible NLP Attacks
In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs.
Markpainting: Adversarial Machine Learning meets Inpainting
Inpainting is a learned interpolation technique that is based on generative modeling and used to populate masked or missing pieces in an image; it has wide applications in picture editing and retouching.
Dataset Inference: Ownership Resolution in Machine Learning
We thus introduce $dataset$ $inference$, the process of identifying whether a suspected model copy has private knowledge from the original model's dataset, as a defense against model stealing.
Manipulating SGD with Data Ordering Attacks
Machine learning is vulnerable to a wide variety of attacks.
Proof-of-Learning: Definitions and Practice
In particular, our analyses and experiments show that an adversary seeking to illegitimately manufacture a proof-of-learning needs to perform *at least* as much work than is needed for gradient descent itself.
CaPC Learning: Confidential and Private Collaborative Learning
There is currently no method that enables machine learning in such a setting, where both confidentiality and privacy need to be preserved, to prevent both explicit and implicit sharing of data.
Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning
DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D' that differs in just one example. If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private.
On the Robustness of Sentiment Analysis for Stock Price Forecasting
Machine learning (ML) models are known to be vulnerable to attacks both at training and test time.
Neighbors From Hell: Voltage Attacks Against Deep Learning Accelerators on Multi-Tenant FPGAs
We find that even when using the strongest attacker circuit, the prediction accuracy of the DL accelerator is not compromised when running at its safe operating frequency.
Data-Free Model Extraction
Current model extraction attacks assume that the adversary has access to a surrogate dataset with characteristics similar to the proprietary data used to train the victim model.
Adversarial Examples in Constrained Domains
To assess how these algorithms perform, we evaluate them in constrained (e. g., network intrusion detection) and unconstrained (e. g., image recognition) domains.
Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings
Our results highlight lesser-known limitations of methods for DP learning in health care, models that exhibit steep tradeoffs between privacy and utility, and models whose predictions are disproportionately influenced by large demographic groups in the training data.
On Attribution of Deepfakes
We discuss the ethical implications of our work, identify where our technique can be used, and highlight that a more meaningful legislative framework is required for a more transparent and ethical use of generative modeling.
Tempered Sigmoid Activations for Deep Learning with Differential Privacy
Because learning sometimes involves sensitive data, machine learning algorithms have been extended to offer privacy for training data.
Label-Only Membership Inference Attacks
We empirically show that our label-only membership inference attacks perform on par with prior attacks that required access to model confidences.
SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systems
Like other systems based on neural networks, recent research has demonstrated that speech and speaker recognition systems are vulnerable to attacks using manipulated inputs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Sponge Examples: Energy-Latency Attacks on Neural Networks
The high energy costs of neural network training and inference led to the use of acceleration hardware such as GPUs and TPUs.
SOAR: Second-Order Adversarial Regularization
Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples.
On the Robustness of Cooperative Multi-Agent Reinforcement Learning
Our results on the StartCraft II multi-agent benchmark demonstrate that c-MARL teams are highly vulnerable to perturbations applied to one of their agent's observations.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Entangled Watermarks as a Defense against Model Extraction
Such pairs are watermarks, which are not sampled from the task distribution and are only known to the defender.
On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping
In this work, we study the feasibility of an attack-agnostic defense relying on artifacts that are common to all poisoning attacks.
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations
Adversarial examples are malicious inputs crafted to induce misclassification.
Machine Unlearning
Once users have shared their data online, it is generally difficult for them to revoke access and ask for the data to be deleted.
Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications
We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution.
Thieves on Sesame Street! Model Extraction of BERT-based APIs
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.
Improving Differentially Private Models with Active Learning
Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.
Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy
Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data.
High Accuracy and High Fidelity Extraction of Neural Networks
In a model extraction attack, an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access.
How Relevant is the Turing Test in the Age of Sophisbots?
Popular culture has contemplated societies of thinking machines for generations, envisioning futures from utopian to dystopian.
Rearchitecting Classification Frameworks For Increased Robustness
and how to design a classification paradigm that leverages these invariances to improve the robustness accuracy trade-off?
MixMatch: A Holistic Approach to Semi-Supervised Learning
Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets.
Prototypical Examples in Deep Learning: Metrics, Characteristics, and Utility
Machine learning (ML) research has investigated prototypes: examples that are representative of the behavior to be learned.
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
Excessive invariance is not limited to models trained to be robust to perturbation-based $\ell_p$-norm adversaries.
On Evaluating Adversarial Robustness
Correctly evaluating defenses against adversarial examples has proven to be extremely difficult.
Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i. e., how close pairs of points from the same class are relative to pairs of points from different classes.
A General Approach to Adding Differential Privacy to Iterative Training Procedures
In this work we address the practical challenges of training machine learning models on privacy-sensitive datasets by introducing a modular approach that minimizes changes to training algorithms, provides a variety of configuration strategies for the privacy mechanism, and then isolates and simplifies the critical logic that computes the final privacy guarantees.
A Marauder's Map of Security and Privacy in Machine Learning
We structure our discussion around three of these directions, which we believe are likely to lead to significant progress.
Cryptography and Security
Adversarial Vision Challenge
The NIPS 2018 Adversarial Vision Challenge is a competition to facilitate measurable progress towards robust machine vision models and more generally applicable adversarial attacks.
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
However, deep learning is often criticized for its lack of robustness in adversarial settings (e. g., vulnerability to adversarial inputs) and general inability to rationalize its predictions.
Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich.
On the Protection of Private Information in Machine Learning Systems: Two Recent Approaches
The recent, remarkable growth of machine learning has led to intense interest in the privacy of the data on which machine learning relies, and to new techniques for preserving privacy.
Ensemble Adversarial Training: Attacks and Defenses
We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss.
Extending Defensive Distillation
Machine learning is vulnerable to adversarial examples: inputs carefully modified to force misclassification.
The Space of Transferable Adversarial Examples
Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time.
On the (Statistical) Detection of Adversarial Examples
Specifically, we augment our ML model with an additional output, in which the model is trained to classify all adversarial inputs.
Adversarial Attacks on Neural Network Policies
Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification.
Towards the Science of Security and Privacy in Machine Learning
Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics.
Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
The approach combines, in a black-box fashion, multiple models trained with disjoint datasets, such as records from different subsets of users.
Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
An adversarial example library for constructing attacks, building defenses, and benchmarking both
Adversarial Perturbations Against Deep Neural Networks for Malware Classification
Deep neural networks, like many other machine learning models, have recently been shown to lack robustness against adversarially crafted inputs.
Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
We demonstrate our attacks on two commercial machine learning classification systems from Amazon (96. 19% misclassification rate) and Google (88. 94%) using only 800 queries of the victim model, thereby showing that existing machine learning approaches are in general vulnerable to systematic black-box attacks regardless of their structure.
Crafting Adversarial Input Sequences for Recurrent Neural Networks
Machine learning models are frequently used to solve complex security problems, as well as to make decisions in sensitive situations like guiding autonomous vehicles or predicting financial market behaviors.
Detection under Privileged Information
In this paper, we consider an alternate learning approach that trains models using "privileged" information--features available at training time but not at runtime--to improve the accuracy and resilience of detection systems.
Practical Black-Box Attacks against Machine Learning
Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN.
The Limitations of Deep Learning in Adversarial Settings
In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs.
Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks
In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs.
