Search Results for author: Nicolas Papernot

Found 99 papers, 44 papers with code

The Adversarial Implications of Variable-Time Inference

1 code implementation5 Sep 2023 Dudi Biton, Aditi Misra, Efrat Levy, Jaidip Kotak, Ron Bitton, Roei Schuster, Nicolas Papernot, Yuval Elovici, Ben Nassi

In our examination of the timing side-channel vulnerabilities associated with this algorithm, we identified the potential to enhance decision-based attacks.

object-detection Object Detection

LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?

no code implementations20 Jul 2023 David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan

Specifically, we demonstrate that semantic censorship can be perceived as an undecidable problem, highlighting the inherent challenges in censorship that arise due to LLMs' programmatic and instruction-following capabilities.

Computer Security Instruction Following

Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGD

no code implementations1 Jul 2023 Anvith Thudi, Hengrui Jia, Casey Meehan, Ilia Shumailov, Nicolas Papernot

In this paper, we develop a new analysis for DP-SGD that captures the intuition that points with similar neighbors in the dataset enjoy better privacy than outliers.

Augment then Smooth: Reconciling Differential Privacy with Certified Robustness

no code implementations14 Jun 2023 Jiapeng Wu, Atiyeh Ashari Ghomi, David Glukhov, Jesse C. Cresswell, Franziska Boenisch, Nicolas Papernot

Differential privacy and randomized smoothing are effective defenses that provide certifiable guarantees for each of these threats, however, it is not well understood how implementing either defense impacts the other.

When Vision Fails: Text Attacks Against ViT and OCR

1 code implementation12 Jun 2023 Nicholas Boucher, Jenny Blessing, Ilia Shumailov, Ross Anderson, Nicolas Papernot

While text-based machine learning models that operate on visual inputs of rendered text have become robust against a wide range of existing attacks, we show that they are still vulnerable to visual adversarial examples encoded as text.

Optical Character Recognition (OCR)

Training Private Models That Know What They Don't Know

no code implementations28 May 2023 Stephan Rabanser, Anvith Thudi, Abhradeep Thakurta, Krishnamurthy Dvijotham, Nicolas Papernot

Training reliable deep learning models which avoid making overconfident but incorrect predictions is a longstanding challenge.

The Curse of Recursion: Training on Generated Data Makes Models Forget

1 code implementation27 May 2023 Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson

It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images.


Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models

no code implementations24 May 2023 Haonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska Boenisch

To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent.

Inference Attack Membership Inference Attack

Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregation

no code implementations9 Jan 2023 Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, Nicolas Papernot

FL is promoted as a privacy-enhancing technology (PET) that provides data minimization: data never "leaves" personal devices and users share only model updates with a server (e. g., a company) coordinating the distributed training.

Federated Learning

Learned Systems Security

no code implementations20 Dec 2022 Roei Schuster, Jin Peng Zhou, Thorsten Eisenhofer, Paul Grubbs, Nicolas Papernot

We analyze the root causes of potentially-increased attack surface in learned systems and develop a framework for identifying vulnerabilities that stem from the use of ML.

Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search

no code implementations5 Oct 2022 Yannis Cattan, Christopher A. Choquette-Choo, Nicolas Papernot, Abhradeep Thakurta

For instance, we achieve 77. 9% accuracy for $(\varepsilon, \delta)=(2, 10^{-5})$ on CIFAR-100 for a model pretrained on ImageNet.

Privacy Preserving

In Differential Privacy, There is Truth: On Vote Leakage in Ensemble Private Learning

1 code implementation22 Sep 2022 Jiaqi Wang, Roei Schuster, Ilia Shumailov, David Lie, Nicolas Papernot

When learning from sensitive data, care must be taken to ensure that training algorithms address privacy concerns.

Dataset Inference for Self-Supervised Models

no code implementations16 Sep 2022 Adam Dziedzic, Haonan Duan, Muhammad Ahmad Kaleem, Nikita Dhawan, Jonas Guan, Yannis Cattan, Franziska Boenisch, Nicolas Papernot

We introduce a new dataset inference defense, which uses the private training set of the victim encoder model to attribute its ownership in the event of stealing.

Density Estimation

Proof-of-Learning is Currently More Broken Than You Think

no code implementations6 Aug 2022 Congyu Fang, Hengrui Jia, Anvith Thudi, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Varun Chandrasekaran, Nicolas Papernot

They empirically argued the benefit of this approach by showing how spoofing--computing a proof for a stolen model--is as expensive as obtaining the proof honestly by training the model.

Learning Theory

$p$-DkNN: Out-of-Distribution Detection Through Statistical Testing of Deep Representations

no code implementations25 Jul 2022 Adam Dziedzic, Stephan Rabanser, Mohammad Yaghini, Armin Ale, Murat A. Erdogdu, Nicolas Papernot

We introduce $p$-DkNN, a novel inference procedure that takes a trained deep neural network and analyzes the similarity structures of its intermediate hidden representations to compute $p$-values associated with the end-to-end model prediction.

Autonomous Driving Out-of-Distribution Detection +1

Efficient Adversarial Training With Data Pruning

no code implementations1 Jul 2022 Maximilian Kaufmann, Yiren Zhao, Ilia Shumailov, Robert Mullins, Nicolas Papernot

In this paper we demonstrate data pruning-a method for increasing adversarial training efficiency through data sub-sampling. We empirically show that data pruning leads to improvements in convergence and reliability of adversarial training, albeit with different levels of utility degradation.

Intrinsic Anomaly Detection for Multi-Variate Time Series

no code implementations29 Jun 2022 Stephan Rabanser, Tim Januschowski, Kashif Rasul, Oliver Borchert, Richard Kurle, Jan Gasthaus, Michael Bohlke-Schneider, Nicolas Papernot, Valentin Flunkert

We introduce a novel, practically relevant variation of the anomaly detection problem in multi-variate time series: intrinsic anomaly detection.

Anomaly Detection Navigate +3

On the Limitations of Stochastic Pre-processing Defenses

1 code implementation19 Jun 2022 Yue Gao, Ilia Shumailov, Kassem Fawaz, Nicolas Papernot

An example of such a defense is to apply a random transformation to inputs prior to feeding them to the model.

Adversarial Robustness

Selective Classification Via Neural Network Training Dynamics

no code implementations26 May 2022 Stephan Rabanser, Anvith Thudi, Kimia Hamidieh, Adam Dziedzic, Nicolas Papernot

Selective classification is the task of rejecting inputs a model would predict incorrectly on through a trade-off between input space coverage and model accuracy.


On the Difficulty of Defending Self-Supervised Learning against Model Extraction

1 code implementation16 May 2022 Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, Nicolas Papernot

We construct several novel attacks and find that approaches that train directly on a victim's stolen representations are query efficient and enable high accuracy for downstream models.

Model extraction Self-Supervised Learning

Bounding Membership Inference

no code implementations24 Feb 2022 Anvith Thudi, Ilia Shumailov, Franziska Boenisch, Nicolas Papernot

We find this greatly reduces the bound on MI positive accuracy.

Differentially Private Speaker Anonymization

no code implementations23 Feb 2022 Ali Shahin Shamsabadi, Brij Mohan Lal Srivastava, Aurélien Bellet, Nathalie Vauquier, Emmanuel Vincent, Mohamed Maouche, Marc Tommasi, Nicolas Papernot

We remove speaker information from these attributes by introducing differentially private feature extractors based on an autoencoder and an automatic speech recognizer, respectively, trained using noise layers.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +2

Increasing the Cost of Model Extraction with Calibrated Proof of Work

no code implementations ICLR 2022 Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, Nicolas Papernot

Since we calibrate the effort required to complete the proof-of-work to each query, this only introduces a slight overhead for regular users (up to 2x).

BIG-bench Machine Learning Model extraction

When the Curious Abandon Honesty: Federated Learning Is Not Private

1 code implementation6 Dec 2021 Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, Nicolas Papernot

Instead, these devices share gradients, parameters, or other model updates, with a central party (e. g., a company) coordinating the training.

Federated Learning Privacy Preserving +1

On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning

no code implementations22 Oct 2021 Anvith Thudi, Hengrui Jia, Ilia Shumailov, Nicolas Papernot

Machine unlearning, i. e. having a model forget about some of its training data, has become increasingly more important as privacy legislation promotes variants of the right-to-be-forgotten.

Hyperparameter Tuning with Renyi Differential Privacy

no code implementations ICLR 2022 Nicolas Papernot, Thomas Steinke

For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood.

Losing Less: A Loss for Differentially Private Deep Learning

no code implementations29 Sep 2021 Ali Shahin Shamsabadi, Nicolas Papernot

In this paper, we are the first to observe that some of this performance can be recovered when training with a loss tailored to DP-SGD; we challenge cross-entropy as the de facto loss for deep learning with DP.

Context-invariant, multi-variate time series representations

no code implementations29 Sep 2021 Stephan Rabanser, Tim Januschowski, Kashif Rasul, Oliver Borchert, Richard Kurle, Jan Gasthaus, Michael Bohlke-Schneider, Nicolas Papernot, Valentin Flunkert

Modern time series corpora, in particular those coming from sensor-based data, exhibit characteristics that have so far not been adequately addressed in the literature on representation learning for time series.

Contrastive Learning Representation Learning +2

A Zest of LIME: Towards Architecture-Independent Model Distances

no code implementations ICLR 2022 Hengrui Jia, Hongyu Chen, Jonas Guan, Ali Shahin Shamsabadi, Nicolas Papernot

In this paper, we instead propose to compute distance between black-box models by comparing their Local Interpretable Model-Agnostic Explanations (LIME).

Interpretability in Safety-Critical FinancialTrading Systems

no code implementations24 Sep 2021 Gabriel Deza, Adelin Travers, Colin Rowat, Nicolas Papernot

Indeed, we show in our evaluation that errors in the forecasting model's predictions alone are not sufficient for trading decisions made based on these forecasts to yield a negative return.


SoK: Machine Learning Governance

no code implementations20 Sep 2021 Varun Chandrasekaran, Hengrui Jia, Anvith Thudi, Adelin Travers, Mohammad Yaghini, Nicolas Papernot

The application of machine learning (ML) in computer systems introduces not only many benefits but also risks to society.

BIG-bench Machine Learning

Bad Characters: Imperceptible NLP Attacks

1 code implementation18 Jun 2021 Nicholas Boucher, Ilia Shumailov, Ross Anderson, Nicolas Papernot

In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs.

Machine Translation

Markpainting: Adversarial Machine Learning meets Inpainting

1 code implementation1 Jun 2021 David Khachaturov, Ilia Shumailov, Yiren Zhao, Nicolas Papernot, Ross Anderson

Inpainting is a learned interpolation technique that is based on generative modeling and used to populate masked or missing pieces in an image; it has wide applications in picture editing and retouching.

BIG-bench Machine Learning

Dataset Inference: Ownership Resolution in Machine Learning

1 code implementation ICLR 2021 Pratyush Maini, Mohammad Yaghini, Nicolas Papernot

We thus introduce $dataset$ $inference$, the process of identifying whether a suspected model copy has private knowledge from the original model's dataset, as a defense against model stealing.

BIG-bench Machine Learning

Proof-of-Learning: Definitions and Practice

2 code implementations9 Mar 2021 Hengrui Jia, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, Nicolas Papernot

In particular, our analyses and experiments show that an adversary seeking to illegitimately manufacture a proof-of-learning needs to perform *at least* as much work than is needed for gradient descent itself.

CaPC Learning: Confidential and Private Collaborative Learning

1 code implementation ICLR 2021 Christopher A. Choquette-Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, Xiao Wang

There is currently no method that enables machine learning in such a setting, where both confidentiality and privacy need to be preserved, to prevent both explicit and implicit sharing of data.

Fairness Federated Learning

Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning

no code implementations11 Jan 2021 Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, Nicholas Carlini

DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D' that differs in just one example. If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private.

BIG-bench Machine Learning

Neighbors From Hell: Voltage Attacks Against Deep Learning Accelerators on Multi-Tenant FPGAs

no code implementations14 Dec 2020 Andrew Boutros, Mathew Hall, Nicolas Papernot, Vaughn Betz

We find that even when using the strongest attacker circuit, the prediction accuracy of the DL accelerator is not compromised when running at its safe operating frequency.

Data-Free Model Extraction

2 code implementations CVPR 2021 Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, Nicolas Papernot

Current model extraction attacks assume that the adversary has access to a surrogate dataset with characteristics similar to the proprietary data used to train the victim model.

Model extraction Transfer Learning

Adversarial Examples in Constrained Domains

no code implementations2 Nov 2020 Ryan Sheatsley, Nicolas Papernot, Michael Weisman, Gunjan Verma, Patrick McDaniel

To assess how these algorithms perform, we evaluate them in constrained (e. g., network intrusion detection) and unconstrained (e. g., image recognition) domains.

Network Intrusion Detection

Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings

no code implementations13 Oct 2020 Vinith M. Suriyakumar, Nicolas Papernot, Anna Goldenberg, Marzyeh Ghassemi

Our results highlight lesser-known limitations of methods for DP learning in health care, models that exhibit steep tradeoffs between privacy and utility, and models whose predictions are disproportionately influenced by large demographic groups in the training data.

Fairness Mortality Prediction +3

On Attribution of Deepfakes

no code implementations20 Aug 2020 Baiwu Zhang, Jin Peng Zhou, Ilia Shumailov, Nicolas Papernot

We discuss the ethical implications of our work, identify where our technique can be used, and highlight that a more meaningful legislative framework is required for a more transparent and ethical use of generative modeling.

DeepFake Detection Face Generation +2

Tempered Sigmoid Activations for Deep Learning with Differential Privacy

1 code implementation28 Jul 2020 Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, Úlfar Erlingsson

Because learning sometimes involves sensitive data, machine learning algorithms have been extended to offer privacy for training data.

Privacy Preserving Privacy Preserving Deep Learning

Label-Only Membership Inference Attacks

1 code implementation28 Jul 2020 Christopher A. Choquette-Choo, Florian Tramer, Nicholas Carlini, Nicolas Papernot

We empirically show that our label-only membership inference attacks perform on par with prior attacks that required access to model confidences.

L2 Regularization

Sponge Examples: Energy-Latency Attacks on Neural Networks

1 code implementation5 Jun 2020 Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, Ross Anderson

The high energy costs of neural network training and inference led to the use of acceleration hardware such as GPUs and TPUs.

Autonomous Vehicles

SOAR: Second-Order Adversarial Regularization

no code implementations4 Apr 2020 Avery Ma, Fartash Faghri, Nicolas Papernot, Amir-Massoud Farahmand

Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples.

Adversarial Robustness

On the Robustness of Cooperative Multi-Agent Reinforcement Learning

1 code implementation8 Mar 2020 Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, Nicolas Papernot

Our results on the StartCraft II multi-agent benchmark demonstrate that c-MARL teams are highly vulnerable to perturbations applied to one of their agent's observations.

Multi-agent Reinforcement Learning reinforcement-learning +1

Entangled Watermarks as a Defense against Model Extraction

2 code implementations27 Feb 2020 Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, Nicolas Papernot

Such pairs are watermarks, which are not sampled from the task distribution and are only known to the defender.

Model extraction Transfer Learning

On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping

1 code implementation26 Feb 2020 Sanghyun Hong, Varun Chandrasekaran, Yiğitcan Kaya, Tudor Dumitraş, Nicolas Papernot

In this work, we study the feasibility of an attack-agnostic defense relying on artifacts that are common to all poisoning attacks.

Data Poisoning

Machine Unlearning

2 code implementations9 Dec 2019 Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot

Once users have shared their data online, it is generally difficult for them to revoke access and ask for the data to be deleted.

Transfer Learning

Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

no code implementations29 Oct 2019 Nicholas Carlini, Úlfar Erlingsson, Nicolas Papernot

We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution.

Adversarial Robustness BIG-bench Machine Learning

Thieves on Sesame Street! Model Extraction of BERT-based APIs

1 code implementation ICLR 2020 Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, Mohit Iyyer

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.

Language Modelling Model extraction +3

Improving Differentially Private Models with Active Learning

no code implementations2 Oct 2019 Zhengli Zhao, Nicolas Papernot, Sameer Singh, Neoklis Polyzotis, Augustus Odena

Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.

Active Learning

Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy

no code implementations25 Sep 2019 Nicolas Papernot, Steve Chien, Shuang Song, Abhradeep Thakurta, Ulfar Erlingsson

Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data.

Privacy Preserving

High Accuracy and High Fidelity Extraction of Neural Networks

no code implementations3 Sep 2019 Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, Nicolas Papernot

In a model extraction attack, an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access.

Model extraction Vocal Bursts Intensity Prediction

How Relevant is the Turing Test in the Age of Sophisbots?

no code implementations30 Aug 2019 Dan Boneh, Andrew J. Grotto, Patrick McDaniel, Nicolas Papernot

Popular culture has contemplated societies of thinking machines for generations, envisioning futures from utopian to dystopian.

Cultural Vocal Bursts Intensity Prediction

Rearchitecting Classification Frameworks For Increased Robustness

no code implementations26 May 2019 Varun Chandrasekaran, Brian Tang, Nicolas Papernot, Kassem Fawaz, Somesh Jha, Xi Wu

and how to design a classification paradigm that leverages these invariances to improve the robustness accuracy trade-off?

Autonomous Driving Classification +1

Prototypical Examples in Deep Learning: Metrics, Characteristics, and Utility

no code implementations ICLR 2019 Nicholas Carlini, Ulfar Erlingsson, Nicolas Papernot

Machine learning (ML) research has investigated prototypes: examples that are representative of the behavior to be learned.

Adversarial Robustness

Analyzing and Improving Representations with the Soft Nearest Neighbor Loss

4 code implementations5 Feb 2019 Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton

We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i. e., how close pairs of points from the same class are relative to pairs of points from different classes.


A General Approach to Adding Differential Privacy to Iterative Training Procedures

4 code implementations15 Dec 2018 H. Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, Peter Kairouz

In this work we address the practical challenges of training machine learning models on privacy-sensitive datasets by introducing a modular approach that minimizes changes to training algorithms, provides a variety of configuration strategies for the privacy mechanism, and then isolates and simplifies the critical logic that computes the final privacy guarantees.

A Marauder's Map of Security and Privacy in Machine Learning

1 code implementation3 Nov 2018 Nicolas Papernot

We structure our discussion around three of these directions, which we believe are likely to lead to significant progress.

Cryptography and Security

Adversarial Vision Challenge

2 code implementations6 Aug 2018 Wieland Brendel, Jonas Rauber, Alexey Kurakin, Nicolas Papernot, Behar Veliqi, Marcel Salathé, Sharada P. Mohanty, Matthias Bethge

The NIPS 2018 Adversarial Vision Challenge is a competition to facilitate measurable progress towards robust machine vision models and more generally applicable adversarial attacks.

Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning

4 code implementations13 Mar 2018 Nicolas Papernot, Patrick McDaniel

However, deep learning is often criticized for its lack of robustness in adversarial settings (e. g., vulnerability to adversarial inputs) and general inability to rationalize its predictions.

Machine Translation Malware Detection

Adversarial Examples that Fool both Computer Vision and Time-Limited Humans

no code implementations NeurIPS 2018 Gamaleldin F. Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian Goodfellow, Jascha Sohl-Dickstein

Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich.

BIG-bench Machine Learning Open-Ended Question Answering

On the Protection of Private Information in Machine Learning Systems: Two Recent Approaches

no code implementations26 Aug 2017 Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Nicolas Papernot, Kunal Talwar, Li Zhang

The recent, remarkable growth of machine learning has led to intense interest in the privacy of the data on which machine learning relies, and to new techniques for preserving privacy.

BIG-bench Machine Learning

Ensemble Adversarial Training: Attacks and Defenses

11 code implementations ICLR 2018 Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel

We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss.

Extending Defensive Distillation

1 code implementation15 May 2017 Nicolas Papernot, Patrick McDaniel

Machine learning is vulnerable to adversarial examples: inputs carefully modified to force misclassification.

BIG-bench Machine Learning

The Space of Transferable Adversarial Examples

2 code implementations11 Apr 2017 Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel

Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time.

On the (Statistical) Detection of Adversarial Examples

no code implementations21 Feb 2017 Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, Patrick McDaniel

Specifically, we augment our ML model with an additional output, in which the model is trained to classify all adversarial inputs.

Malware Classification Network Intrusion Detection

Adversarial Attacks on Neural Network Policies

1 code implementation8 Feb 2017 Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel

Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification.

Towards the Science of Security and Privacy in Machine Learning

no code implementations11 Nov 2016 Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, Michael Wellman

Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics.

BIG-bench Machine Learning Decision Making

Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data

8 code implementations18 Oct 2016 Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal Talwar

The approach combines, in a black-box fashion, multiple models trained with disjoint datasets, such as records from different subsets of users.

Transfer Learning

Adversarial Perturbations Against Deep Neural Networks for Malware Classification

no code implementations14 Jun 2016 Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, Patrick McDaniel

Deep neural networks, like many other machine learning models, have recently been shown to lack robustness against adversarially crafted inputs.

BIG-bench Machine Learning Classification +3

Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

no code implementations24 May 2016 Nicolas Papernot, Patrick McDaniel, Ian Goodfellow

We demonstrate our attacks on two commercial machine learning classification systems from Amazon (96. 19% misclassification rate) and Google (88. 94%) using only 800 queries of the victim model, thereby showing that existing machine learning approaches are in general vulnerable to systematic black-box attacks regardless of their structure.

BIG-bench Machine Learning

Crafting Adversarial Input Sequences for Recurrent Neural Networks

1 code implementation28 Apr 2016 Nicolas Papernot, Patrick McDaniel, Ananthram Swami, Richard Harang

Machine learning models are frequently used to solve complex security problems, as well as to make decisions in sensitive situations like guiding autonomous vehicles or predicting financial market behaviors.

Autonomous Vehicles BIG-bench Machine Learning +1

Detection under Privileged Information

no code implementations31 Mar 2016 Z. Berkay Celik, Patrick McDaniel, Rauf Izmailov, Nicolas Papernot, Ryan Sheatsley, Raquel Alvarez, Ananthram Swami

In this paper, we consider an alternate learning approach that trains models using "privileged" information--features available at training time but not at runtime--to improve the accuracy and resilience of detection systems.

Face Recognition Malware Classification +1

Practical Black-Box Attacks against Machine Learning

17 code implementations8 Feb 2016 Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram Swami

Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN.

BIG-bench Machine Learning

The Limitations of Deep Learning in Adversarial Settings

11 code implementations24 Nov 2015 Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami

In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs.

Adversarial Attack Adversarial Defense

Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks

2 code implementations14 Nov 2015 Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, Ananthram Swami

In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs.

Autonomous Vehicles BIG-bench Machine Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.