no code implementations • 16 May 2022 • Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, Nicolas Papernot
We construct several novel attacks and find that approaches that train directly on a victim's stolen representations are query efficient and enable high accuracy for downstream models.
no code implementations • ICLR 2022 • Natalie Dullerud, Karsten Roth, Kimia Hamidieh, Nicolas Papernot, Marzyeh Ghassemi
Deep metric learning (DML) enables learning with less supervision through its emphasis on the similarity structure of representations.
no code implementations • 24 Feb 2022 • Anvith Thudi, Ilia Shumailov, Franziska Boenisch, Nicolas Papernot
In practice, this means that models need to be trained with DP guarantees that greatly decrease their accuracy.
no code implementations • 23 Feb 2022 • Ali Shahin Shamsabadi, Brij Mohan Lal Srivastava, Aurélien Bellet, Nathalie Vauquier, Emmanuel Vincent, Mohamed Maouche, Marc Tommasi, Nicolas Papernot
We remove speaker information from these attributes by introducing differentially private feature extractors based on an autoencoder and an automatic speech recognizer, respectively, trained using noise layers.
no code implementations • 6 Feb 2022 • Shimaa Ahmed, Yash Wani, Ali Shahin Shamsabadi, Mohammad Yaghini, Ilia Shumailov, Nicolas Papernot, Kassem Fawaz
Recent years have seen a surge of popularity of acoustics-enabled personal devices powered by machine learning.
no code implementations • ICLR 2022 • Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, Nicolas Papernot
Since we calibrate the effort required to complete the proof-of-work to each query, this only introduces a slight overhead for regular users (up to 2x).
1 code implementation • 6 Dec 2021 • Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, Nicolas Papernot
This is because prior efforts exclusively consider passive attackers that are honest-but-curious.
no code implementations • 22 Oct 2021 • Anvith Thudi, Hengrui Jia, Ilia Shumailov, Nicolas Papernot
Machine unlearning, i. e. having a model forget about some of its training data, has become increasingly more important as privacy legislation promotes variants of the right-to-be-forgotten.
no code implementations • ICLR 2022 • Nicolas Papernot, Thomas Steinke
For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood.
no code implementations • 29 Sep 2021 • Adam Dziedzic, Christopher A. Choquette-Choo, Natalie Dullerud, Vinith Menon Suriyakumar, Ali Shahin Shamsabadi, Muhammad Ahmad Kaleem, Somesh Jha, Nicolas Papernot, Xiao Wang
We find that our techniques outperform all others in the centralized setting.
no code implementations • ICLR 2022 • Hengrui Jia, Hongyu Chen, Jonas Guan, Ali Shahin Shamsabadi, Nicolas Papernot
In this paper, we instead propose to compute distance between black-box models by comparing their Local Interpretable Model-Agnostic Explanations (LIME).
no code implementations • 29 Sep 2021 • Ali Shahin Shamsabadi, Nicolas Papernot
In this paper, we are the first to observe that some of this performance can be recovered when training with a loss tailored to DP-SGD; we challenge cross-entropy as the de facto loss for deep learning with DP.
no code implementations • 29 Sep 2021 • Stephan Rabanser, Tim Januschowski, Kashif Rasul, Oliver Borchert, Richard Kurle, Jan Gasthaus, Michael Bohlke-Schneider, Nicolas Papernot, Valentin Flunkert
Modern time series corpora, in particular those coming from sensor-based data, exhibit characteristics that have so far not been adequately addressed in the literature on representation learning for time series.
1 code implementation • 27 Sep 2021 • Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, Nicolas Papernot
In this work, we first taxonomize approaches and metrics of approximate unlearning.
no code implementations • 24 Sep 2021 • Gabriel Deza, Adelin Travers, Colin Rowat, Nicolas Papernot
Indeed, we show in our evaluation that errors in the forecasting model's predictions alone are not sufficient for trading decisions made based on these forecasts to yield a negative return.
no code implementations • 20 Sep 2021 • Varun Chandrasekaran, Hengrui Jia, Anvith Thudi, Adelin Travers, Mohammad Yaghini, Nicolas Papernot
The application of machine learning (ML) in computer systems introduces not only many benefits but also risks to society.
no code implementations • 3 Aug 2021 • Adelin Travers, Lorna Licollari, Guanghan Wang, Varun Chandrasekaran, Adam Dziedzic, David Lie, Nicolas Papernot
In the white-box setting, we instantiate this class with a joint, multi-stage optimization attack.
1 code implementation • 18 Jun 2021 • Nicholas Boucher, Ilia Shumailov, Ross Anderson, Nicolas Papernot
In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs.
1 code implementation • 1 Jun 2021 • David Khachaturov, Ilia Shumailov, Yiren Zhao, Nicolas Papernot, Ross Anderson
Inpainting is a learned interpolation technique that is based on generative modeling and used to populate masked or missing pieces in an image; it has wide applications in picture editing and retouching.
1 code implementation • ICLR 2021 • Pratyush Maini, Mohammad Yaghini, Nicolas Papernot
We thus introduce $dataset$ $inference$, the process of identifying whether a suspected model copy has private knowledge from the original model's dataset, as a defense against model stealing.
1 code implementation • NeurIPS 2021 • Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A. Erdogdu, Ross Anderson
Machine learning is vulnerable to a wide variety of attacks.
1 code implementation • 9 Mar 2021 • Hengrui Jia, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, Nicolas Papernot
In particular, our analyses and experiments show that an adversary seeking to illegitimately manufacture a proof-of-learning needs to perform *at least* as much work than is needed for gradient descent itself.
1 code implementation • ICLR 2021 • Christopher A. Choquette-Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, Xiao Wang
There is currently no method that enables machine learning in such a setting, where both confidentiality and privacy need to be preserved, to prevent both explicit and implicit sharing of data.
no code implementations • 11 Jan 2021 • Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, Nicholas Carlini
DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D' that differs in just one example. If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private.
no code implementations • 1 Jan 2021 • Gabriel Deza, Colin Rowat, Nicolas Papernot
Machine learning (ML) models are known to be vulnerable to attacks both at training and test time.
no code implementations • 14 Dec 2020 • Andrew Boutros, Mathew Hall, Nicolas Papernot, Vaughn Betz
We find that even when using the strongest attacker circuit, the prediction accuracy of the DL accelerator is not compromised when running at its safe operating frequency.
1 code implementation • CVPR 2021 • Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, Nicolas Papernot
Current model extraction attacks assume that the adversary has access to a surrogate dataset with characteristics similar to the proprietary data used to train the victim model.
no code implementations • 2 Nov 2020 • Ryan Sheatsley, Nicolas Papernot, Michael Weisman, Gunjan Verma, Patrick McDaniel
To assess how these algorithms perform, we evaluate them in constrained (e. g., network intrusion detection) and unconstrained (e. g., image recognition) domains.
no code implementations • 13 Oct 2020 • Vinith M. Suriyakumar, Nicolas Papernot, Anna Goldenberg, Marzyeh Ghassemi
Our results highlight lesser-known limitations of methods for DP learning in health care, models that exhibit steep tradeoffs between privacy and utility, and models whose predictions are disproportionately influenced by large demographic groups in the training data.
no code implementations • 20 Aug 2020 • Baiwu Zhang, Jin Peng Zhou, Ilia Shumailov, Nicolas Papernot
We discuss the ethical implications of our work, identify where our technique can be used, and highlight that a more meaningful legislative framework is required for a more transparent and ethical use of generative modeling.
no code implementations • 28 Jul 2020 • Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, Úlfar Erlingsson
Because learning sometimes involves sensitive data, machine learning algorithms have been extended to offer privacy for training data.
1 code implementation • 28 Jul 2020 • Christopher A. Choquette-Choo, Florian Tramer, Nicholas Carlini, Nicolas Papernot
We empirically show that our label-only membership inference attacks perform on par with prior attacks that required access to model confidences.
no code implementations • 13 Jul 2020 • Hadi Abdullah, Kevin Warren, Vincent Bindschaedler, Nicolas Papernot, Patrick Traynor
Like other systems based on neural networks, recent research has demonstrated that speech and speaker recognition systems are vulnerable to attacks using manipulated inputs.
1 code implementation • 5 Jun 2020 • Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, Ross Anderson
The high energy costs of neural network training and inference led to the use of acceleration hardware such as GPUs and TPUs.
no code implementations • 4 Apr 2020 • Avery Ma, Fartash Faghri, Nicolas Papernot, Amir-Massoud Farahmand
Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples.
no code implementations • 8 Mar 2020 • Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, Nicolas Papernot
Our results on the StartCraft II multi-agent benchmark demonstrate that c-MARL teams are highly vulnerable to perturbations applied to one of their agent's observations.
1 code implementation • 27 Feb 2020 • Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, Nicolas Papernot
Such pairs are watermarks, which are not sampled from the task distribution and are only known to the defender.
1 code implementation • 26 Feb 2020 • Sanghyun Hong, Varun Chandrasekaran, Yiğitcan Kaya, Tudor Dumitraş, Nicolas Papernot
In this work, we study the feasibility of an attack-agnostic defense relying on artifacts that are common to all poisoning attacks.
1 code implementation • ICML 2020 • Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, Jörn-Henrik Jacobsen
Adversarial examples are malicious inputs crafted to induce misclassification.
2 code implementations • 9 Dec 2019 • Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot
Once users have shared their data online, it is generally difficult for them to revoke access and ask for the data to be deleted.
no code implementations • 29 Oct 2019 • Nicholas Carlini, Úlfar Erlingsson, Nicolas Papernot
We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution.
1 code implementation • ICLR 2020 • Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, Mohit Iyyer
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.
no code implementations • 2 Oct 2019 • Zhengli Zhao, Nicolas Papernot, Sameer Singh, Neoklis Polyzotis, Augustus Odena
Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.
no code implementations • 25 Sep 2019 • Nicolas Papernot, Steve Chien, Shuang Song, Abhradeep Thakurta, Ulfar Erlingsson
Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data.
no code implementations • 3 Sep 2019 • Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, Nicolas Papernot
In a model extraction attack, an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access.
no code implementations • 30 Aug 2019 • Dan Boneh, Andrew J. Grotto, Patrick McDaniel, Nicolas Papernot
Popular culture has contemplated societies of thinking machines for generations, envisioning futures from utopian to dystopian.
no code implementations • 26 May 2019 • Varun Chandrasekaran, Brian Tang, Nicolas Papernot, Kassem Fawaz, Somesh Jha, Xi Wu
and how to design a classification paradigm that leverages these invariances to improve the robustness accuracy trade-off?
28 code implementations • NeurIPS 2019 • David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel
Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets.
no code implementations • ICLR 2019 • Nicholas Carlini, Ulfar Erlingsson, Nicolas Papernot
Machine learning (ML) research has investigated prototypes: examples that are representative of the behavior to be learned.
no code implementations • 25 Mar 2019 • Jörn-Henrik Jacobsen, Jens Behrmannn, Nicholas Carlini, Florian Tramèr, Nicolas Papernot
Excessive invariance is not limited to models trained to be robust to perturbation-based $\ell_p$-norm adversaries.
4 code implementations • 18 Feb 2019 • Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, Alexey Kurakin
Correctly evaluating defenses against adversarial examples has proven to be extremely difficult.
2 code implementations • 5 Feb 2019 • Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton
We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i. e., how close pairs of points from the same class are relative to pairs of points from different classes.
4 code implementations • 15 Dec 2018 • H. Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, Peter Kairouz
In this work we address the practical challenges of training machine learning models on privacy-sensitive datasets by introducing a modular approach that minimizes changes to training algorithms, provides a variety of configuration strategies for the privacy mechanism, and then isolates and simplifies the critical logic that computes the final privacy guarantees.
1 code implementation • 3 Nov 2018 • Nicolas Papernot
We structure our discussion around three of these directions, which we believe are likely to lead to significant progress.
Cryptography and Security
2 code implementations • 6 Aug 2018 • Wieland Brendel, Jonas Rauber, Alexey Kurakin, Nicolas Papernot, Behar Veliqi, Marcel Salathé, Sharada P. Mohanty, Matthias Bethge
The NIPS 2018 Adversarial Vision Challenge is a competition to facilitate measurable progress towards robust machine vision models and more generally applicable adversarial attacks.
4 code implementations • 13 Mar 2018 • Nicolas Papernot, Patrick McDaniel
However, deep learning is often criticized for its lack of robustness in adversarial settings (e. g., vulnerability to adversarial inputs) and general inability to rationalize its predictions.
2 code implementations • ICLR 2018 • Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, Úlfar Erlingsson
Models and examples built with TensorFlow
no code implementations • NeurIPS 2018 • Gamaleldin F. Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian Goodfellow, Jascha Sohl-Dickstein
Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich.
no code implementations • 26 Aug 2017 • Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Nicolas Papernot, Kunal Talwar, Li Zhang
The recent, remarkable growth of machine learning has led to intense interest in the privacy of the data on which machine learning relies, and to new techniques for preserving privacy.
11 code implementations • ICLR 2018 • Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel
We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss.
no code implementations • 15 May 2017 • Nicolas Papernot, Patrick McDaniel
Machine learning is vulnerable to adversarial examples: inputs carefully modified to force misclassification.
1 code implementation • 11 Apr 2017 • Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel
Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time.
no code implementations • 21 Feb 2017 • Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, Patrick McDaniel
Specifically, we augment our ML model with an additional output, in which the model is trained to classify all adversarial inputs.
no code implementations • 8 Feb 2017 • Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel
Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification.
no code implementations • 11 Nov 2016 • Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, Michael Wellman
Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics.
7 code implementations • 18 Oct 2016 • Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal Talwar
The approach combines, in a black-box fashion, multiple models trained with disjoint datasets, such as records from different subsets of users.
14 code implementations • 3 Oct 2016 • Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, Rujun Long, Patrick McDaniel
An adversarial example library for constructing attacks, building defenses, and benchmarking both
no code implementations • 14 Jun 2016 • Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, Patrick McDaniel
Deep neural networks, like many other machine learning models, have recently been shown to lack robustness against adversarially crafted inputs.
no code implementations • 24 May 2016 • Nicolas Papernot, Patrick McDaniel, Ian Goodfellow
We demonstrate our attacks on two commercial machine learning classification systems from Amazon (96. 19% misclassification rate) and Google (88. 94%) using only 800 queries of the victim model, thereby showing that existing machine learning approaches are in general vulnerable to systematic black-box attacks regardless of their structure.
1 code implementation • 28 Apr 2016 • Nicolas Papernot, Patrick McDaniel, Ananthram Swami, Richard Harang
Machine learning models are frequently used to solve complex security problems, as well as to make decisions in sensitive situations like guiding autonomous vehicles or predicting financial market behaviors.
no code implementations • 31 Mar 2016 • Z. Berkay Celik, Patrick McDaniel, Rauf Izmailov, Nicolas Papernot, Ryan Sheatsley, Raquel Alvarez, Ananthram Swami
In this paper, we consider an alternate learning approach that trains models using "privileged" information--features available at training time but not at runtime--to improve the accuracy and resilience of detection systems.
17 code implementations • 8 Feb 2016 • Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram Swami
Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN.
12 code implementations • 24 Nov 2015 • Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami
In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs.
1 code implementation • 14 Nov 2015 • Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, Ananthram Swami
In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs.