Search Results for author: Nicolas Papernot

Found 74 papers, 32 papers with code

On the Difficulty of Defending Self-Supervised Learning against Model Extraction

no code implementations16 May 2022 Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, Nicolas Papernot

We construct several novel attacks and find that approaches that train directly on a victim's stolen representations are query efficient and enable high accuracy for downstream models.

Model extraction Self-Supervised Learning

Bounding Membership Inference

no code implementations24 Feb 2022 Anvith Thudi, Ilia Shumailov, Franziska Boenisch, Nicolas Papernot

In practice, this means that models need to be trained with DP guarantees that greatly decrease their accuracy.

Differentially Private Speaker Anonymization

no code implementations23 Feb 2022 Ali Shahin Shamsabadi, Brij Mohan Lal Srivastava, Aurélien Bellet, Nathalie Vauquier, Emmanuel Vincent, Mohamed Maouche, Marc Tommasi, Nicolas Papernot

We remove speaker information from these attributes by introducing differentially private feature extractors based on an autoencoder and an automatic speech recognizer, respectively, trained using noise layers.

Automatic Speech Recognition Disentanglement

Increasing the Cost of Model Extraction with Calibrated Proof of Work

no code implementations ICLR 2022 Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, Nicolas Papernot

Since we calibrate the effort required to complete the proof-of-work to each query, this only introduces a slight overhead for regular users (up to 2x).

Model extraction

On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning

no code implementations22 Oct 2021 Anvith Thudi, Hengrui Jia, Ilia Shumailov, Nicolas Papernot

Machine unlearning, i. e. having a model forget about some of its training data, has become increasingly more important as privacy legislation promotes variants of the right-to-be-forgotten.

Hyperparameter Tuning with Renyi Differential Privacy

no code implementations ICLR 2022 Nicolas Papernot, Thomas Steinke

For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood.

A Zest of LIME: Towards Architecture-Independent Model Distances

no code implementations ICLR 2022 Hengrui Jia, Hongyu Chen, Jonas Guan, Ali Shahin Shamsabadi, Nicolas Papernot

In this paper, we instead propose to compute distance between black-box models by comparing their Local Interpretable Model-Agnostic Explanations (LIME).

Losing Less: A Loss for Differentially Private Deep Learning

no code implementations29 Sep 2021 Ali Shahin Shamsabadi, Nicolas Papernot

In this paper, we are the first to observe that some of this performance can be recovered when training with a loss tailored to DP-SGD; we challenge cross-entropy as the de facto loss for deep learning with DP.

Context-invariant, multi-variate time series representations

no code implementations29 Sep 2021 Stephan Rabanser, Tim Januschowski, Kashif Rasul, Oliver Borchert, Richard Kurle, Jan Gasthaus, Michael Bohlke-Schneider, Nicolas Papernot, Valentin Flunkert

Modern time series corpora, in particular those coming from sensor-based data, exhibit characteristics that have so far not been adequately addressed in the literature on representation learning for time series.

Contrastive Learning Representation Learning +1

Interpretability in Safety-Critical FinancialTrading Systems

no code implementations24 Sep 2021 Gabriel Deza, Adelin Travers, Colin Rowat, Nicolas Papernot

Indeed, we show in our evaluation that errors in the forecasting model's predictions alone are not sufficient for trading decisions made based on these forecasts to yield a negative return.

SoK: Machine Learning Governance

no code implementations20 Sep 2021 Varun Chandrasekaran, Hengrui Jia, Anvith Thudi, Adelin Travers, Mohammad Yaghini, Nicolas Papernot

The application of machine learning (ML) in computer systems introduces not only many benefits but also risks to society.

Bad Characters: Imperceptible NLP Attacks

1 code implementation18 Jun 2021 Nicholas Boucher, Ilia Shumailov, Ross Anderson, Nicolas Papernot

In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs.

Machine Translation

Markpainting: Adversarial Machine Learning meets Inpainting

1 code implementation1 Jun 2021 David Khachaturov, Ilia Shumailov, Yiren Zhao, Nicolas Papernot, Ross Anderson

Inpainting is a learned interpolation technique that is based on generative modeling and used to populate masked or missing pieces in an image; it has wide applications in picture editing and retouching.

Dataset Inference: Ownership Resolution in Machine Learning

1 code implementation ICLR 2021 Pratyush Maini, Mohammad Yaghini, Nicolas Papernot

We thus introduce $dataset$ $inference$, the process of identifying whether a suspected model copy has private knowledge from the original model's dataset, as a defense against model stealing.

Proof-of-Learning: Definitions and Practice

1 code implementation9 Mar 2021 Hengrui Jia, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, Nicolas Papernot

In particular, our analyses and experiments show that an adversary seeking to illegitimately manufacture a proof-of-learning needs to perform *at least* as much work than is needed for gradient descent itself.

CaPC Learning: Confidential and Private Collaborative Learning

1 code implementation ICLR 2021 Christopher A. Choquette-Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, Xiao Wang

There is currently no method that enables machine learning in such a setting, where both confidentiality and privacy need to be preserved, to prevent both explicit and implicit sharing of data.

Fairness Federated Learning

Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning

no code implementations11 Jan 2021 Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, Nicholas Carlini

DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D' that differs in just one example. If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private.

Neighbors From Hell: Voltage Attacks Against Deep Learning Accelerators on Multi-Tenant FPGAs

no code implementations14 Dec 2020 Andrew Boutros, Mathew Hall, Nicolas Papernot, Vaughn Betz

We find that even when using the strongest attacker circuit, the prediction accuracy of the DL accelerator is not compromised when running at its safe operating frequency.

Data-Free Model Extraction

1 code implementation CVPR 2021 Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, Nicolas Papernot

Current model extraction attacks assume that the adversary has access to a surrogate dataset with characteristics similar to the proprietary data used to train the victim model.

Model extraction Transfer Learning

Adversarial Examples in Constrained Domains

no code implementations2 Nov 2020 Ryan Sheatsley, Nicolas Papernot, Michael Weisman, Gunjan Verma, Patrick McDaniel

To assess how these algorithms perform, we evaluate them in constrained (e. g., network intrusion detection) and unconstrained (e. g., image recognition) domains.

Network Intrusion Detection

Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings

no code implementations13 Oct 2020 Vinith M. Suriyakumar, Nicolas Papernot, Anna Goldenberg, Marzyeh Ghassemi

Our results highlight lesser-known limitations of methods for DP learning in health care, models that exhibit steep tradeoffs between privacy and utility, and models whose predictions are disproportionately influenced by large demographic groups in the training data.

Fairness Mortality Prediction +1

On Attribution of Deepfakes

no code implementations20 Aug 2020 Baiwu Zhang, Jin Peng Zhou, Ilia Shumailov, Nicolas Papernot

We discuss the ethical implications of our work, identify where our technique can be used, and highlight that a more meaningful legislative framework is required for a more transparent and ethical use of generative modeling.

DeepFake Detection Face Generation +2

Tempered Sigmoid Activations for Deep Learning with Differential Privacy

no code implementations28 Jul 2020 Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, Úlfar Erlingsson

Because learning sometimes involves sensitive data, machine learning algorithms have been extended to offer privacy for training data.

Privacy Preserving Deep Learning

Label-Only Membership Inference Attacks

1 code implementation28 Jul 2020 Christopher A. Choquette-Choo, Florian Tramer, Nicholas Carlini, Nicolas Papernot

We empirically show that our label-only membership inference attacks perform on par with prior attacks that required access to model confidences.

L2 Regularization

SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systems

no code implementations13 Jul 2020 Hadi Abdullah, Kevin Warren, Vincent Bindschaedler, Nicolas Papernot, Patrick Traynor

Like other systems based on neural networks, recent research has demonstrated that speech and speaker recognition systems are vulnerable to attacks using manipulated inputs.

Automatic Speech Recognition Speaker Identification +1

Sponge Examples: Energy-Latency Attacks on Neural Networks

1 code implementation5 Jun 2020 Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, Ross Anderson

The high energy costs of neural network training and inference led to the use of acceleration hardware such as GPUs and TPUs.

Autonomous Vehicles

SOAR: Second-Order Adversarial Regularization

no code implementations4 Apr 2020 Avery Ma, Fartash Faghri, Nicolas Papernot, Amir-Massoud Farahmand

Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples.

Adversarial Robustness

On the Robustness of Cooperative Multi-Agent Reinforcement Learning

no code implementations8 Mar 2020 Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, Nicolas Papernot

Our results on the StartCraft II multi-agent benchmark demonstrate that c-MARL teams are highly vulnerable to perturbations applied to one of their agent's observations.

Multi-agent Reinforcement Learning reinforcement-learning

Entangled Watermarks as a Defense against Model Extraction

1 code implementation27 Feb 2020 Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, Nicolas Papernot

Such pairs are watermarks, which are not sampled from the task distribution and are only known to the defender.

Model extraction Transfer Learning

On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping

1 code implementation26 Feb 2020 Sanghyun Hong, Varun Chandrasekaran, Yiğitcan Kaya, Tudor Dumitraş, Nicolas Papernot

In this work, we study the feasibility of an attack-agnostic defense relying on artifacts that are common to all poisoning attacks.

Data Poisoning

Machine Unlearning

2 code implementations9 Dec 2019 Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot

Once users have shared their data online, it is generally difficult for them to revoke access and ask for the data to be deleted.

Transfer Learning

Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

no code implementations29 Oct 2019 Nicholas Carlini, Úlfar Erlingsson, Nicolas Papernot

We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution.

Adversarial Robustness

Thieves on Sesame Street! Model Extraction of BERT-based APIs

1 code implementation ICLR 2020 Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, Mohit Iyyer

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.

Language Modelling Model extraction +3

Improving Differentially Private Models with Active Learning

no code implementations2 Oct 2019 Zhengli Zhao, Nicolas Papernot, Sameer Singh, Neoklis Polyzotis, Augustus Odena

Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.

Active Learning

Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy

no code implementations25 Sep 2019 Nicolas Papernot, Steve Chien, Shuang Song, Abhradeep Thakurta, Ulfar Erlingsson

Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data.

High Accuracy and High Fidelity Extraction of Neural Networks

no code implementations3 Sep 2019 Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, Nicolas Papernot

In a model extraction attack, an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access.

Model extraction

How Relevant is the Turing Test in the Age of Sophisbots?

no code implementations30 Aug 2019 Dan Boneh, Andrew J. Grotto, Patrick McDaniel, Nicolas Papernot

Popular culture has contemplated societies of thinking machines for generations, envisioning futures from utopian to dystopian.

Rearchitecting Classification Frameworks For Increased Robustness

no code implementations26 May 2019 Varun Chandrasekaran, Brian Tang, Nicolas Papernot, Kassem Fawaz, Somesh Jha, Xi Wu

and how to design a classification paradigm that leverages these invariances to improve the robustness accuracy trade-off?

Autonomous Driving Classification +1

Prototypical Examples in Deep Learning: Metrics, Characteristics, and Utility

no code implementations ICLR 2019 Nicholas Carlini, Ulfar Erlingsson, Nicolas Papernot

Machine learning (ML) research has investigated prototypes: examples that are representative of the behavior to be learned.

Adversarial Robustness

Analyzing and Improving Representations with the Soft Nearest Neighbor Loss

2 code implementations5 Feb 2019 Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton

We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i. e., how close pairs of points from the same class are relative to pairs of points from different classes.

A General Approach to Adding Differential Privacy to Iterative Training Procedures

4 code implementations15 Dec 2018 H. Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, Peter Kairouz

In this work we address the practical challenges of training machine learning models on privacy-sensitive datasets by introducing a modular approach that minimizes changes to training algorithms, provides a variety of configuration strategies for the privacy mechanism, and then isolates and simplifies the critical logic that computes the final privacy guarantees.

A Marauder's Map of Security and Privacy in Machine Learning

1 code implementation3 Nov 2018 Nicolas Papernot

We structure our discussion around three of these directions, which we believe are likely to lead to significant progress.

Cryptography and Security

Adversarial Vision Challenge

2 code implementations6 Aug 2018 Wieland Brendel, Jonas Rauber, Alexey Kurakin, Nicolas Papernot, Behar Veliqi, Marcel Salathé, Sharada P. Mohanty, Matthias Bethge

The NIPS 2018 Adversarial Vision Challenge is a competition to facilitate measurable progress towards robust machine vision models and more generally applicable adversarial attacks.

Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning

4 code implementations13 Mar 2018 Nicolas Papernot, Patrick McDaniel

However, deep learning is often criticized for its lack of robustness in adversarial settings (e. g., vulnerability to adversarial inputs) and general inability to rationalize its predictions.

Machine Translation Malware Detection

Adversarial Examples that Fool both Computer Vision and Time-Limited Humans

no code implementations NeurIPS 2018 Gamaleldin F. Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian Goodfellow, Jascha Sohl-Dickstein

Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich.

On the Protection of Private Information in Machine Learning Systems: Two Recent Approaches

no code implementations26 Aug 2017 Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Nicolas Papernot, Kunal Talwar, Li Zhang

The recent, remarkable growth of machine learning has led to intense interest in the privacy of the data on which machine learning relies, and to new techniques for preserving privacy.

Ensemble Adversarial Training: Attacks and Defenses

11 code implementations ICLR 2018 Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel

We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss.

Extending Defensive Distillation

no code implementations15 May 2017 Nicolas Papernot, Patrick McDaniel

Machine learning is vulnerable to adversarial examples: inputs carefully modified to force misclassification.

The Space of Transferable Adversarial Examples

1 code implementation11 Apr 2017 Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel

Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time.

On the (Statistical) Detection of Adversarial Examples

no code implementations21 Feb 2017 Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, Patrick McDaniel

Specifically, we augment our ML model with an additional output, in which the model is trained to classify all adversarial inputs.

Malware Classification Network Intrusion Detection

Adversarial Attacks on Neural Network Policies

no code implementations8 Feb 2017 Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel

Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification.


Towards the Science of Security and Privacy in Machine Learning

no code implementations11 Nov 2016 Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, Michael Wellman

Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics.

Decision Making

Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data

7 code implementations18 Oct 2016 Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal Talwar

The approach combines, in a black-box fashion, multiple models trained with disjoint datasets, such as records from different subsets of users.

Transfer Learning

Adversarial Perturbations Against Deep Neural Networks for Malware Classification

no code implementations14 Jun 2016 Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, Patrick McDaniel

Deep neural networks, like many other machine learning models, have recently been shown to lack robustness against adversarially crafted inputs.

Classification General Classification +2

Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

no code implementations24 May 2016 Nicolas Papernot, Patrick McDaniel, Ian Goodfellow

We demonstrate our attacks on two commercial machine learning classification systems from Amazon (96. 19% misclassification rate) and Google (88. 94%) using only 800 queries of the victim model, thereby showing that existing machine learning approaches are in general vulnerable to systematic black-box attacks regardless of their structure.

Crafting Adversarial Input Sequences for Recurrent Neural Networks

1 code implementation28 Apr 2016 Nicolas Papernot, Patrick McDaniel, Ananthram Swami, Richard Harang

Machine learning models are frequently used to solve complex security problems, as well as to make decisions in sensitive situations like guiding autonomous vehicles or predicting financial market behaviors.

Autonomous Vehicles General Classification

Detection under Privileged Information

no code implementations31 Mar 2016 Z. Berkay Celik, Patrick McDaniel, Rauf Izmailov, Nicolas Papernot, Ryan Sheatsley, Raquel Alvarez, Ananthram Swami

In this paper, we consider an alternate learning approach that trains models using "privileged" information--features available at training time but not at runtime--to improve the accuracy and resilience of detection systems.

Face Recognition Malware Classification +1

Practical Black-Box Attacks against Machine Learning

17 code implementations8 Feb 2016 Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram Swami

Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN.

The Limitations of Deep Learning in Adversarial Settings

12 code implementations24 Nov 2015 Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami

In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs.

Adversarial Attack Adversarial Defense

Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks

1 code implementation14 Nov 2015 Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, Ananthram Swami

In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs.

Autonomous Vehicles

Cannot find the paper you are looking for? You can Submit a new open access paper.