no code implementations • 1 Mar 2024 • Ashwinee Panda, Christopher A. Choquette-Choo, Zhengming Zhang, Yaoqing Yang, Prateek Mittal
When large language models are trained on private data, it can be a significant privacy risk for them to memorize and regurgitate sensitive information.
no code implementations • 7 Feb 2024 • Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, Peter Henderson
We develop methods to identify critical regions that are vital for safety guardrails, and that are disentangled from utility-relevant regions at both the neuron and rank levels.
no code implementations • 20 Jan 2024 • Jiachen T. Wang, Prateek Mittal, Ruoxi Jia
This work aims to address an open problem in data valuation literature concerning the efficient computation of Data Shapley for weighted $K$ nearest neighbor algorithm (WKNN-Shapley).
no code implementations • 9 Jan 2024 • Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, Prateek Mittal
We introduce DP-ZO, a new method for fine-tuning large language models that preserves the privacy of training data by privatizing zeroth-order optimization.
no code implementations • 19 Oct 2023 • Chong Xiang, Tong Wu, Sihui Dai, Jonathan Petit, Suman Jana, Prateek Mittal
State-of-the-art defenses against adversarial patch attacks can now achieve strong certifiable robustness with a marginal drop in model utility.
1 code implementation • 5 Oct 2023 • Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson
Optimizing large language models (LLMs) for downstream use cases often involves the customization of pre-trained LLMs through further fine-tuning.
no code implementations • 30 Aug 2023 • Jiachen T. Wang, Yuqing Zhu, Yu-Xiang Wang, Ruoxi Jia, Prateek Mittal
Data valuation aims to quantify the usefulness of individual data sources in training machine learning (ML) models, and is a critical aspect of data-centric ML research.
no code implementations • 23 Aug 2023 • Tinghao Xie, Xiangyu Qi, Ping He, Yiming Li, Jiachen T. Wang, Prateek Mittal
We present a novel defense, against backdoor attacks on Deep Neural Networks (DNNs), wherein adversaries covertly implant malicious behaviors (backdoors) into DNNs.
no code implementations • 3 Aug 2023 • Prateek Mittal, Puneet Goyal, Joohi Chauhan
The experimental results show that the proposed network outperforms the other methods, a significant difference of 11. 57% and 6. 34% in accuracy is observed for image and text classification, respectively, when compared with the second-best performing method.
1 code implementation • 22 Jun 2023 • Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, Prateek Mittal
Recently, there has been a surge of interest in integrating vision into Large Language Models (LLMs), exemplified by Visual Language Models (VLMs) such as Flamingo and GPT-4.
no code implementations • 2 May 2023 • Tong Wu, Ashwinee Panda, Jiachen T. Wang, Prateek Mittal
Based on the general paradigm of DP-ICL, we instantiate several techniques showing how to privatize ICL for text classification and language generation.
no code implementations • 17 Apr 2023 • Jiachen T. Wang, Saeed Mahloujifar, Tong Wu, Ruoxi Jia, Prateek Mittal
In this paper, we propose a new differential privacy paradigm called estimate-verify-release (EVR), which addresses the challenges of providing a strict upper bound for privacy parameter in DP compositions by converting an estimate of privacy parameter into a formal guarantee.
no code implementations • 21 Feb 2023 • Sihui Dai, Wenxin Ding, Arjun Nitin Bhagoji, Daniel Cullina, Ben Y. Zhao, Haitao Zheng, Prateek Mittal
Finding classifiers robust to adversarial examples is critical for their safe deployment.
no code implementations • 21 Feb 2023 • Sihui Dai, Saeed Mahloujifar, Chong Xiang, Vikash Sehwag, Pin-Yu Chen, Prateek Mittal
Using our framework, we present the first leaderboard, MultiRobustBench, for benchmarking multiattack evaluation which captures performance across attack types and attack strengths.
1 code implementation • 3 Feb 2023 • Jacob Brown, Xi Jiang, Van Tran, Arjun Nitin Bhagoji, Nguyen Phong Hoang, Nick Feamster, Prateek Mittal, Vinod Yegneswaran
In this paper, we explore how machine learning (ML) models can (1) help streamline the detection process, (2) improve the potential of using large-scale datasets for censorship detection, and (3) discover new censorship instances and blocking signatures missed by existing heuristic methods.
1 code implementation • ICLR 2023 • Xiangyu Qi, Tinghao Xie, Tinghao_Xie1, Yiming Li, Saeed Mahloujifar, Prateek Mittal
This latent separation is so pervasive that a family of backdoor defenses directly take it as a default assumption (dubbed latent separability assumption), based on which to identify poison samples via cluster analysis in the latent space.
no code implementations • 29 Jan 2023 • Tong Wu, Feiran Jia, Xiangyu Qi, Jiachen T. Wang, Vikash Sehwag, Saeed Mahloujifar, Prateek Mittal
Recently, test-time adaptation (TTA) has been proposed as a promising solution for addressing distribution shifts.
no code implementations • 8 Dec 2022 • Ashwinee Panda, Xinyu Tang, Vikash Sehwag, Saeed Mahloujifar, Prateek Mittal
A major direction in differentially private machine learning is differentially private fine-tuning: pretraining a model on a source of "public data" and transferring the extracted features to downstream tasks.
no code implementations • 16 Sep 2022 • Jiachen T. Wang, Saeed Mahloujifar, Shouda Wang, Ruoxi Jia, Prateek Mittal
As an application of our analysis, we show that PTR and our theoretical results can be used to design differentially private variants for byzantine robust training algorithms that use robust statistics for gradients aggregation.
1 code implementation • 15 Sep 2022 • Edoardo Debenedetti, Vikash Sehwag, Prateek Mittal
Additionally, investigating the reasons for the robustness of our models, we show that it is easier to generate strong attacks during training when using our recipe and that this leads to better robustness at test time.
no code implementations • 22 Jul 2022 • Tong Wu, Tianhao Wang, Vikash Sehwag, Saeed Mahloujifar, Prateek Mittal
Our attack can be easily deployed in the real world since it only requires rotating the object, as we show in both image classification and object detection applications.
1 code implementation • 20 Jun 2022 • Christian Cianfarani, Arjun Nitin Bhagoji, Vikash Sehwag, Ben Y. Zhao, Prateek Mittal, Haitao Zheng
Representation learning, i. e. the generation of representations useful for downstream applications, is a task of fundamental importance that underlies much of the success of deep neural networks (DNNs).
2 code implementations • 12 Jun 2022 • Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael W. Mahoney, Joseph E. Gonzalez, Kannan Ramchandran, Prateek Mittal
In this type of attack, the goal of the attacker is to use poisoned updates to implant so-called backdoors into the learned model such that, at test time, the model's outputs can be fixed to a given target for certain inputs.
2 code implementations • 26 May 2022 • Xiangyu Qi, Tinghao Xie, Jiachen T. Wang, Tong Wu, Saeed Mahloujifar, Prateek Mittal
First, we uncover a post-hoc workflow underlying most prior work, where defenders passively allow the attack to proceed and then leverage the characteristics of the post-attacked model to uncover poison samples.
1 code implementation • 26 May 2022 • Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, Prateek Mittal
This latent separation is so pervasive that a family of backdoor defenses directly take it as a default assumption (dubbed latent separability assumption), based on which to identify poison samples via cluster analysis in the latent space.
1 code implementation • 28 Apr 2022 • Sihui Dai, Saeed Mahloujifar, Prateek Mittal
Based on our generalization bound, we propose variation regularization (VR) which reduces variation of the feature extractor across the source threat model during training.
1 code implementation • 3 Feb 2022 • Chong Xiang, Alexander Valtchanov, Saeed Mahloujifar, Prateek Mittal
An attacker can use a single physically-realizable adversarial patch to make the object detector miss the detection of victim objects and undermine the functionality of object detection applications.
1 code implementation • 12 Dec 2021 • Ashwinee Panda, Saeed Mahloujifar, Arjun N. Bhagoji, Supriyo Chakraborty, Prateek Mittal
Federated learning is inherently vulnerable to model poisoning attacks because its decentralized nature allows attackers to participate with compromised devices.
no code implementations • 15 Oct 2021 • Xinyu Tang, Saeed Mahloujifar, Liwei Song, Virat Shejwalkar, Milad Nasr, Amir Houmansadr, Prateek Mittal
The goal of this work is to train ML models that have high membership privacy while largely preserving their utility; we therefore aim for an empirical membership privacy guarantee as opposed to the provable privacy guarantees provided by techniques like differential privacy, as such techniques are shown to deteriorate model utility.
no code implementations • 11 Oct 2021 • Sihui Dai, Saeed Mahloujifar, Prateek Mittal
To address this, we analyze the direct impact of activation shape on robustness through PAFs and observe that activation shapes with positive outputs on negative inputs and with high finite curvature can increase robustness.
1 code implementation • 20 Aug 2021 • Chong Xiang, Saeed Mahloujifar, Prateek Mittal
Remarkably, PatchCleanser achieves 83. 9% top-1 clean accuracy and 62. 1% top-1 certified robust accuracy against a 2%-pixel square patch anywhere on the image for the 1000-class ImageNet dataset.
1 code implementation • 26 Apr 2021 • Chong Xiang, Prateek Mittal
Recent provably robust defenses generally follow the PatchGuard framework by using CNNs with small receptive fields and secure feature aggregation for robust model predictions.
2 code implementations • ICLR 2022 • Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, Prateek Mittal
We circumvent this challenge by using additional data from proxy distributions learned by advanced generative models.
1 code implementation • 16 Apr 2021 • Arjun Nitin Bhagoji, Daniel Cullina, Vikash Sehwag, Prateek Mittal
In particular, it is critical to determine classifier-agnostic bounds on the training loss to establish when learning is possible.
3 code implementations • ICLR 2021 • Vikash Sehwag, Mung Chiang, Prateek Mittal
We demonstrate that SSD outperforms most existing detectors based on unlabeled data by a large margin.
1 code implementation • 5 Feb 2021 • Chong Xiang, Prateek Mittal
In this paper, we propose DetectorGuard as the first general framework for building provably robust object detectors against localized patch hiding attacks.
no code implementations • 17 Jan 2021 • Peng Gao, Fei Shao, Xiaoyuan Liu, Xusheng Xiao, Haoyuan Liu, Zheng Qin, Fengyuan Xu, Prateek Mittal, Sanjeev R. Kulkarni, Dawn Song
Log-based cyber threat hunting has emerged as an important solution to counter sophisticated cyber attacks.
1 code implementation • 26 Oct 2020 • Peng Gao, Fei Shao, Xiaoyuan Liu, Xusheng Xiao, Zheng Qin, Fengyuan Xu, Prateek Mittal, Sanjeev R. Kulkarni, Dawn Song
Log-based cyber threat hunting has emerged as an important solution to counter sophisticated attacks.
1 code implementation • 19 Oct 2020 • Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, Matthias Hein
As a research community, we are still lacking a systematic understanding of the progress on adversarial robustness which often makes it hard to identify the most promising ideas in training robust models.
no code implementations • 8 Jul 2020 • Liwei Song, Vikash Sehwag, Arjun Nitin Bhagoji, Prateek Mittal
With our evaluation across 6 OOD detectors, we find that the choice of in-distribution data, model architecture and OOD data have a strong impact on OOD detection performance, inducing false positive rates in excess of $70\%$.
BIG-bench Machine Learning Out of Distribution (OOD) Detection
no code implementations • 24 Jun 2020 • Vikash Sehwag, Rajvardhan Oak, Mung Chiang, Prateek Mittal
With increasing expressive power, deep neural networks have significantly improved the state-of-the-art on image classification datasets, such as ImageNet.
2 code implementations • 17 May 2020 • Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, Prateek Mittal
In this paper, we propose a general defense framework called PatchGuard that can achieve high provable robustness while maintaining high clean accuracy against localized adversarial patches.
1 code implementation • 5 Apr 2020 • Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateek Mittal, Tal Rabin
For private training, we are about 6x faster than SecureNN, 4. 4x faster than ABY3 and about 2-60x more communication efficient.
1 code implementation • 24 Mar 2020 • Liwei Song, Prateek Mittal
Machine learning models are prone to memorizing sensitive data, making them vulnerable to membership inference attacks in which an adversary aims to guess if an input sample was used to train the model.
1 code implementation • 9 Mar 2020 • David Marco Sommer, Liwei Song, Sameer Wagh, Prateek Mittal
In this work, we take the first step in proposing a formal framework to study the design of such verification mechanisms for data deletion requests -- also known as machine unlearning -- in the context of systems that provide machine learning as a service (MLaaS).
4 code implementations • NeurIPS 2020 • Vikash Sehwag, Shiqi Wang, Prateek Mittal, Suman Jana
We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously.
8 code implementations • 10 Dec 2019 • Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, Sen Zhao
FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches.
1 code implementation • NeurIPS 2019 • Arjun Nitin Bhagoji, Daniel Cullina, Prateek Mittal
In this paper, we use optimal transport to characterize the minimum possible loss in an adversarial classification scenario.
no code implementations • 14 Jun 2019 • Vikash Sehwag, Shiqi Wang, Prateek Mittal, Suman Jana
In this work, we rigorously study the extension of network pruning strategies to preserve both benign accuracy and robustness of a network.
1 code implementation • 24 May 2019 • Liwei Song, Reza Shokri, Prateek Mittal
To perform the membership inference attacks, we leverage the existing inference methods that exploit model predictions.
no code implementations • 5 May 2019 • Vikash Sehwag, Arjun Nitin Bhagoji, Liwei Song, Chawin Sitawarin, Daniel Cullina, Mung Chiang, Prateek Mittal
A large body of recent work has investigated the phenomenon of evasion attacks using adversarial examples for deep learning systems, where the addition of norm-bounded perturbations to the test inputs leads to incorrect output classification.
no code implementations • NeurIPS 2018 • Daniel Cullina, Arjun Nitin Bhagoji, Prateek Mittal
We then explicitly derive the adversarial VC-dimension for halfspace classifiers in the presence of a sample-wise norm-constrained adversary of the type commonly studied for evasion attacks and show that it is the same as the standard VC-dimension, closing an open question.
2 code implementations • ICLR 2019 • Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, Seraphin Calo
Federated learning distributes model training among a multitude of agents, who, guided by privacy concerns, perform training using their local data but share only model parameter updates, for iterative aggregation at the server.
no code implementations • 17 Nov 2018 • Anatoly Shusterman, Lachlan Kang, Yarden Haskal, Yosef Meltser, Prateek Mittal, Yossi Oren, Yuval Yarom
In this work we investigate these attacks under a different attack model, in which the adversary is capable of running a small amount of unprivileged code on the target user's computer.
no code implementations • 10 Sep 2018 • Daniel Cullina, Negar Kiyavash, Prateek Mittal, H. Vincent Poor
This estimator searches for an alignment in which the intersection of the correlated graphs using this alignment has a minimum degree of $k$.
1 code implementation • 25 Jun 2018 • Peng Gao, Xusheng Xiao, Ding Li, Zhichun Li, Kangkook Jee, Zhen-Yu Wu, Chung Hwan Kim, Sanjeev R. Kulkarni, Prateek Mittal
To facilitate the task of expressing anomalies based on expert knowledge, our system provides a domain-specific query language, SAQL, which allows analysts to express models for (1) rule-based anomalies, (2) time-series anomalies, (3) invariant-based anomalies, and (4) outlier-based anomalies.
Cryptography and Security Databases
no code implementations • 5 Jun 2018 • Daniel Cullina, Arjun Nitin Bhagoji, Prateek Mittal
We then explicitly derive the adversarial VC-dimension for halfspace classifiers in the presence of a sample-wise norm-constrained adversary of the type commonly studied for evasion attacks and show that it is the same as the standard VC-dimension, closing an open question.
1 code implementation • 26 Feb 2018 • Thee Chanyaswad, Alex Dytso, H. Vincent Poor, Prateek Mittal
noise to each element of the matrix, this method is often sub-optimal as it forfeits an opportunity to exploit the structural characteristics typically associated with matrix analysis.
1 code implementation • 18 Feb 2018 • Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia, Mung Chiang, Prateek Mittal
In this paper, we propose and examine security attacks against sign recognition systems for Deceiving Autonomous caRs with Toxic Signs (we call the proposed attacks DARTS).
1 code implementation • 9 Jan 2018 • Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia, Prateek Mittal, Mung Chiang
Our attack pipeline generates adversarial samples which are robust to the environmental conditions and noisy image transformations present in the physical world.
no code implementations • 2 Jan 2018 • Thee Chanyaswad, Alex Dytso, H. Vincent Poor, Prateek Mittal
To address this challenge, we propose a novel differential privacy mechanism called the Matrix-Variate Gaussian (MVG) mechanism, which adds a matrix-valued noise drawn from a matrix-variate Gaussian distribution, and we rigorously prove that the MVG mechanism preserves $(\epsilon,\delta)$-differential privacy.
1 code implementation • 31 Aug 2017 • Thee Chanyaswad, Changchang Liu, Prateek Mittal
A key challenge facing the design of differential privacy in the non-interactive setting is to maintain the utility of the released data.
Cryptography and Security
1 code implementation • 24 Aug 2017 • Liwei Song, Prateek Mittal
Voice assistants like Siri enable us to control IoT devices conveniently with voice commands, however, they also provide new attack opportunities for adversaries.
Cryptography and Security
no code implementations • 9 Apr 2017 • Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, Prateek Mittal
We propose the use of data transformations as a defense against evasion attacks on ML classifiers.
no code implementations • 25 Mar 2016 • Daniel Cullina, Kushagra Singhal, Negar Kiyavash, Prateek Mittal
We ask the question "Does there exist a regime where the network cannot be deanonymized perfectly, yet the community structure could be learned?."