Search Results for author:
Found 87 papers, 33 papers with code
A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems
Large Language Model (LLM) systems are inherently compositional, with individual LLM serving as the core foundation with additional layers of objects such as plugins, sandbox, and so on.
PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails
More recent LLMs often incorporate an additional layer of defense, a Guard Model, which is a second LLM that is designed to check and moderate the output response of the primary LLM.
Do Large Code Models Understand Programming Concepts? A Black-box Approach
Large Language Models' success on text generation has also made them better at code generation and coding tasks.
A Somewhat Robust Image Watermark against Diffusion-based Editing Models
In response, we develop a novel technique, RIW (Robust Invisible Watermarking), to embed invisible watermarks leveraging adversarial example techniques.
Publicly Detectable Watermarking for Language Models
We construct the first provable watermarking scheme for language models with public detectability or verifiability: we use a private key for watermarking and a public key for watermark detection.
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Large language models (LLMs) have recently shown great advances in a variety of tasks, including natural language understanding and generation.
Why Train More? Effective and Efficient Membership Inference via Memorization
Many practical black-box MIAs require query access to the data distribution (the same distribution where the private data is drawn) to train shadow models.
Identifying and Mitigating the Security Risks of Generative AI
However, GenAI can be used just as well by attackers to generate new attacks and increase the velocity and efficacy of existing attacks.
Theoretically Principled Trade-off for Stateful Defenses against Query-Based Black-Box Attacks
This work aims to address this gap by offering a theoretical characterization of the trade-off between detection and false positive rates for stateful defenses.
Pareto-Secure Machine Learning (PSML): Fingerprinting and Securing Inference Serving Systems
Second, we counter the proposed attack with a noise-based defense mechanism that thwarts fingerprinting by adding noise to the specified performance metrics.
Two Heads are Better than One: Towards Better Adversarial Robustness by Combining Transduction and Rejection
Nevertheless, under recent strong adversarial attacks (GMSA, which has been shown to be much more effective than AutoAttack against transduction), Goldwasser et al.'s work was shown to have low performance in a practical deep-learning setting.
Rethinking Diversity in Deep Neural Network Testing
We note that the objective of testing DNNs is specific and well-defined: identifying inputs that lead to misclassifications.
Stratified Adversarial Robustness with Rejection
We theoretically analyze the stratified rejection setting and propose a novel defense method -- Adversarial Training with Consistent Prediction-based Rejection (CPR) -- for building a robust selective classifier.
ASPEST: Bridging the Gap Between Active Learning and Selective Prediction
In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain while increasing accuracy and coverage.
Efficient Symbolic Reasoning for Neural-Network Verification
They allow us to encode many verification problems for neural networks as quadratic programs.
Stateful Defenses for Machine Learning Models Are Not Yet Secure Against Black-box Attacks
Such stateful defenses aim to defend against black-box attacks by tracking the query history and detecting and rejecting queries that are "similar" and thus preventing black-box attacks from finding useful gradients and making progress towards finding adversarial attacks within a reasonable query budget.
The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning
foundation models) has recently become a prevalent learning paradigm, where one first pre-trains a representation using large-scale unlabeled data, and then learns simple predictors on top of the representation using small labeled data from the downstream tasks.
Learning Modulo Theories
Recent techniques that integrate \emph{solver layers} into Deep Neural Networks (DNNs) have shown promise in bridging a long-standing gap between inductive learning and symbolic reasoning techniques.
Private Multi-Winner Voting for Machine Learning
We use our mechanisms to enable privacy-preserving multi-label learning in the central setting by extending the canonical single-label technique: PATE.
Federated Boosted Decision Trees with Differential Privacy
There is great demand for scalable, secure, and efficient privacy-preserving machine learning models that can be trained over distributed data.
Overparameterization from Computational Constraints
In particular, for computationally bounded learners, we extend the recent result of Bubeck and Sellke [NeurIPS'2021] which shows that robust models might need more parameters, to the computational regime and show that bounded learners could provably need an even larger number of parameters.
Constraining the Attack Space of Machine Learning Models with Distribution Clamping Preprocessing
Preprocessing and outlier detection techniques have both been applied to neural networks to increase robustness with varying degrees of success.
Optimal Membership Inference Bounds for Adaptive Composition of Sampled Gaussian Mechanisms
A common countermeasure against MI attacks is to utilize differential privacy (DP) during model training to mask the presence of individual examples.
Concept-based Explanations for Out-Of-Distribution Detectors
Based on these metrics, we propose an unsupervised framework for learning a set of concepts that satisfy the desired properties of high detection completeness and concept separability, and demonstrate its effectiveness in providing concept-based explanations for diverse off-the-shelf OOD detectors.
A Quantitative Geometric Approach to Neural-Network Smoothness
In this work, we provide a unified theoretical framework, a quantitative geometric approach, to address the Lipschitz constant estimation.
D4: Detection of Adversarial Diffusion Deepfakes Using Disjoint Ensembles
D4 uses an ensemble of models over disjoint subsets of the frequency spectrum to significantly improve adversarial robustness.
An Exploration of Multicalibration Uniform Convergence Bounds
Given the prevalence of ERM sample complexity bounds, our proposed framework enables machine learning practitioners to easily understand the convergence behavior of multicalibration error for a myriad of classifier architectures.
Revisiting Adversarial Robustness of Classifiers With a Reject Option
Motivated by this metric, we propose novel loss functions and a robust training method -- \textit{stratified adversarial training with rejection} (SATR) -- for a classifier with reject option, where the goal is to accept and correctly-classify small input perturbations, while allowing the rejection of larger input perturbations that cannot be correctly classified.
Towards Evaluating the Robustness of Neural Networks Learned by Transduction
There has been emerging interest in using transductive learning for adversarial robustness (Goldwasser et al., NeurIPS 2020; Wu et al., ICML 2020; Wang et al., ArXiv 2021).
Privacy Implications of Shuffling
\ldp deployments are vulnerable to inference attacks as an adversary can link the noisy responses to their identity and subsequently, auxiliary information using the \textit{order} of the data.
Fast and Sample-Efficient Domain Adaptation for Autoencoder-Based End-to-End Communication
In this paper, we address the setting where the target domain has only limited labeled data from a distribution that is expected to change frequently.
Less is More: Dimension Reduction Finds On-Manifold Adversarial Examples in Hard-Label Attacks
It was recently shown in the gradient-level setting that regular adversarial examples leave the data manifold, while their on-manifold counterparts are in fact generalization errors.
Fairness Properties of Face Recognition and Obfuscation Systems
We answer this question with an analytical and empirical exploration of recent face obfuscation systems.
Few-Shot Domain Adaptation For End-to-End Communication
The problem of end-to-end learning of a communication system using an autoencoder -- consisting of an encoder, channel, and decoder modeled using neural networks -- has recently been shown to be an effective approach.
Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
This observation leads to two challenging tasks: (1) unsupervised accuracy estimation, which aims to estimate the accuracy of a pre-trained classifier on a set of unlabeled test inputs; (2) error detection, which aims to identify mis-classified test inputs.
Towards Adversarial Robustness via Transductive Learning
Compared to traditional "test-time" defenses, these defense mechanisms "dynamically retrain" the model based on test time input via transductive learning; and theoretically, attacking these defenses boils down to bilevel optimization, which seems to raise the difficulty for adaptive attacks.
A Shuffling Framework for Local Differential Privacy
ldp deployments are vulnerable to inference attacks as an adversary can link the noisy responses to their identity and subsequently, auxiliary information using the order of the data.
Causally Constrained Data Synthesis for Private Data Release
However for real-world applications, the privacy of data is critical.
Hard-label Manifolds: Unexpected Advantages of Query Efficiency for Finding On-manifold Adversarial Examples
It was recently shown in the gradient-level setting that regular adversarial examples leave the data manifold, while their on-manifold counterparts are in fact generalization errors.
Exploring Adversarial Robustness of Deep Metric Learning
Deep Metric Learning (DML), a widely-used technique, involves learning a distance metric between pairs of samples.
CaPC Learning: Confidential and Private Collaborative Learning
There is currently no method that enables machine learning in such a setting, where both confidentiality and privacy need to be preserved, to prevent both explicit and implicit sharing of data.
Adversarial Deep Metric Learning
To the best of our knowledge, we are the first to systematically analyze this dependence effect and propose a principled approach for robust training of deep metric learning networks that accounts for the nuances of metric losses.
Generalized Universal Approximation for Certified Networks
To certify safety and robustness of neural networks, researchers have successfully applied abstract interpretation, primarily using interval bound propagation.
Test-Time Adaptation and Adversarial Robustness
On the positive side, we show that, if one is allowed to access the training data, then Domain Adversarial Neural Networks (${\sf DANN}$), an algorithm designed for unsupervised domain adaptation, can provide nontrivial robustness in the test-time maximin threat model against strong transfer attacks and adaptive fixed point attacks.
Sample Complexity of Adversarially Robust Linear Classification on Separated Data
This shows that for very well-separated data, convergence rates of $O(\frac{1}{n})$ are achievable, which is not the case otherwise.
ShadowNet: A Secure and Efficient On-device Model Inference System for Convolutional Neural Networks
ShadowNet protects the model privacy with Trusted Execution Environment (TEE) while securely outsourcing the heavy linear layers of the model to the untrusted hardware accelerators.
Is Private Learning Possible with Instance Encoding?
A private machine learning algorithm hides as much as possible about its training data while still preserving accuracy.
Informative Outlier Matters: Robustifying Out-of-distribution Detection Using Outlier Mining
We show that, by mining informative auxiliary OOD data, one can significantly improve OOD detection performance, and somewhat surprisingly, generalize to unseen adversarial attacks.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
A General Framework For Detecting Anomalous Inputs to DNN Classifiers
We propose an unsupervised anomaly detection framework based on the internal DNN layer representations in the form of a meta-algorithm with configurable components.
Interval Universal Approximation for Neural Networks
This is a crucial question, as our constructive proof of IUA is exponential in the size of the approximation domain.
Robust and Accurate Authorship Attribution via Program Normalization
To address these emerging issues, we formulate this security challenge into a general threat model, the $\textit{relational adversary}$, that allows an arbitrary number of the semantics-preserving transformations to be applied to an input in any problem space.
ATOM: Robustifying Out-of-distribution Detection Using Outlier Mining
We show that, by mining informative auxiliary OOD data, one can significantly improve OOD detection performance, and somewhat surprisingly, generalize to unseen adversarial attacks.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Continuous Release of Data Streams under both Centralized and Local Differential Privacy
To our knowledge, this is the first LDP algorithm for publishing streaming data.
Representation Bayesian Risk Decompositions and Multi-Source Domain Adaptation
Recent studies provide hints and failure examples for domain invariant representation learning, a common approach for this problem, but the explanations provided are somewhat different and do not provide a unified picture.
A Separation Result Between Data-oblivious and Data-aware Poisoning Attacks
Some of the stronger poisoning attacks require the full knowledge of the training data.
Robust Out-of-distribution Detection for Neural Networks
Formally, we extensively study the problem of Robust Out-of-Distribution Detection on common OOD detection approaches, and show that state-of-the-art OOD detectors can be easily fooled by adding small perturbations to the in-distribution and OOD inputs.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Face-Off: Adversarial Face Obfuscation
We implement and evaluate Face-Off to find that it deceives three commercial face recognition services from Microsoft, Amazon, and Face++.
Cryptography and Security
Analyzing Accuracy Loss in Randomized Smoothing Defenses
In test-time attacks an adversary crafts adversarial examples, which are specially crafted perturbations imperceptible to humans which, when added to an input example, force a machine learning model to misclassify the given input example.
CAUSE: Learning Granger Causality from Event Sequences using Attribution Methods
We study the problem of learning Granger causality between event types from asynchronous, interdependent, multi-type event sequences.
GRAPHITE: Generating Automatic Physical Examples for Machine-Learning Attacks on Computer Vision Systems
We address three key requirements for practical attacks for the real-world: 1) automatically constraining the size and shape of the attack so it can be applied with stickers, 2) transform-robustness, i. e., robustness of a attack to environmental physical variations such as viewpoint and lighting changes, and 3) supporting attacks in not only white-box, but also black-box hard-label scenarios, so that the adversary can attack proprietary models.
Semantic Robustness of Models of Source Code
Deep neural networks are vulnerable to adversarial examples - small input perturbations that result in incorrect predictions.
Attribution-Based Confidence Metric For Deep Neural Networks
These experiments demonstrate the effectiveness of the ABC metric to make DNNs more trustworthy and resilient.
Generating Semantic Adversarial Examples with Differentiable Rendering
In this paper we explore semantic adversarial examples (SAEs) where an attacker creates perturbations in the semantic space representing the environment that produces input for the ML model.
On the Need for Topology-Aware Generative Models for Manifold-Based Defenses
These defenses rely on the assumption that data lie in a manifold of a lower dimension than the input space.
Improving Utility and Security of the Shuffler-based Differential Privacy
When collecting information, local differential privacy (LDP) alleviates privacy concerns of users because their private information is randomized before being sent it to the central aggregator.
Data-Dependent Differentially Private Parameter Learning for Directed Graphical Models
Our solution optimizes for the utility of inference queries over the DGM and \textit{adds noise that is customized to the properties of the private input dataset and the graph structure of the DGM}.
An Investigation of Data Poisoning Defenses for Online Learning
Data poisoning attacks -- where an adversary can modify a small fraction of training data, with the goal of forcing the trained classifier to high loss -- are an important threat for machine learning in many applications.
Adversarially Robust Learning Could Leverage Computational Hardness
On the reverse directions, we also show that the existence of such learning task in which computational robustness beats information theoretic robustness requires computational hardness by implying (average-case) hardness of NP.
Rearchitecting Classification Frameworks For Increased Robustness
and how to design a classification paradigm that leverages these invariances to improve the robustness accuracy trade-off?
Robust Attribution Regularization
An emerging problem in trustworthy machine learning is to train models that produce robust interpretations for their predictions.
Attribution-driven Causal Analysis for Detection of Adversarial Examples
We study the robustness of machine learning models on benign and adversarial inputs in this neighborhood.
Privacy-Preserving Collaborative Prediction using Random Forests
We study the problem of privacy-preserving machine learning (PPML) for ensemble methods, focusing our effort on random forests.
Exploring Connections Between Active Learning and Model Extraction
This has resulted in the surge of Machine Learning-as-a-Service (MLaaS) - cloud services that provide (a) tools and resources to learn the model, and (b) a user-friendly query interface to access the model.
Concise Explanations of Neural Networks using Adversarial Training
Our first contribution is a theoretical exploration of how these two properties (when using attributions based on Integrated Gradients, or IG) are related to adversarial training, for a class of 1-layer networks (which includes logistic regression models for binary and multi-class classification); for these networks we show that (a) adversarial training using an $\ell_\infty$-bounded adversary produces models with sparse attribution vectors, and (b) natural model-training while encouraging stable explanations (via an extra term in the loss function), is equivalent to adversarial training.
Explainable Black-Box Attacks Against Model-based Authentication
Establishing unique identities for both humans and end systems has been an active research problem in the security community, giving rise to innovative machine learning-based authentication techniques.
BIG-bench Machine Learning
Explainable Artificial Intelligence (XAI)
Neural-Augmented Static Analysis of Android Communication
We address the problem of discovering communication links between applications in the popular Android mobile operating system, an important problem for security and privacy in Android.
Towards Understanding Limitations of Pixel Discretization Against Adversarial Attacks
We analyze our results in a theoretical framework and offer strong evidence that pixel discretization is unlikely to work on all but the simplest of the datasets.
Semantic Adversarial Deep Learning
However, existing approaches to generating adversarial examples and devising robust ML algorithms mostly ignore the semantics and context of the overall system containing the ML component.
The Manifold Assumption and Defenses Against Adversarial Perturbations
Interestingly, we find that a recent objective by Madry et al. encourages training a model that satisfies well our formal version of the goodness property, but has a weak control of points that are wrong but with low confidence.
Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training
In this paper we study leveraging confidence information induced by adversarial training to reinforce adversarial robustness of a given adversarially trained model.
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks.
Analyzing the Robustness of Nearest Neighbors to Adversarial Examples
Our analysis shows that its robustness properties depend critically on the value of k - the classifier may be inherently non-robust for small k, but its robustness approaches that of the Bayes Optimal classifier for fast-growing k. We propose a novel modified 1-nearest neighbor classifier, and guarantee its robustness in the large sample limit.
Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics
This paper takes a first step to remedy this disconnect and proposes a private SGD algorithm to address \emph{both} issues in an integrated manner.
Practical Black-Box Attacks against Machine Learning
Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN.
Revisiting Differentially Private Regression: Lessons From Learning Theory and their Consequences
Perhaps more importantly, our theory reveals that the most basic mechanism in differential privacy, output perturbation, can be used to obtain a better tradeoff for all convex-Lipschitz-bounded learning tasks.
The Limitations of Deep Learning in Adversarial Settings
In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs.
Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks
In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs.
