no code implementations • 12 Feb 2025 • Minghong Fang, Xilong Wang, Neil Zhenqiang Gong
Our experimental results across different settings show that the Normalized attack can greatly disrupt non-ensemble Byzantine-robust methods, and our ensemble approach offers substantial resistance against poisoning attacks.
1 code implementation • 18 Oct 2024 • Zedian Shao, Hongbin Liu, Jaden Mu, Neil Zhenqiang Gong
In a prompt injection attack, an attacker injects a prompt into the original one, aiming to make the LLM follow the injected prompt and perform a task chosen by the attacker.
1 code implementation • 15 Oct 2024 • Zhongye Liu, Hongbin Liu, Yuepeng Hu, Zedian Shao, Neil Zhenqiang Gong
Our theoretical analysis shows that symmetric accuracy is an unbiased evaluation metric that remains unaffected by the imbalance of VH testing cases with varying answers when an MLLM is randomly guessing the answers, whereas traditional accuracy is prone to such imbalance.
1 code implementation • 2 Oct 2024 • Xilong Wang, Hao Fu, Jindong Wang, Neil Zhenqiang Gong
In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs.
1 code implementation • 21 Jul 2024 • Zonghao Huang, Neil Zhenqiang Gong, Michael K. Reiter
Auditing the use of data in training machine-learning (ML) models is an increasingly pressing challenge, as myriad ML practitioners routinely leverage the effort of content creators to train models without their permission.
1 code implementation • 12 Jul 2024 • Zedian Shao, Hongbin Liu, Yuepeng Hu, Neil Zhenqiang Gong
In particular, our MLLM-Refusal optimizes a nearly-imperceptible refusal perturbation and adds it to an image, causing target MLLMs to likely refuse a safe prompt containing the perturbed image and a safe question.
no code implementations • 9 Jul 2024 • Yuqi Jia, Minghong Fang, Hongbin Liu, Jinghuai Zhang, Neil Zhenqiang Gong
Existing defenses mainly focus on protecting the training phase of FL such that the learnt global model is poison free.
1 code implementation • 4 Jul 2024 • Zhengyuan Jiang, Moyang Guo, Yuepeng Hu, Jinyuan Jia, Neil Zhenqiang Gong
In this work, we propose the first image watermarks with certified robustness guarantees against removal and forgery attacks.
no code implementations • 1 Jul 2024 • Dongping Chen, Jiawen Shi, Yao Wan, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun
Additionally, we also explore the utility and trustworthiness of LLM in the self-cognition state, revealing that the self-cognition state enhances some specific tasks such as creative writing and exaggeration.
no code implementations • 23 Jun 2024 • Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, Bhuwan Dhingra
We propose ReCaLL (Relative Conditional Log-Likelihood), a novel membership inference attack (MIA) to detect LLMs' pretraining data by leveraging their conditional language modeling capabilities.
1 code implementation • 11 Jun 2024 • Hongbin Liu, Moyang Guo, Zhengyuan Jiang, Lun Wang, Neil Zhenqiang Gong
The increasing realism of synthetic speech, driven by advancements in text-to-speech models, raises ethical concerns regarding impersonation and disinformation.
1 code implementation • 9 May 2024 • Yixin Wu, Xinlei He, Pascal Berrang, Mathias Humbert, Michael Backes, Neil Zhenqiang Gong, Yang Zhang
This paper fills the gap by conducting a systematic privacy analysis of inductive GNNs through the lens of link stealing attacks, one of the most popular attacks that are specifically designed for GNNs.
no code implementations • 8 Apr 2024 • Jiacheng Du, Jiahui Hu, Zhibo Wang, Peng Sun, Neil Zhenqiang Gong, Kui Ren, Chun Chen
However, recent studies have shown that clients' private training data can be reconstructed from shared gradients in FL, a vulnerability known as gradient inversion attacks (GIAs).
1 code implementation • 5 Apr 2024 • Zhengyuan Jiang, Moyang Guo, Yuepeng Hu, Neil Zhenqiang Gong
Our key idea is to assign a unique watermark to each user of the GenAI service and embed this watermark into the AI-generated content created by that user.
1 code implementation • 26 Mar 2024 • Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, Neil Zhenqiang Gong
In this work, we propose JudgeDeceiver, an optimization-based prompt injection attack to LLM-as-a-Judge.
no code implementations • 5 Mar 2024 • Yichang Xu, Ming Yin, Minghong Fang, Neil Zhenqiang Gong
Recent studies have revealed that federated learning (FL), once considered secure due to clients not sharing their private data with the server, is vulnerable to attacks such as client-side training data distribution inference, where a malicious client can recreate the victim's data.
1 code implementation • 22 Feb 2024 • Wen Huang, Hongbin Liu, Minxin Guo, Neil Zhenqiang Gong
We find that existing MLLMs such as GPT-4V, LLaVA-1. 5, and MiniGPT-v2 hallucinate for a large fraction of the instances in our benchmark.
no code implementations • 22 Feb 2024 • Hongbin Liu, Michael K. Reiter, Neil Zhenqiang Gong
However, foundation models are vulnerable to backdoor attacks and a backdoored foundation model is a single-point-of-failure of the AI ecosystem, e. g., multiple downstream classifiers inherit the backdoor vulnerabilities simultaneously.
no code implementations • 18 Feb 2024 • Ming Yin, Yichang Xu, Minghong Fang, Neil Zhenqiang Gong
Current poisoning attacks on federated recommender systems often rely on additional information, such as the local training data of genuine users or item popularity.
1 code implementation • 10 Jan 2024 • Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao liu, Heng Ji, Hongyi Wang, huan zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, Joaquin Vanschoren, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, Yue Zhao
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
1 code implementation • CVPR 2024 • Jinghuai Zhang, Hongbin Liu, Jinyuan Jia, Neil Zhenqiang Gong
In this work we take the first step to analyze the limitations of existing backdoor attacks and propose new DPBAs called CorruptEncoder to CL.
1 code implementation • 3 Dec 2023 • Yuqi Jia, Saeed Vahidian, Jingwei Sun, Jianyi Zhang, Vyacheslav Kungurtsev, Neil Zhenqiang Gong, Yiran Chen
This process allows local devices to train smaller surrogate models while enabling the training of a larger global model on the server, effectively minimizing resource utilization.
no code implementations • 20 Oct 2023 • Yuqi Jia, Minghong Fang, Neil Zhenqiang Gong
In SelfishAttack, a set of selfish clients aim to achieve competitive advantages over the remaining non-selfish ones, i. e., the final learnt local models of the selfish clients are more accurate than those of the non-selfish ones.
1 code implementation • 19 Oct 2023 • Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, Neil Zhenqiang Gong
Existing attacks are special cases in our framework.
1 code implementation • 4 Oct 2023 • Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, Lichao Sun
However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests.
1 code implementation • 29 Sep 2023 • Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, Xing Xie
Moreover, DyVal-generated samples are not only evaluation sets, but also helpful data for fine-tuning to improve the performance of LLMs on existing benchmarks.
no code implementations • 11 Jun 2023 • Minglei Yin, Bin Liu, Neil Zhenqiang Gong, Xin Li
Our proposed method can simultaneously (1) secure VARS from adversarial attacks characterized by local perturbations by image reconstruction based on global vision transformers; and (2) accurately detect adversarial examples using a novel contrastive learning approach.
1 code implementation • 7 Jun 2023 • Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, Xing Xie
Furthermore, we present a comprehensive analysis to understand the mystery behind prompt robustness and its transferability.
Cross-Lingual Paraphrase Identification
Machine Translation
+5
1 code implementation • 5 May 2023 • Zhengyuan Jiang, Jinghuai Zhang, Neil Zhenqiang Gong
Specifically, a watermark is embedded into an AI-generated content before it is released.
1 code implementation • 26 Mar 2023 • Jinyuan Jia, Yupei Liu, Yuepeng Hu, Neil Zhenqiang Gong
PORE can transform any existing recommender system to be provably robust against any untargeted data poisoning attacks, which aim to reduce the overall performance of a recommender system.
no code implementations • CVPR 2023 • Jinghuai Zhang, Jinyuan Jia, Hongbin Liu, Neil Zhenqiang Gong
Existing certified defenses against adversarial point clouds suffer from a key limitation: their certified robustness guarantees are probabilistic, i. e., they produce an incorrect certified robustness guarantee with some probability.
no code implementations • 7 Jan 2023 • Wenjie Qu, Jinyuan Jia, Neil Zhenqiang Gong
For the first question, we show that the cloud service only needs to provide two APIs, which we carefully design, to enable a client to certify the robustness of its downstream classifier with a minimal number of queries to the APIs.
no code implementations • 13 Dec 2022 • Minghong Fang, Jia Liu, Neil Zhenqiang Gong, Elizabeth S. Bentley
Asynchronous FL aims to address this challenge by enabling the server to update the model once any client's model update reaches it without waiting for other clients' model updates.
no code implementations • 6 Dec 2022 • Hongbin Liu, Wenjie Qu, Jinyuan Jia, Neil Zhenqiang Gong
In this work, we perform the first systematic, principled measurement study to understand whether and when a pre-trained encoder can address the limitations of secure or privacy-preserving supervised learning algorithms.
2 code implementations • 15 Nov 2022 • Jinghuai Zhang, Hongbin Liu, Jinyuan Jia, Neil Zhenqiang Gong
In this work, we take the first step to analyze the limitations of existing backdoor attacks and propose new DPBAs called CorruptEncoder to CL.
1 code implementation • 26 Oct 2022 • Haolin Yuan, Bo Hui, Yuchen Yang, Philippe Burlina, Neil Zhenqiang Gong, Yinzhi Cao
Federated learning (FL) allows multiple clients to collaboratively train a deep learning model.
no code implementations • 20 Oct 2022 • Xiaoyu Cao, Jinyuan Jia, Zaixi Zhang, Neil Zhenqiang Gong
Existing defenses focus on preventing a small number of malicious clients from poisoning the global model via robust federated learning methods and detecting malicious clients when there are a large number of them.
1 code implementation • 3 Oct 2022 • Jinyuan Jia, Wenjie Qu, Neil Zhenqiang Gong
In this work, we propose MultiGuard, the first provably robust defense against adversarial examples to multi-label classification.
no code implementations • 2 Oct 2022 • Xiaoyu Cao, Zaixi Zhang, Jinyuan Jia, Neil Zhenqiang Gong
Our key idea is to divide the clients into groups, learn a global model for each group of clients using any existing federated learning method, and take a majority vote among the global models to classify a test input.
1 code implementation • 25 Jul 2022 • Xinlei He, Hongbin Liu, Neil Zhenqiang Gong, Yang Zhang
The results show that early stopping can mitigate the membership inference attack, but with the cost of model's utility degradation.
1 code implementation • 19 Jul 2022 • Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, Neil Zhenqiang Gong
FLDetector aims to detect and remove the majority of the malicious clients such that a Byzantine-robust FL method can learn an accurate global model using the remaining clients.
no code implementations • 13 May 2022 • Hongbin Liu, Jinyuan Jia, Neil Zhenqiang Gong
In this work, we propose PoisonedEncoder, a data poisoning attack to contrastive learning.
1 code implementation • 16 Mar 2022 • Xiaoyu Cao, Neil Zhenqiang Gong
Specifically, we assume the attacker injects fake clients to a federated learning system and sends carefully crafted fake local model updates to the cloud server during training, such that the learnt global model has low accuracy for many indiscriminate test inputs.
1 code implementation • 15 Jan 2022 • Yupei Liu, Jinyuan Jia, Hongbin Liu, Neil Zhenqiang Gong
A pre-trained encoder may be deemed confidential because its training requires lots of data and computation resources as well as its public release may facilitate misuse of AI, e. g., for deepfakes generation.
1 code implementation • 23 Nov 2021 • Huanrui Yang, Xiaoxuan Yang, Neil Zhenqiang Gong, Yiran Chen
We therefore propose HERO, a Hessian-enhanced robust optimization method, to minimize the Hessian eigenvalues through a gradient-based training process, simultaneously improving the generalization and quantization performance.
no code implementations • 28 Oct 2021 • Jinyuan Jia, Hongbin Liu, Neil Zhenqiang Gong
A pre-trained foundation model is like an ``operating system'' of the AI ecosystem.
Anomaly Detection In Surveillance Videos
Self-Supervised Learning
no code implementations • 13 Sep 2021 • Yuankun Yang, Chenyue Liang, Hongyu He, Xiaoyu Cao, Neil Zhenqiang Gong
A key limitation of passive detection is that it cannot detect fake faces that are generated by new deepfake generation methods.
no code implementations • 25 Aug 2021 • Hongbin Liu, Jinyuan Jia, Wenjie Qu, Neil Zhenqiang Gong
EncoderMI can be used 1) by a data owner to audit whether its (public) data was used to pre-train an image encoder without its authorization or 2) by an attacker to compromise privacy of the training data when it is private/sensitive.
6 code implementations • 1 Aug 2021 • Jinyuan Jia, Yupei Liu, Neil Zhenqiang Gong
In particular, our BadEncoder injects backdoors into a pre-trained image encoder such that the downstream classifiers built based on the backdoored image encoder for different downstream tasks simultaneously inherit the backdoor behavior.
no code implementations • 5 Jul 2021 • Xiaoyu Cao, Neil Zhenqiang Gong
Existing studies mainly focused on improving the detection performance in non-adversarial settings, leaving security of deepfake detection in adversarial settings largely unexplored.
no code implementations • 28 May 2021 • Yongji Wu, Lu Yin, Defu Lian, Mingyang Yin, Neil Zhenqiang Gong, Jingren Zhou, Hongxia Yang
With the rapid development of these services in the last two decades, users have accumulated a massive amount of behavior data.
1 code implementation • 28 May 2021 • Yongji Wu, Defu Lian, Neil Zhenqiang Gong, Lu Yin, Mingyang Yin, Jingren Zhou, Hongxia Yang
Inspired by the idea of vector quantization that uses cluster centroids to approximate items, we propose LISA (LInear-time Self Attention), which enjoys both the effectiveness of vanilla self-attention and the efficiency of sparse attention.
no code implementations • CVPR 2021 • Hongbin Liu, Jinyuan Jia, Neil Zhenqiang Gong
Our first major theoretical contribution is that we show PointGuard provably predicts the same label for a 3D point cloud when the number of adversarially modified, added, and/or deleted points is bounded.
no code implementations • 18 Feb 2021 • Minghong Fang, Minghao Sun, Qi Li, Neil Zhenqiang Gong, Jin Tian, Jia Liu
Our empirical results show that the proposed defenses can substantially reduce the estimation errors of the data poisoning attacks.
no code implementations • 3 Feb 2021 • Xiaoyu Cao, Jinyuan Jia, Neil Zhenqiang Gong
We show that our ensemble federated learning with any base federated learning algorithm is provably secure against malicious clients.
no code implementations • 7 Jan 2021 • Hai Huang, Jiaming Mu, Neil Zhenqiang Gong, Qi Li, Bin Liu, Mingwei Xu
Specifically, we formulate our attack as an optimization problem, such that the injected ratings would maximize the number of normal users to whom the target items are recommended.
1 code implementation • 5 Jan 2021 • Bo Hui, Yuchen Yang, Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, Yinzhi Cao
The success of the former heavily depends on the quality of the shadow model, i. e., the transferability between the shadow and the target; the latter, given only blackbox probing access to the target model, cannot make an effective inference of unknowns, compared with MI attacks using shadow models, due to the insufficient number of qualified samples labeled with ground truth membership information.
1 code implementation • 27 Dec 2020 • Xiaoyu Cao, Minghong Fang, Jia Liu, Neil Zhenqiang Gong
Finally, the service provider computes the average of the normalized local model updates weighted by their trust scores as a global model update, which is used to update the global model.
no code implementations • 24 Dec 2020 • Binghui Wang, Jinyuan Jia, Neil Zhenqiang Gong
In this work, we aim to address the key limitation of existing pMRF-based methods.
no code implementations • 7 Dec 2020 • Jinyuan Jia, Yupei Liu, Xiaoyu Cao, Neil Zhenqiang Gong
Moreover, our evaluation results on MNIST and CIFAR10 show that the intrinsic certified robustness guarantees of kNN and rNN outperform those provided by state-of-the-art certified defenses.
no code implementations • ICLR 2022 • Jinyuan Jia, Binghui Wang, Xiaoyu Cao, Hongbin Liu, Neil Zhenqiang Gong
For instance, our method can build a classifier that achieves a certified top-3 accuracy of 69. 2\% on ImageNet when an attacker can arbitrarily perturb 5 pixels of a testing image.
no code implementations • 26 Oct 2020 • Jinyuan Jia, Binghui Wang, Neil Zhenqiang Gong
Moreover, to be robust against post-processing, we leverage Turbo codes, a type of error-correcting codes, to encode the message before embedding it to the DNN classifier.
no code implementations • 24 Aug 2020 • Binghui Wang, Jinyuan Jia, Xiaoyu Cao, Neil Zhenqiang Gong
Specifically, we prove the certified robustness guarantee of any GNN for both node and graph classifications against structural perturbation.
Cryptography and Security
no code implementations • 22 Aug 2020 • Hongbin Liu, Jinyuan Jia, Neil Zhenqiang Gong
Bagging, a popular ensemble learning framework, randomly creates some subsamples of the training data, trains a base model for each subsample using a base learner, and takes majority vote among the base models when making predictions.
1 code implementation • 11 Aug 2020 • Jinyuan Jia, Xiaoyu Cao, Neil Zhenqiang Gong
Specifically, we show that bagging with an arbitrary base learning algorithm provably predicts the same label for a testing example when the number of modified, deleted, and/or inserted training examples is bounded by a threshold.
2 code implementations • 19 Jun 2020 • Zaixi Zhang, Jinyuan Jia, Binghui Wang, Neil Zhenqiang Gong
Specifically, we propose a \emph{subgraph based backdoor attack} to GNN for graph classification.
1 code implementation • 5 May 2020 • Xinlei He, Jinyuan Jia, Michael Backes, Neil Zhenqiang Gong, Yang Zhang
In this work, we propose the first attacks to steal a graph from the outputs of a GNN model that is trained on the graph.
no code implementations • 26 Feb 2020 • Binghui Wang, Xiaoyu Cao, Jinyuan Jia, Neil Zhenqiang Gong
Specifically, in this work, we study the feasibility and effectiveness of certifying robustness against backdoor attacks using a recent technique called randomized smoothing.
no code implementations • 19 Feb 2020 • Minghong Fang, Neil Zhenqiang Gong, Jia Liu
Given the number of fake users the attacker can inject, we formulate the crafting of rating scores for the fake users as an optimization problem.
no code implementations • 9 Feb 2020 • Jinyuan Jia, Binghui Wang, Xiaoyu Cao, Neil Zhenqiang Gong
However, several recent studies showed that community detection is vulnerable to adversarial structural perturbation.
1 code implementation • ICLR 2020 • Jinyuan Jia, Xiaoyu Cao, Binghui Wang, Neil Zhenqiang Gong
For example, our method can obtain an ImageNet classifier with a certified top-5 accuracy of 62. 8\% when the $\ell_2$-norms of the adversarial perturbations are less than 0. 5 (=127/255).
no code implementations • 26 Nov 2019 • Minghong Fang, Xiaoyu Cao, Jinyuan Jia, Neil Zhenqiang Gong
Our empirical results on four real-world datasets show that our attacks can substantially increase the error rates of the models learnt by the federated learning methods that were claimed to be robust against Byzantine failures of some client devices.
no code implementations • 5 Nov 2019 • Xiaoyu Cao, Jinyuan Jia, Neil Zhenqiang Gong
Local Differential Privacy (LDP) protocols enable an untrusted data collector to perform privacy-preserving data analytics.
Data Poisoning
Cryptography and Security
Distributed, Parallel, and Cluster Computing
no code implementations • 28 Oct 2019 • Xiaoyu Cao, Jinyuan Jia, Neil Zhenqiang Gong
Our key observation is that a DNN classifier can be uniquely represented by its classification boundary.
3 code implementations • 23 Sep 2019 • Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, Neil Zhenqiang Gong
Specifically, given a black-box access to the target classifier, the attacker trains a binary classifier, which takes a data sample's confidence score vector predicted by the target classifier as an input and predicts the data sample to be a member or non-member of the target classifier's training dataset.
no code implementations • 17 Sep 2019 • Jinyuan Jia, Neil Zhenqiang Gong
To defend against inference attacks, we can add carefully crafted noise into the public data to turn them into adversarial examples, such that attackers' classifiers make incorrect predictions for the private data.
no code implementations • 1 Mar 2019 • Binghui Wang, Neil Zhenqiang Gong
Results show that our attacks 1) can effectively evade graph-based classification methods; 2) do not require access to the true parameters, true training dataset, and/or complete graph; and 3) outperform the existing attack for evading collective classification methods and some graph neural network methods.
Cryptography and Security
no code implementations • 4 Dec 2018 • Binghui Wang, Jinyuan Jia, Neil Zhenqiang Gong
To address the computational challenge, we propose to jointly learn the edge weights and propagate the reputation scores, which is essentially an approximate solution to the optimization problem.
no code implementations • 11 Sep 2018 • Minghong Fang, Guolei Yang, Neil Zhenqiang Gong, Jia Liu
To address the challenge, we formulate the poisoning attacks as an optimization problem, solving which determines the rating scores for the fake users.
1 code implementation • 13 May 2018 • Jinyuan Jia, Neil Zhenqiang Gong
Specifically, game-theoretic defenses require solving intractable optimization problems, while correlation-based defenses incur large utility loss of users' public data.
no code implementations • 14 Feb 2018 • Binghui Wang, Neil Zhenqiang Gong
In this work, we propose attacks on stealing the hyperparameters that are learned by a learner.
no code implementations • 17 Sep 2017 • Xiaoyu Cao, Neil Zhenqiang Gong
Our key observation is that adversarial examples are close to the classification boundary.