Search Results for author: Jindong Gu

Found 53 papers, 14 papers with code

Stop Reasoning! When Multimodal LLMs with Chain-of-Thought Reasoning Meets Adversarial Images

no code implementations22 Feb 2024 Zefeng Wang, Zhen Han, Shuo Chen, Fan Xue, Zifeng Ding, Xun Xiao, Volker Tresp, Philip Torr, Jindong Gu

Our research evaluates the adversarial robustness of MLLMs when employing CoT reasoning, finding that CoT marginally improves adversarial robustness against existing attack methods.

Adversarial Robustness

Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images

1 code implementation20 Jan 2024 Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, Wei Liu

Once attackers maliciously induce high energy consumption and latency time (energy-latency cost) during inference of VLMs, it will exhaust computational resources.

Does Few-shot Learning Suffer from Backdoor Attacks?

no code implementations31 Dec 2023 Xinwei Liu, Xiaojun Jia, Jindong Gu, Yuan Xun, Siyuan Liang, Xiaochun Cao

However, in this paper, we propose the Few-shot Learning Backdoor Attack (FLBA) to show that FSL can still be vulnerable to backdoor attacks.

Backdoor Attack Few-Shot Learning

XAI for In-hospital Mortality Prediction via Multimodal ICU Data

1 code implementation29 Dec 2023 Xingqiao Li, Jindong Gu, Zhiyong Wang, Yancheng Yuan, Bo Du, Fengxiang He

To address this issue, this paper proposes an eXplainable Multimodal Mortality Predictor (X-MMP) approaching an efficient, explainable AI solution for predicting in-hospital mortality via multimodal ICU data.

Decision Making Mortality Prediction

Initialization Matters for Adversarial Transfer Learning

no code implementations10 Dec 2023 Andong Hua, Jindong Gu, Zhiyu Xue, Nicholas Carlini, Eric Wong, Yao Qin

Specifically, we reveal that with a standard pretrained model, Parameter-Efficient Finetuning~(PEFT) methods either fail to be adversarially robust or continue to exhibit significantly degraded adversarial robustness on downstream tasks, even with adversarial training during finetuning.

Adversarial Robustness Image Classification +1

OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization

no code implementations7 Dec 2023 Dongchen Han, Xiaojun Jia, Yang Bai, Jindong Gu, Yang Liu, Xiaochun Cao

Investigating the generation of high-transferability adversarial examples is crucial for uncovering VLP models' vulnerabilities in practical scenarios.

Adversarial Attack Data Augmentation +2

TranSegPGD: Improving Transferability of Adversarial Examples on Semantic Segmentation

no code implementations3 Dec 2023 Xiaojun Jia, Jindong Gu, Yihao Huang, Simeng Qin, Qing Guo, Yang Liu, Xiaochun Cao

At the second stage, the pixels are divided into different branches based on their transferable property which is dependent on Kullback-Leibler divergence.

Adversarial Attack Image Classification +2

Improving Adversarial Transferability via Model Alignment

no code implementations30 Nov 2023 Avery Ma, Amir-Massoud Farahmand, Yangchen Pan, Philip Torr, Jindong Gu

During the alignment process, the parameters of the source model are fine-tuned to minimize an alignment loss.

Understanding and Improving In-Context Learning on Vision-language Models

no code implementations29 Nov 2023 Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Philip Torr, Volker Tresp, Jindong Gu

Our findings indicate that ICL in VLMs is predominantly driven by the textual information in the demonstrations whereas the visual information in the demonstrations barely affects the ICL performance.

In-Context Learning

Benchmarking Robustness of Text-Image Composed Retrieval

no code implementations24 Nov 2023 Shitong Sun, Jindong Gu, Shaogang Gong

In this paper, we perform the first robustness study and establish three new diversified benchmarks for systematic analysis of text-image composed retrieval against natural corruptions in both vision and text and further probe textural understanding.

Attribute Benchmarking +1

SPOT! Revisiting Video-Language Models for Event Understanding

no code implementations21 Nov 2023 Gengyuan Zhang, Jinhe Bi, Jindong Gu, Yanyu Chen, Volker Tresp

This raises a question: with such weak supervision, can video representation in video-language models gain the ability to distinguish even factual discrepancies in textual description and understand fine-grained events?

Attribute Video Understanding

Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworks

no code implementations24 Oct 2023 Xiaojun Jia, Jianshu Li, Jindong Gu, Yang Bai, Xiaochun Cao

Besides, we provide theoretical analysis to show the model robustness can be improved by the single-step adversarial training with sampled subnetworks.

Boosting Fair Classifier Generalization through Adaptive Priority Reweighing

1 code implementation15 Sep 2023 Zhihao Hu, Yiran Xu, Mengnan Du, Jindong Gu, Xinmei Tian, Fengxiang He

Our adaptive reweighing method prioritizes samples closer to the decision boundary and assigns a higher weight to improve the generalizability of fair classifiers.

Decision Making Fairness

Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks

no code implementations12 Sep 2023 Jindong Gu, Fangyun Wei, Philip Torr, Han Hu

In this work, we first taxonomize the stochastic defense strategies against QBBA.

Multi-event Video-Text Retrieval

1 code implementation ICCV 2023 Gengyuan Zhang, Jisen Ren, Jindong Gu, Volker Tresp

In this study, we introduce the Multi-event Video-Text Retrieval (MeVTR) task, addressing scenarios in which each video contains multiple different events, as a niche scenario of the conventional Video-Text Retrieval Task.

Language Modelling Retrieval +2

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

no code implementations21 Aug 2023 Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, Volker Tresp

FedDAT is the first approach that enables an efficient distributed finetuning of foundation models for a variety of heterogeneous Vision-Language tasks.

Federated Learning Knowledge Distillation +1

FedPop: Federated Population-based Hyperparameter Tuning

no code implementations16 Aug 2023 Haokun Chen, Denis Krompass, Jindong Gu, Volker Tresp

Similar to conventional ML pipelines, the client local optimization and server aggregation procedure in FL are sensitive to the hyperparameter (HP) selection.

Computational Efficiency Evolutionary Algorithms +1

A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

1 code implementation24 Jul 2023 Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr

This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e. g. Flamingo), image-text matching models (e. g.

Image-text matching Language Modelling +4

Reliable Evaluation of Adversarial Transferability

no code implementations14 Jun 2023 Wenqian Yu, Jindong Gu, Zhijiang Li, Philip Torr

Adversarial examples (AEs) with small adversarial perturbations can mislead deep neural networks (DNNs) into wrong predictions.

Towards Robust Prompts on Vision-Language Models

no code implementations17 Apr 2023 Jindong Gu, Ahmad Beirami, Xuezhi Wang, Alex Beutel, Philip Torr, Yao Qin

With the advent of vision-language models (VLMs) that can perform in-context and prompt-based learning, how can we design prompting approaches that robustly generalize to distribution shift and can be used on novel classes outside the support set of the prompts?

In-Context Learning

Backdoor Defense via Adaptively Splitting Poisoned Dataset

1 code implementation CVPR 2023 Kuofeng Gao, Yang Bai, Jindong Gu, Yong Yang, Shu-Tao Xia

With the split clean data pool and polluted data pool, ASD successfully defends against backdoor attacks during training.

backdoor defense

Influencer Backdoor Attack on Semantic Segmentation

no code implementations21 Mar 2023 Haoheng Lan, Jindong Gu, Philip Torr, Hengshuang Zhao

In this work, we explore backdoor attacks on segmentation models to misclassify all pixels of a victim class by injecting a specific trigger on non-victim pixels during inferences, which is dubbed Influencer Backdoor Attack (IBA).

Backdoor Attack Position +2

Explainability and Robustness of Deep Visual Classification Models

no code implementations3 Jan 2023 Jindong Gu

The vulnerability of deep neural networks poses challenges to current visual classification models.

Classification Image Classification +1

Do DALL-E and Flamingo Understand Each Other?

no code implementations ICCV 2023 Hang Li, Jindong Gu, Rajat Koner, Sahand Sharifzadeh, Volker Tresp

To study this question, we propose a reconstruction task where Flamingo generates a description for a given image and DALL-E uses this description as input to synthesize a new image.

Image Captioning Image Reconstruction +3

SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness

1 code implementation25 Jul 2022 Jindong Gu, Hengshuang Zhao, Volker Tresp, Philip Torr

Since SegPGD can create more effective adversarial examples, the adversarial training with our SegPGD can boost the robustness of segmentation models.

Adversarial Attack Segmentation +1

Towards Efficient Adversarial Training on Vision Transformers

no code implementations21 Jul 2022 Boxi Wu, Jindong Gu, Zhifeng Li, Deng Cai, Xiaofei He, Wei Liu

Vision Transformer (ViT), as a powerful alternative to Convolutional Neural Network (CNN), has received much attention.

Watermark Vaccine: Adversarial Attacks to Prevent Watermark Removal

1 code implementation17 Jul 2022 Xinwei Liu, Jian Liu, Yang Bai, Jindong Gu, Tao Chen, Xiaojun Jia, Xiaochun Cao

Inspired by the vulnerability of DNNs on adversarial perturbations, we propose a novel defence mechanism by adversarial machine learning for good.

FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation

no code implementations ICCV 2023 Haokun Chen, Ahmed Frikha, Denis Krompass, Jindong Gu, Volker Tresp

Real-world applications usually involve a distribution shift across the datasets of the different clients, which hurts the generalization ability of the clients to unseen samples from their respective data distributions.

Federated Learning

ECOLA: Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations

no code implementations17 Mar 2022 Zhen Han, Ruotong Liao, Jindong Gu, Yao Zhang, Zifeng Ding, Yujia Gu, Heinz Köppl, Hinrich Schütze, Volker Tresp

Since conventional knowledge embedding models cannot take full advantage of the abundant textual information, there have been extensive research efforts in enhancing knowledge embedding using texts.

Knowledge Graph Embedding Link Prediction +1

Adversarial Examples on Segmentation Models Can be Easy to Transfer

no code implementations22 Nov 2021 Jindong Gu, Hengshuang Zhao, Volker Tresp, Philip Torr

The high transferability achieved by our method shows that, in contrast to the observations in previous work, adversarial examples on a segmentation model can be easy to transfer to other segmentation models.

Adversarial Robustness Attribute +5

Are Vision Transformers Robust to Patch Perturbations?

no code implementations20 Nov 2021 Jindong Gu, Volker Tresp, Yao Qin

However, when ViTs are attacked by an adversary, the attention mechanism can be easily fooled to focus more on the adversarially perturbed patches and cause a mistake.

Image Classification

Are Vision Transformers Robust to Patch-wise Perturbations?

no code implementations29 Sep 2021 Jindong Gu, Volker Tresp, Yao Qin

Based on extensive qualitative and quantitative experiments, we discover that ViT's stronger robustness to natural corrupted patches and higher vulnerability against adversarial patches are both caused by the attention mechanism.

Image Classification

Simple Distillation Baselines for Improving Small Self-supervised Models

1 code implementation21 Jun 2021 Jindong Gu, Wei Liu, Yonglong Tian

While large self-supervised models have rivalled the performance of their supervised counterparts, small models still struggle.

Attacking Adversarial Attacks as A Defense

no code implementations9 Jun 2021 Boxi Wu, Heng Pan, Li Shen, Jindong Gu, Shuai Zhao, Zhifeng Li, Deng Cai, Xiaofei He, Wei Liu

In this work, we find that the adversarial attacks can also be vulnerable to small perturbations.

Quantifying Predictive Uncertainty in Medical Image Analysis with Deep Kernel Learning

1 code implementation1 Jun 2021 Zhiliang Wu, Yinchong Yang, Jindong Gu, Volker Tresp

We propose an uncertainty-aware deep kernel learning model which permits the estimation of the uncertainty in the prediction by a pipeline of a Convolutional Neural Network and a sparse Gaussian Process.

Capsule Network is Not More Robust than Convolutional Network

no code implementations CVPR 2021 Jindong Gu, Volker Tresp, Han Hu

The examination reveals five major new/different components in CapsNet: a transformation process, a dynamic routing layer, a squashing function, a marginal loss other than cross-entropy loss, and an additional class-conditional reconstruction loss for regularization.

Image Classification

Effective and Efficient Vote Attack on Capsule Networks

1 code implementation ICLR 2021 Jindong Gu, Baoyuan Wu, Volker Tresp

As alternatives to CNNs, the recently proposed Capsule Networks (CapsNets) are shown to be more robust to white-box attacks than CNNs under popular attack protocols.

Adversarial Robustness

Interpretable Graph Capsule Networks for Object Recognition

no code implementations3 Dec 2020 Jindong Gu, Volker Tresp

In the proposed model, individual classification explanations can be created effectively and efficiently.

Adversarial Robustness Object +1

Introspective Learning by Distilling Knowledge from Online Self-explanation

no code implementations19 Sep 2020 Jindong Gu, Zhiliang Wu, Volker Tresp

Motivated by the conclusion, we propose an implementation of introspective learning by distilling knowledge from online self-explanations.

Knowledge Distillation

Search for Better Students to Learn Distilled Knowledge

no code implementations30 Jan 2020 Jindong Gu, Volker Tresp

The knowledge of a well-performed teacher is distilled to a student with a small architecture.

Knowledge Distillation Model Compression

Neural Network Memorization Dissection

no code implementations21 Nov 2019 Jindong Gu, Volker Tresp

What is the difference between DNNs trained with random labels and the ones trained with true labels?

Memorization

Improving the Robustness of Capsule Networks to Image Affine Transformations

no code implementations CVPR 2020 Jindong Gu, Volker Tresp

Our investigation reveals that the routing procedure contributes neither to the generalization ability nor to the affine robustness of the CapsNets.

Semantics for Global and Local Interpretation of Deep Neural Networks

no code implementations21 Oct 2019 Jindong Gu, Volker Tresp

Deep neural networks (DNNs) with high expressiveness have achieved state-of-the-art performance in many tasks.

Contextual Prediction Difference Analysis for Explaining Individual Image Classifications

no code implementations21 Oct 2019 Jindong Gu, Volker Tresp

In this work, we first show that PDA can suffer from saturated classifiers.

Understanding Bias in Machine Learning

no code implementations2 Sep 2019 Jindong Gu, Daniela Oelke

Bias is known to be an impediment to fair decisions in many domains such as human resources, the public sector, health care etc.

BIG-bench Machine Learning

Saliency Methods for Explaining Adversarial Attacks

no code implementations22 Aug 2019 Jindong Gu, Volker Tresp

The idea behind saliency methods is to explain the classification decisions of neural networks by creating so-called saliency maps.

General Classification

Understanding Individual Decisions of CNNs via Contrastive Backpropagation

2 code implementations5 Dec 2018 Jindong Gu, Yinchong Yang, Volker Tresp

The experiments and analysis conclude that the explanations generated by LRP are not class-discriminative.

General Classification

Semi-supervised Outlier Detection using Generative And Adversary Framework

no code implementations ICLR 2018 Jindong Gu, Matthias Schubert, Volker Tresp

In the adversarial process of training CorGAN, the Generator is supposed to generate outlier samples for negative class, and the Discriminator as an one-class classifier is trained to distinguish data from training datasets (i. e. positive class) and generated data from the Generator (i. e. negative class).

General Classification Multi-class Classification +2

Cannot find the paper you are looking for? You can Submit a new open access paper.