Search Results for author:
Found 46 papers, 22 papers with code
Re-Imagining Multimodal Instruction Tuning: A Representation View
Multimodal instruction tuning has proven to be an effective strategy for achieving zero-shot generalization by fine-tuning pre-trained Large Multimodal Models (LMMs) with instruction-following data.
KGIF: Optimizing Relation-Aware Recommendations with Knowledge Graph Information Fusion
This study introduces the Knowledge Graph Attention Network with Information Fusion (KGIF), a specialized framework designed to merge entity and relation embeddings explicitly through a tailored self-attention mechanism.
Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics
In particular, we introduce two untargeted attack objectives that leverage spatial foundations to destabilize robotic actions, and a targeted attack objective that manipulates the robotic trajectory.
Target-driven Attack for Large Language Models
To solve this problem, we propose our target-driven black-box attack method to maximize the KL divergence between the conditional probabilities of the clean text and the attack text to redefine the attack's goal.
Visual Fourier Prompt Tuning
To address this challenge, we draw inspiration from human visual cognition, and propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models.
M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
Instruction tuning has emerged as an effective strategy for achieving zero-shot generalization by finetuning pretrained models on diverse multimodal tasks.
Visual Agents as Fast and Slow Thinkers
With this novel design, we advocate a flexible system, hierarchical reasoning capabilities, and a transparent decision-making pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence.
Ranked #198 on
Visual Question Answering
on MM-Vet
Radiance Field Learners As UAV First-Person Viewers
First-Person-View (FPV) holds immense potential for revolutionizing the trajectory of Unmanned Aerial Vehicles (UAVs), offering an exhilarating avenue for navigating complex building structures.
Diff-PIC: Revolutionizing Particle-In-Cell Nuclear Fusion Simulation with Diffusion Models
The rapid development of AI highlights the pressing need for sustainable energy, a critical global challenge for decades.
Inertial Confinement Fusion Forecasting via Large Language Models
Controlled fusion energy is deemed pivotal for the advancement of human civilization.
AMD: Automatic Multi-step Distillation of Large-scale Vision Models
Transformer-based architectures have become the de-facto standard models for diverse vision tasks owing to their superior performance.
EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction
Specifically, we employ a heavy encoder to separately encode the product context and attribute.
Self-supervised Adversarial Training of Monocular Depth Estimation against Physical-World Attacks
Our experiments with two representative MDE networks demonstrate improved robustness against various adversarial attacks, with minimal impact on benign performance.
ProMotion: Prototypes As Motion Learners
We capitalize on a dual mechanism involving the feature denoiser and the prototypical learner to decipher the intricacies of motion.
Prototypical Transformer as Unified Motion Learners
In this work, we introduce the Prototypical Transformer (ProtoFormer), a general and unified framework that approaches various motion tasks from a prototype perspective.
SSGA-Net: Stepwise Spatial Global-local Aggregation Networks for for Autonomous Driving
Spatial global-local aggregation fuses the local information from the neighboring frames and global semantics from the current frame to eliminate the feature degradation; 3).
BadPart: Unified Black-box Adversarial Patch Attacks against Pixel-wise Regression Tasks
Our attack prototype, named BadPart, is evaluated on both MDE and OFE tasks, utilizing a total of 7 models.
Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval
Correspondingly, a single text embedding may be less expressive to capture the video embedding and empower the retrieval.
Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?
As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning.
Image Translation as Diffusion Visual Programmers
We introduce the novel Diffusion Visual Programmer (DVP), a neuro-symbolic image translation framework.
Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning
Existing approaches often fully fine-tune a dual-branch encoder-decoder framework with a complicated feature fusion strategy for achieving multimodal semantic segmentation, which is training-costly due to the massive parameter updates in feature extraction and fusion.
Ranked #6 on
Semantic Segmentation
on NYU Depth v2
CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation
The advancement of computer vision has pushed visual analysis tasks from still images to the video domain.
ClusterFormer: Clustering As A Universal Visual Learner
This paper presents CLUSTERFORMER, a universal vision model that is based on the CLUSTERing paradigm with TransFORMER.
E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning
Specifically, we introduce a set of learnable key-value prompts and visual prompts into self-attention and input layers, respectively, to improve the effectiveness of model fine-tuning.
MixPAVE: Mix-Prompt Tuning for Few-shot Product Attribute Value Extraction
In this work, we propose a novel prompt tuning approach with Mixed Prompts for few-shot Attribute Value Extraction, namely MixPAVE.
CLUSTSEG: Clustering for Universal Segmentation
We present CLUSTSEG, a general, transformer-based framework that tackles different image segmentation tasks (i. e., superpixel, semantic, instance, and panoptic) through a unified neural clustering scheme.
Fusion is Not Enough: Single Modal Attacks on Fusion Models for 3D Object Detection
We argue that the weakest link of fusion models depends on their most vulnerable modality, and propose an attack framework that targets advanced camera-LiDAR fusion-based 3D object detection models through camera-only adversarial attacks.
TransFlow: Transformer as Flow Learner
Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement.
Exploiting Logic Locking for a Neural Trojan Attack on Machine Learning Accelerators
If an incorrect secret key is used, a set of deterministic errors is produced in locked modules, restricting unauthorized use.
Adversarial Training of Self-supervised Monocular Depth Estimation against Physical-World Attacks
We improve adversarial robustness against physical-world attacks using L0-norm-bounded perturbation in training.
Learning Equivariant Segmentation with Instance-Unique Querying
Prevalent state-of-the-art instance segmentation methods fall into a query-based scheme, in which instance masks are derived by querying the image feature using a set of instance-aware embeddings.
Visual Recognition with Deep Nearest Centroids
We devise deep nearest centroids (DNC), a conceptually elegant yet surprisingly effective network for large-scale visual recognition, by revisiting Nearest Centroids, one of the most classic and simple classifiers.
Towards Unbiased Label Distribution Learning for Facial Pose Estimation Using Anisotropic Spherical Gaussian
In this paper, we propose an Anisotropic Spherical Gaussian (ASG)-based LDL approach for facial pose estimation.
Physical Attack on Monocular Depth Estimation with Optimal Adversarial Patches
Experimental results show that our method can generate stealthy, effective, and robust adversarial patches for different target objects and models and achieves more than 6 meters mean depth estimation error and 93% attack success rate (ASR) in object detection with a patch of 1/9 of the vehicle's rear area.
GL-RG: Global-Local Representation Granularity for Video Captioning
Video captioning is a challenging task as it needs to accurately transform visual understanding into natural language description.
Deep Partial Multiplex Network Embedding
Network embedding is an effective technique to learn the low-dimensional representations of nodes in networks.
WebFormer: The Web-page Transformer for Structure Information Extraction
Structure information extraction refers to the task of extracting structured text fields from web pages, such as extracting a product offer from a shopping page including product title, description, brand and price.
DG-Labeler and DGL-MOTS Dataset: Boost the Autonomous Driving Perception
The existing MOTS studies face two critical challenges: 1) the published datasets inadequately capture the real-world complexity for network training to address various driving settings; 2) the working pipeline annotation tool is under-studied in the literature to improve the quality of MOTS learning examples.
TF-Blender: Temporal Feature Blender for Video Object Detection
One of the popular solutions is to exploit the temporal information and enhance per-frame representation through aggregating features from neighboring frames.
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation
Video instance segmentation (VIS) is a new and critical task in computer vision.
Hierarchical Attention Fusion for Geo-Localization
We extract the hierarchical feature maps from a convolutional neural network (CNN) and organically fuse the extracted features for image representations.
Semantic Aware Data Augmentation for Cell Nuclei Microscopical Images With Artificial Neural Networks
The importance of this subject is nested in the amount of training data that artificial neural networks need to accurately identify and segment objects in images and the infeasibility of acquiring a sufficient dataset within the biomedical field.
DenserNet: Weakly Supervised Visual Localization Using Multi-scale Feature Aggregation
In this work, we introduce a Denser Feature Network (DenserNet) for visual localization.
A Vector-based Representation to Enhance Head Pose Estimation
This paper proposes to use the three vectors in a rotation matrix as the representation in head pose estimation and develops a new neural network based on the characteristic of such representation.
Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning
Memory is important for the agent to avoid repeating certain tasks unnecessarily and in order for it to adapt adequately to new scenes, therefore, we make use of meta-learning.
Ranked #1 on
Visual Navigation
on AI2-THOR
Visual Localization for Autonomous Driving: Mapping the Accurate Location in the City Maze
It is a common practice for vehicles to use GPS to acquire location information.
