Search Results for author:
Found 90 papers, 62 papers with code
IPDN: Image-enhanced Prompt Decoding Network for 3D Referring Expression Segmentation
To tackle intent ambiguity, we designed a Prompt-Aware Decoder (PAD) that guides the decoding process by deriving task-driven signals from the interaction between the expression and visual features.
StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization
Story visualization has gained increasing attention in artificial intelligence.
FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression
Despite a big leap forward in capability, multimodal large language models (MLLMs) tend to behave like a sloth in practical use, i. e., slow response and large latency.
Ranked #68 on
Visual Question Answering
on MM-Vet
RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation
The RG-SAN consists of the Text-driven Localization Module (TLM) and the Rule-guided Weak Supervision (RWS) strategy.
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
In particular, we reveal that visual tokens will stop contributing to reasoning when the text tokens receive enough image information, yielding obvious visual redundancy.
Any-to-3D Generation via Hybrid Diffusion Supervision
To our knowledge, this is the first method to generate 3D objects from any modality prompts.
$γ-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models
In $\gamma$-MoD, a novel metric is proposed to guide the deployment of MoDs in the MLLM, namely rank of attention maps (ARank).
Ranked #166 on
Visual Question Answering
on MM-Vet
DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion
The rapid progress of Deepfake technology has made face swapping highly realistic, raising concerns about the malicious use of fabricated facial content.
I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing
We will open-source I2EBench, including all instructions, input images, human annotations, edited images from all evaluated methods, and a simple script for evaluating the results from new IIE models.
TraDiffusion: Trajectory-Based Training-Free Image Generation
In this work, we propose a training-free, trajectory-based controllable T2I approach, termed TraDiffusion.
StealthDiffusion: Towards Evading Diffusion Forensic Detection through Diffusion Model
The rapid progress in generative models has given rise to the critical task of AI-Generated Content Stealth (AIGC-S), which aims to create AI-generated images that can evade both forensic detectors and human inspection.
ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results.
3D-GRES: Generalized 3D Referring Expression Segmentation
3D Referring Expression Segmentation (3D-RES) is dedicated to segmenting a specific instance within a 3D space based on a natural language description.
INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model
However, the quadratic complexity of the vision encoder in MLLMs constrains the resolution of input images.
Ranked #162 on
Visual Question Answering
on MM-Vet
Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model
Specifically, we introduce the DiffPNG framework, a straightforward yet effective approach that fully capitalizes on the diffusion's architecture for segmentation by decomposing the process into a sequence of localization, segmentation, and refinement steps.
Multi-branch Collaborative Learning Network for 3D Visual Grounding
3D referring expression comprehension (3DREC) and segmentation (3DRES) have overlapping objectives, indicating their potential for collaboration.
Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models
We identify three types of relationship co-occurrences that lead to hallucinations: relationship-relationship, subject-relationship, and relationship-object.
AnyTrans: Translate AnyText in the Image with Large Scale Models
This paper introduces AnyTrans, an all-encompassing framework for the task-Translate AnyText in the Image (TATI), which includes multilingual text translation and text fusion within images.
Beat: Bi-directional One-to-Many Embedding Alignment for Text-based Person Retrieval
To address this issue, we propose an effective bi-directional one-to-many embedding paradigm that offers a clear optimization direction for each sample, thus mitigating the optimization problem.
SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation
These strategies are designed to extract the most accurate masks from SAM's output, thus guiding the training of the student model with enhanced precision.
Image Captioning via Dynamic Path Customization
This paper explores a novel dynamic network for vision and language tasks, where the inferring structure is customized on the fly for different inputs.
X-Oscar: A Progressive Framework for High-quality Text-guided 3D Animatable Avatar Generation
Recent advancements in automatic 3D avatar generation guided by text have made significant progress.
Deep Instruction Tuning for Segment Anything Model
Recently, Segment Anything Model (SAM) has become a research hotspot in the fields of multimedia and computer vision, which exhibits powerful yet versatile capabilities on various (un) conditional image segmentation tasks.
DiffusionFace: Towards a Comprehensive Dataset for Diffusion-Based Face Forgery Analysis
The rapid progress in deep learning has given rise to hyper-realistic facial forgery methods, leading to concerns related to misinformation and security risks.
Toward Open-Set Human Object Interaction Detection
To address this challenge, we introduce a simple Disentangled HOI Detection (DHD) model for detecting novel relationships by integrating an open-set object detector with a Visual Language Model (VLM).
Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models
In this paper, we propose a novel parameter and computation efficient tuning method for Multi-modal Large Language Models (MLLMs), termed Efficient Attention Skipping (EAS).
Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization
Text-to-3D-aware face (T3D Face) generation and manipulation is an emerging research hot spot in machine learning, which still suffers from low efficiency and poor quality.
Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
Contrary to previous works, we study this problem from the perspective of image resolution, and reveal that a combination of low- and high-resolution visual features can effectively mitigate this shortcoming.
Ranked #152 on
Visual Question Answering
on MM-Vet
Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks
In particular, InstDiffEdit aims to employ the cross-modal attention ability of existing diffusion models to achieve instant mask guidance during the diffusion steps.
Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation
Referring Remote Sensing Image Segmentation (RRSIS) is a new challenge that combines computer vision and natural language processing, delineating specific regions in aerial images as described by textual queries.
X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation
Nevertheless, a substantial domain gap exists between 2D images and 3D assets, primarily attributed to variations in camera-related attributes and the exclusive presence of foreground objects.
Towards Omni-supervised Referring Expression Segmentation
To address this issue, we propose a new learning task for RES called Omni-supervised Referring Expression Segmentation (Omni-RES), which aims to make full use of unlabeled, fully labeled and weakly labeled data, e. g., referring points or grounding boxes, for efficient RES training.
Semi-Supervised Panoptic Narrative Grounding
Remarkably, our SS-PNG-NW+ outperforms fully-supervised models with only 30% and 50% supervision data, exceeding their performance by 0. 8% and 1. 1% respectively.
NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learning
To address this, we introduce two cascading modules based on the barycenter of the mask, which are Coordinate Guided Aggregation (CGA) and Barycenter Driven Localization (BDL), responsible for segmentation and detection, respectively.
JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues
The rising importance of 3D understanding, pivotal in computer vision, autonomous driving, and robotics, is evident.
Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
In this paper, we aim at parameter and computation efficient transfer learning (PCETL) for VLP models.
3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation
In 3D Referring Expression Segmentation (3D-RES), the earlier approach adopts a two-stage paradigm, extracting segmentation proposals and then matching them with referring expressions.
Continual Face Forgery Detection via Historical Distribution Preserving
In this paper, we focus on a novel and challenging problem: Continual Face Forgery Detection (CFFD), which aims to efficiently learn from new forgery attacks without forgetting previous ones.
Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation
Insufficient synergy neglects the idea that a robust 3D representation should align with the joint vision-language space, rather than independently aligning with each modality.
Towards General Visual-Linguistic Face Forgery Detection
To address this issues, in this paper, we propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer
Sharpness-Aware Minimization (SAM) is a popular solution that smooths the loss landscape by minimizing the maximized change of training loss when adding a perturbation to the weight.
Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting
In addition, we also experiment DVP with the recently popular adapter approach to keep the most parameters of PLMs intact when adapting to VL tasks, helping PLMs achieve a quick shift between single- and multi-modal tasks.
Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision-language instructed model as LaVIN.
X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance
Text-driven 3D stylization is a complex and crucial task in the fields of computer vision (CV) and computer graphics (CG), aimed at transforming a bare mesh to fit a target text.
Active Teacher for Semi-Supervised Object Detection
In this paper, we study teacher-student learning from the perspective of data initialization and propose a novel algorithm called Active Teacher(Source code are available at: \url{https://github. com/HunterJ-Lin/ActiveTeacher}) for semi-supervised object detection (SSOD).
Towards End-to-end Semi-supervised Learning for One-stage Object Detection
In addition to this challenge, we also reveal two key issues in one-stage SSOD, which are low-quality pseudo-labeling and multi-task optimization conflict, respectively.
Towards Efficient Visual Adaption via Structural Re-parameterization
Experimental results show the superior performance and efficiency of RepAdapter than the state-of-the-art PETL methods.
Towards Local Visual Modeling for Image Captioning
In this paper, we study the local visual modeling with grid features for image captioning, which is critical for generating accurate and detailed captions.
Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Network
Extensive experiments on the PNG benchmark dataset demonstrate the effectiveness and efficiency of our method.
RefCLIP: A Universal Teacher for Weakly Supervised Referring Expression Comprehension
Based on RefCLIP, we further propose the first model-agnostic weakly supervised training scheme for existing REC models, where RefCLIP acts as a mature teacher to generate pseudo-labels for teaching common REC models.
RefTeacher: A Strong Baseline for Semi-Supervised Referring Expression Comprehension
In this paper, we present the first attempt of semi-supervised learning for REC and propose a strong baseline method called RefTeacher.
Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach
One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight.
Clover: Towards A Unified Video-Language Alignment and Fusion Model
We then introduce \textbf{Clover}\textemdash a Correlated Video-Language pre-training method\textemdash towards a universal Video-Language model for solving multiple video understanding tasks with neither performance nor efficiency compromise.
Ranked #1 on
Video Question Answering
on LSMDC-FiB
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval
However, cross-grained contrast, which is the contrast between coarse-grained representations and fine-grained representations, has rarely been explored in prior research.
Ranked #12 on
Video Retrieval
on MSVD
A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehension
But the most encouraging finding is that with much less training overhead and parameters, SimREC can still achieve better performance than a set of large-scale pre-trained models, e. g., UNITER and VILLA, portraying the special role of REC in existing V&L research.
Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks
Despite the exciting performance, Transformer is criticized for its excessive parameters and computation cost.
PixelFolder: An Efficient Progressive Pixel Synthesis Network for Image Generation
Pixel synthesis is a promising research paradigm for image generation, which can well exploit pixel-wise prior knowledge for generation.
End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation
Extensive experiments on HICO-Det dataset demonstrate that our model discovers potential interactive pairs and enables the recognition of unseen HOIs.
Human-Object Interaction Detection
Knowledge Distillation
+4
SeqTR: A Simple yet Universal Network for Visual Grounding
In this paper, we propose a simple yet universal network termed SeqTR for visual grounding tasks, e. g., phrase localization, referring expression comprehension (REC) and segmentation (RES).
Ranked #11 on
Referring Expression Segmentation
on RefCOCO testB
Global2Local: A Joint-Hierarchical Attention for Video Captioning
Recently, automatic video captioning has attracted increasing attention, where the core challenge lies in capturing the key semantic items, like objects and actions as well as their spatial-temporal correlations from the redundant frames and semantic content.
DIFNet: Boosting Visual Information Flow for Image Captioning
Current Image captioning (IC) methods predict textual words sequentially based on the input visual information from the visual feature extractor and the partially generated sentence information.
Towards Language-guided Visual Recognition via Dynamic Convolutions
Based on the LaConv module, we further build the first fully language-driven convolution network, termed as LaConvNet, which can unify the visual recognition and multi-modal reasoning in one forward structure.
RSTNet: Captioning With Adaptive Attention on Visual and Non-Visual Words
Then, we build a BERTbased language model to extract language context and propose Adaptive-Attention (AA) module on top of a transformer decoder to adaptively measure the contribution of visual and language cues before making decisions for word prediction.
Dual-Level Collaborative Transformer for Image Captioning
Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning.
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering
Due to the superior ability of global dependency modeling, Transformer and its variants have become the primary choice of many vision-and-language tasks.
Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network
The latter contains a Global Adaptive Controller that can adaptively fuse the global information into the decoder to guide the caption generation.
Fast Class-wise Updating for Online Hashing
To achieve fast online adaptivity, a class-wise updating method is developed to decompose the binary code learning and alternatively renew the hash functions in a class-wise fashion, which well addresses the burden on large amounts of training batches.
Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation
In addition, we address a key challenge in this multi-task setup, i. e., the prediction conflict, with two innovative designs namely, Consistency Energy Maximization (CEM) and Adaptive Soft Non-Located Suppression (ASNLS).
Generalized Referring Expression Comprehension
Referring Expression
+2
A Real-time Global Inference Network for One-stage Referring Expression Comprehension
Referring Expression Comprehension (REC) is an emerging research spot in computer vision, which refers to detecting the target region in an image given an text description.
Variational Structured Semantic Inference for Diverse Image Captioning
To model these two inherent diversities in image captioning, we propose a Variational Structured Semantic Inferring model (termed VSSI-cap) executed in a novel structured encoder-inferer-decoder schema.
SSAH: Semi-supervised Adversarial Deep Hashing with Self-paced Hard Sample Generation
In this paper, we propose a novel Semi-supervised Self-pace Adversarial Hashing method, named SSAH to solve the above problems in a unified framework.
Hadamard Codebook Based Deep Hashing
Specifically, we utilize an off-the-shelf algorithm to generate a binary Hadamard codebook to satisfy the requirement of bit independence and bit balance, which subsequently serves as the desired outputs of the hash functions learning.
Toward 3D Object Reconstruction from Stereo Images
Inferring the 3D shape of an object from an RGB image has shown impressive results, however, existing methods rely primarily on recognizing the most similar 3D model from the training set to solve the problem.
Sketch-Specific Data Augmentation for Freehand Sketch Recognition
Conventional methods for this task often rely on the availability of the temporal order of sketch strokes, additional cues acquired from different modalities and supervised augmentation of sketch datasets with real images, which also limit the applicability and feasibility of these methods in real scenarios.
Deep Semantic Parsing of Freehand Sketches with Homogeneous Transformation, Soft-Weighted Loss, and Staged Learning
First, we propose a homogeneous transformation method to address the problem of domain adaptation.
Semantic-aware Image Deblurring
Specially, we propose a novel Structured-Spatial Semantic Embedding model for image deblurring (termed S3E-Deblur), which introduces a novel Structured-Spatial Semantic tree model (S3-tree) to bridge two basic tasks in computer vision: image deblurring (ImD) and image captioning (ImC).
Scene-based Factored Attention for Image Captioning
Specifically, the proposed module first embeds the scene concepts into factored weights explicitly and attends the visual information extracted from the input image.
Semi-Supervised Adversarial Monocular Depth Estimation
In this paper, we address the problem of monocular depth estimation when only a limited number of training image-depth pairs are available.
Information Competing Process for Learning Diversified Representations
Learning representations with diversified information remains as an open problem.
Supervised Online Hashing via Similarity Distribution Learning
In this paper, we propose to model the similarity distributions between the input data and the hashing codes, upon which a novel supervised online hashing method, dubbed as Similarity Distribution based Online Hashing (SDOH), is proposed, to keep the intrinsic semantic relationship in the produced Hamming space.
Hadamard Matrix Guided Online Hashing
We then treat the learning of hash functions as a set of binary classification problems to fit the assigned target code.
Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images
Then, a context-aware fusion module is introduced to adaptively select high-quality reconstructions for each part (e. g., table legs) from different coarse 3D volumes to obtain a fused 3D volume.
Ranked #4 on
3D Object Reconstruction
on Data3D−R2N2
Towards Optimal Discrete Online Hashing with Balanced Similarity
In this paper, we propose a novel supervised online hashing method, termed Balanced Similarity for Online Discrete Hashing (BSODH), to solve the above problems in a unified framework.
Semantic and Contrast-Aware Saliency
The two pathways characterize both long-term and short-term attention cues and are integrated dynamically using maxima normalization.
Deep Saliency Hashing
As the core of DSaH, the saliency loss guides the attention network to mine discriminative regions from pairs of images.
GroupCap: Group-Based Image Captioning With Structured Relevance and Diversity Constraints
In offline optimization, we adopt an end-to-end formulation, which jointly trains the visual tree parser, the structured relevance and diversity constraints, as well as the LSTM based captioning model.
The Effectiveness of Instance Normalization: a Strong Baseline for Single Image Dehazing
We propose a novel deep neural network architecture for the challenging problem of single image dehazing, which aims to recover the clear image from a degraded hazy image.
Exploring Implicit Image Statistics for Visual Representativeness Modeling
In this paper, we propose a computational model of visual representativeness by integrating cognitive theories of representativeness heuristics with computer vision and machine learning techniques.
