Search Results for author:
Found 89 papers, 45 papers with code
Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness
Instruction tuning fine-tunes pre-trained Multi-modal Large Language Models (MLLMs) to handle real-world tasks.
SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation
To adapt SILMM to LMMs with continuous features, we propose a diversity mechanism to obtain diverse representations and a kernel-based continuous DPO for alignment.
HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing
HumanEdit bridges this gap by employing human annotators to construct data pairs and administrators to provide feedback.
AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea
Instruction-based image editing aims to modify specific image elements with natural language instructions.
Unified Generative and Discriminative Training for Multi-modal Large Language Models
Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation.
RADAR: Robust Two-stage Modality-incomplete Industrial Anomaly Detection
To address this practical challenge, we introduce a first-of-its-kind study that comprehensively investigates Modality-Incomplete Industrial Anomaly Detection (MIIAD), to consider the imperfect learning environment in which the multimodal information may be incomplete.
Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration
In this paper, we propose UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting.
Align$^2$LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
Recent advances in Multi-modal Large Language Models (MLLMs), such as LLaVA-series models, are driven by massive machine-generated instruction-following data tuning.
NTIRE 2024 Challenge on Stereo Image Super-Resolution: Methods and Results
This paper summarizes the 3rd NTIRE challenge on stereo image super-resolution (SR) with a focus on new solutions and results.
Topological GCN for Improving Detection of Hip Landmarks from B-Mode Ultrasound Images
In this work, we propose a novel hip landmark detection model by integrating the Topological GCN (TGCN) with an Improved Conformer (TGCN-ICF) into a unified frame-work to improve detection performance.
TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition
Considering this, we introduce an innovative PEFT method, TeamLoRA, consisting of a collaboration and competition module for experts, and thus achieving the right balance of effectiveness and efficiency: (i) For collaboration, a novel knowledge-sharing and -organizing mechanism is devised to appropriately reduce the scale of matrix operations, thereby boosting the training and inference speed.
Ranked #187 on
Visual Question Answering
on MM-Vet
Auto-Encoding Morph-Tokens for Multimodal LLM
For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge.
Sim-Grasp: Learning 6-DOF Grasp Policies for Cluttered Environments Using a Synthetic Benchmark
In this paper, we present Sim-Grasp, a robust 6-DOF two-finger grasping system that integrates advanced language models for enhanced object manipulation in cluttered environments.
WorldGPT: Empowering LLM as Multimodal World Model
As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios.
LASER: Tuning-Free LLM-Driven Attention Control for Efficient Text-conditioned Image-to-Animation
Revolutionary advancements in text-to-image models have unlocked new dimensions for sophisticated content creation, e. g., text-conditioned image editing, allowing us to edit the diverse images that convey highly complex visual concepts according to the textual guidance.
Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales
Their ability to execute intricate compositional reasoning tasks is also constrained, culminating in a stagnation of learning progression for these models.
HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models
Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks.
Ranked #189 on
Visual Question Answering
on MM-Vet
Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning
Large Language Models (LLMs) demonstrate remarkable proficiency in comprehending and handling text-based tasks.
DIEM: Decomposition-Integration Enhancing Multimodal Insights
On MM-Vet our method achieves an improvement in MM-Vet scores increasing from 31. 1 to 32. 4.
Learning Coupled Dictionaries from Unpaired Data for Image Super-Resolution
The difficulty of acquiring high-resolution (HR) and low-resolution (LR) image pairs in real scenarios limits the performance of existing learning-based image super-resolution (SR) methods in the real world.
HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data
Multi-modal Large Language Models (MLLMs) tuned on machine-generated instruction-following data have demonstrated remarkable performance in various multi-modal understanding and generation tasks.
Revisiting the Domain Shift and Sample Uncertainty in Multi-source Active Domain Transfer
Active Domain Adaptation (ADA) aims to maximally boost model adaptation in a new target domain by actively selecting a limited number of target data to annotate. This setting neglects the more practical scenario where training data are collected from multiple sources.
De-fine: Decomposing and Refining Visual Programs with Auto-Feedback
Visual programming, a modular and generalizable paradigm, integrates different modules and Python operators to solve various vision-language tasks.
Towards Complex-query Referring Image Segmentation: A Novel Benchmark
Referring Image Understanding (RIS) has been extensively studied over the past decade, leading to the development of advanced algorithms.
Survey on Deep Face Restoration: From Non-blind to Blind and Beyond
We also discuss notable benchmarks commonly utilized in the field.
I3: Intent-Introspective Retrieval Conditioned on Instructions
Recent studies indicate that dense retrieval models struggle to perform well on a wide variety of retrieval tasks that lack dedicated training data, as different retrieval tasks often entail distinct search intents.
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task.
Multi-Scale Prototypical Transformer for Whole Slide Image Classification
The PT is developed to reduce redundant instances in bags by integrating prototypical learning into the Transformer architecture.
Weakly Supervised Lesion Detection and Diagnosis for Breast Cancers with Partially Annotated Ultrasound Images
To address this issue, a novel Two-Stage Detection and Diagnosis Network (TSDDNet) is proposed based on weakly supervised learning to enhance diagnostic accuracy of the ultrasound-based CAD for breast cancers.
Multi-scale Efficient Graph-Transformer for Whole Slide Image Classification
The key idea of MEGT is to adopt two independent Efficient Graph-based Transformer (EGT) branches to process the low-resolution and high-resolution patch embeddings (i. e., tokens in a Transformer) of WSIs, respectively, and then fuse these tokens via a multi-scale feature fusion module (MFFM).
Sim-Suction: Learning a Suction Grasp Policy for Cluttered Environments Using a Synthetic Benchmark
This paper presents Sim-Suction, a robust object-aware suction grasp policy for mobile manipulation platforms with dynamic camera viewpoints, designed to pick up unknown objects from cluttered environments.
Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Document
GOSE initiates by generating preliminary relation predictions on entity pairs extracted from a scanned image of the document.
Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaboration
Recent text-to-image generation models have shown promising results in generating high-fidelity photo-realistic images.
InstructVid2Vid: Controllable Video Editing with Natural Language Instructions
We introduce InstructVid2Vid, an end-to-end diffusion-based methodology for video editing guided by human language instructions.
Sim-MEES: Modular End-Effector System Grasping Dataset for Mobile Manipulators in Cluttered Environments
Our dataset generation process combines analytic models and dynamic simulations of the entire cluttered environment to provide accurate grasp labels.
Fast MRI Reconstruction via Edge Attention
To solve this problem, we propose a lightweight and accurate Edge Attention MRI Reconstruction Network (EAMRI) to reconstruct images with edge guidance.
EWT: Efficient Wavelet-Transformer for Single Image Denoising
Transformer-based image denoising methods have achieved encouraging results in the past year.
PFT-SSR: Parallax Fusion Transformer for Stereo Image Super-Resolution
Stereo image super-resolution aims to boost the performance of image super-resolution by exploiting the supplementary information provided by binocular systems.
Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zero-shot manner.
Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization
Though effective, prompt tuning under few-shot settings on the one hand heavily relies on a good initialization of soft prompts.
Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by learning the ``soft prompts'' to condition frozen pre-training models.
Lightweight Real-time Semantic Segmentation Network with Efficient Transformer and CNN
In the past decade, convolutional neural networks (CNNs) have shown prominence for semantic segmentation.
Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding
To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.
Are Binary Annotations Sufficient? Video Moment Retrieval via Hierarchical Uncertainty-Based Active Learning
Moreover, we treat the uncertainty score of frames in a video as a whole, and estimate the difficulty of each video, which can further relieve the burden of video selection.
Efficient Image Super-Resolution with Feature Interaction Weighted Hybrid Network
Lightweight image super-resolution aims to reconstruct high-resolution images from low-resolution images using low computational costs.
DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention
Furthermore, we show that the hidden state dimension can be approximated by extending the Johnson-Lindenstrauss lemma, optimizing the attention in bilinear form.
Fine-Grained Semantically Aligned Vision-Language Pre-Training
Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks.
Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos
In this paper, we introduce a new task, named Temporal Emotion Localization in videos~(TEL), which aims to detect human emotions and localize their corresponding temporal boundaries in untrimmed videos with aligned subtitles.
Quantized Consensus under Data-Rate Constraints and DoS Attacks: A Zooming-In and Holding Approach
This paper is concerned with the quantized consensus problem for uncertain nonlinear multi-agent systems under data-rate constraints and Denial-of-Service (DoS) attacks.
Masked Autoencoders that Listen
Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers.
Ranked #4 on
Speaker Identification
on VoxCeleb1
(using extra training data)
Snow Mask Guided Adaptive Residual Network for Image Snow Removal
Recently, some methods have been proposed for snow removing, and most methods deal with snow images directly as the optimization object.
BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval
In this scenario, the input image serves as an intuitive context and background for the search, while the corresponding language expressly requests new traits on how specific characteristics of the query image should be modified in order to get the intended target image.
Cross-receptive Focused Inference Network for Lightweight Image Super-Resolution
Recently, Transformer-based methods have shown impressive performance in single image super-resolution (SISR) tasks due to the ability of global feature extraction.
Multiple Degradation and Reconstruction Network for Single Image Denoising via Knowledge Distillation
Single image denoising (SID) has achieved significant breakthroughs with the development of deep learning.
Lightweight Bimodal Network for Single-Image Super-Resolution via Symmetric CNN and Recursive Transformer
Single-image super-resolution (SISR) has achieved significant breakthroughs with the development of deep learning.
NTIRE 2022 Challenge on Stereo Image Super-Resolution: Methods and Results
In this paper, we summarize the 1st NTIRE challenge on stereo image super-resolution (restoration of rich details in a pair of low-resolution stereo images) with a focus on new solutions and results.
CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution
Then, we design an efficient Feature Refinement Module (FRM) to enhance the encoded features.
Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning
To systematically measure the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.
Feature Distillation Interaction Weighting Network for Lightweight Image Super-Resolution
Convolutional neural networks based single-image super-resolution (SISR) has made great progress in recent years.
MAGIC: Multimodal relAtional Graph adversarIal inferenCe for Diverse and Unpaired Text-based Image Captioning
We propose the Multimodal relAtional Graph adversarIal inferenCe (MAGIC) framework for diverse and unpaired TextCap.
A Systematic Survey of Deep Learning-based Single-Image Super-Resolution
This is an exhaustive survey of SISR, which can help researchers better understand SISR and inspire more exciting research in this field.
FBSNet: A Fast Bilateral Symmetrical Network for Real-Time Semantic Segmentation
Specifically, FBSNet employs a symmetrical encoder-decoder structure with two branches, semantic information branch and spatial detail branch.
Transformer for Single Image Super-Resolution
LTB is composed of a series of Efficient Transformers (ET), which occupies a small GPU memory occupation, thanks to the specially designed Efficient Multi-Head Attention (EMHA).
Structure-Preserving Deraining with Residue Channel Prior Guidance
Although these methods can remove part of the rain streaks, it is difficult for them to adapt to real-world scenarios and restore high-quality rain-free images with clear and accurate structures.
Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inference
Secondly, we introduce semantic coherence learning to explicitly encourage the semantic coherence of the adaptive hierarchical graph network from three hierarchies.
Feedback Network for Mutually Boosted Stereo Image Super-Resolution and Disparity Estimation
Besides the cross-view information exploitation in the low-resolution (LR) space, HR representations produced by the SR process are utilized to perform HR disparity estimation with higher accuracy, through which the HR features can be aggregated to generate a finer SR result.
Lightweight Image Super-Resolution with Multi-scale Feature Interaction Network
Recently, the single image super-resolution (SISR) approaches with deep and complex convolutional neural network structures have achieved promising performance.
Efficient and Accurate Multi-scale Topological Network for Single Image Dehazing
To achieve this, we propose a Multi-scale Topological Network (MSTN) to fully explore the features at different scales.
Scale-Aware Network with Regional and Semantic Attentions for Crowd Counting under Cluttered Background
Although the emergence of deep learning has greatly promoted the development of this field, crowd counting under cluttered background is still a serious challenge.
Robust Meta-learning with Noise via Eigen-Reptile
In particular, we first cast the meta-overfitting problem (overfitting on sampling and label noise) as a gradient noise problem since few available samples cause meta-learner to overfit on existing examples (clean or corrupted) of an individual task at every gradient step.
Revisiting Factorizing Aggregated Posterior in Learning Disentangled Representations
In this paper, we provide a theoretical explanation that low total correlation of sampled representation cannot guarantee low total correlation of the mean representation.
MDCN: Multi-scale Dense Cross Network for Image Super-Resolution
Among them, MDCB aims to detect multi-scale features and maximize the use of image features flow at different scales, HFDB focuses on adaptively recalibrate channel-wise feature responses to achieve feature distillation, and DRB attempts to reconstruct SR images with different upsampling factors in a single model.
Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling
In this paper, we focus on enhancing the generalization ability of the VIST model by considering the few-shot setting.
Disentangle Perceptual Learning through Online Contrastive Learning
Perceptual learning approaches like perceptual loss are empirically powerful for such tasks but they usually rely on the pre-trained classification network to provide features, which are not necessarily optimal in terms of visual perception of image transformation.
Universal Phone Recognition with a Multilingual Allophone System
Multilingual models can improve language processing, particularly for low resource situations, by sharing parameters across languages.
Towards Zero-shot Learning for Automatic Phonemic Transcription
The difficulty of this task is that phoneme inventories often differ between the training languages and the target language, making it infeasible to recognize unseen phonemes.
Fast Loop Closure Detection via Binary Content
Our method is compared with the state-of-the-art loop closure detection methods and the results show that it outperforms the traditional methods at both recall rate and speed.
HighEr-Resolution Network for Image Demosaicing and Enhancing
However, plenty of studies have shown that global information is crucial for image restoration tasks like image demosaicing and enhancing.
Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
Visual navigation is a task of training an embodied agent by intelligently navigating to a target object (e. g., television) using only visual observations.
Walking with MIND: Mental Imagery eNhanceD Embodied QA
The experimental results and further analysis prove that the agent with the MIND module is superior to its counterparts not only in EQA performance but in many other aspects such as route planning, behavioral interpretation, and the ability to generalize from a few examples.
Adversarial camera stickers: A physical camera-based attack on deep learning systems
In this work, we consider an alternative question: is it possible to fool deep classifiers, over all perceived objects of a certain type, by physically manipulating the camera itself?
A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling
This paper compares five types of pooling functions both theoretically and experimentally, with special focus on their performance of localization.
Sound Audio and Speech Processing
An Effective Single-Image Super-Resolution Model Using Squeeze-and-Excitation Networks
Recent works on single-image super-resolution are concentrated on improving performance through enhancing spatial encoding between convolutional layers.
Multi-scale Residual Network for Image Super-Resolution
Meanwhile, we let these features interact with each other to get the most efficacious image information, we call this structure Multi-scale Residual Block (MSRB).
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval
Constructing a joint representation invariant across different modalities (e. g., video, language) is of significant importance in many multimedia applications.
Ranked #37 on
Video Retrieval
on MSR-VTT
A Comparison of deep learning methods for environmental sound
On these features, we apply five models: Gaussian Mixture Model (GMM), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Convolutional Deep Neural Net- work (CNN) and i-vector.
Learning Filter Banks Using Deep Learning For Acoustic Signals
Based on the procedure of log Mel-filter banks, we design a filter bank learning layer.
Very Deep Convolutional Neural Networks for Raw Waveforms
Our CNNs, with up to 34 weight layers, are efficient to optimize over very long sequences (e. g., vector of size 32000), necessary for processing acoustic waveforms.
