Search Results for author:
Found 87 papers, 48 papers with code
SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity
Building on SDHA, we further analyze various spike-driven space-time attention designs and identify an optimal scheme that delivers appealing performance for video tasks, while maintaining only linear temporal complexity.
Learning Fine-grained Domain Generalization via Hyperbolic State Space Hallucination
Then, the pre- and post- style hallucinate state embeddings are projected into the hyperbolic manifold.
DGFamba: Learning Flow Factorized State Space for Visual Domain Generalization
By aligning these probability paths in the latent space, the state embeddings are able to represent the same content distribution regardless of the style differences.
Anatomically and Metabolically Informed Diffusion for Unified Denoising and Segmentation in Low-Count PET Imaging
In this work, we present the anatomically and metabolically informed diffusion (AMDiff) model, a unified framework for denoising and lesion/organ segmentation in low-count PET imaging.
Step-Video-TI2V Technical Report: A State-of-the-Art Text-Driven Image-to-Video Generation Model
We present Step-Video-TI2V, a state-of-the-art text-driven image-to-video generation model with 30B parameters, capable of generating videos up to 102 frames based on both text and image inputs.
TAIL: Text-Audio Incremental Learning
Many studies combine text and audio to capture multi-modal information but they overlook the model's generalization ability on new datasets.
Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction
Based on our new StepEval-Audio-360 evaluation benchmark, Step-Audio achieves state-of-the-art performance in human evaluations, especially in terms of instruction following.
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
We present Step-Video-T2V, a state-of-the-art text-to-video pre-trained model with 30B parameters and the ability to generate videos up to 204 frames in length.
WisdomBot: Tuning Large Language Models with Artificial Intelligence Knowledge
Large language models (LLMs) have emerged as powerful tools in natural language processing (NLP), showing a promising future of artificial generated intelligence (AGI).
Universal Fine-grained Visual Categorization by Concept Guided Learning
The discriminative concepts is utilized to guide the fine-grained representation learning.
Ranked #8 on
Fine-Grained Image Classification
on NABirds
Fine-Grained Image Classification
Fine-Grained Visual Categorization
+4
A Generalizable 3D Diffusion Framework for Low-Dose and Few-View Cardiac SPECT
This work introduced DiffSPECT-3D, a diffusion framework for 3D cardiac SPECT imaging that effectively adapts to different acquisition settings without requiring further network re-training or fine-tuning.
Discretized Gaussian Representation for Tomographic Reconstruction
To further enhance computational efficiency, we introduce a Fast Volume Reconstruction technique that aggregates the contributions of these Gaussians into a discretized volume in a highly parallelized fashion.
Towards Small Object Editing: A Benchmark Dataset and A Training-Free Approach
A plethora of text-guided image editing methods has recently been developed by leveraging the impressive capabilities of large-scale diffusion-based generative models especially Stable Diffusion.
Grounding is All You Need? Dual Temporal Grounding for Video Dialog
In the realm of video dialog response generation, the understanding of video content and the temporal nuances of conversation history are paramount.
Personalized Knowledge Tracing through Student Representation Reconstruction and Class Imbalance Mitigation
Extensive experimental results on four publicly available educational datasets demonstrate the advanced predictive performance of PKT in comparison with 16 state-of-the-art models.
Semantic Alignment for Multimodal Large Language Models
By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis and align the semantics of different images before feeding them into LLM.
Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation
In this paper, we present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier.
DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Driving
Such datasets are expected to cover various driving scenarios with adverse weather, lighting conditions and diverse moving objects.
Described Spatial-Temporal Video Detection
However, in the video domain, the existing setting, i. e., spatial-temporal video grounding (STVG), is formulated to only detect one pre-existing object in each frame, ignoring the fact that language descriptions can involve none or multiple entities within a video.
Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaboration
As these systems grapple with shifting data distributions between the cloud and devices, the traditional approach of fine-tuning-based adaptation (FTA) exists the following issues: the costly and time-consuming data annotation required by FTA and the looming risk of model overfitting.
Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition
Existing research of video understanding still struggles to achieve in-depth comprehension and reasoning in complex videos, primarily due to the under-exploration of two key bottlenecks: fine-grained spatial-temporal perceptive understanding and cognitive-level video scene comprehension.
Dose-aware Diffusion Model for 3D Low-dose PET: Multi-institutional Validation with Reader Study and Real Low-dose Data
However, existing models have often resulted in compromised image quality when achieving low-dose PET and have limited generalizability to different image noise-levels, acquisition protocols, and patient populations.
Spider: A Unified Framework for Context-dependent Concept Segmentation
Different from the context-independent (CI) concepts such as human, car, and airplane, context-dependent (CD) concepts require higher visual understanding ability, such as camouflaged object and medical lesion.
GOOD: Towards Domain Generalized Orientated Object Detection
In this paper, we propose the task of domain generalized oriented object detection, which intends to explore the generalization of oriented object detectors on arbitrary unseen target domains.
Cross-Level Multi-Instance Distillation for Self-Supervised Fine-Grained Visual Categorization
To comprehensively learn the relation between informative patches and fine-grained semantics, the multi-instance knowledge distillation is implemented on both the region/image crop pairs from the teacher and student net, and the region-image crops inside the teacher / student net, which we term as intra-level multi-instance distillation and inter-level multi-instance distillation.
Fine-Grained Visual Categorization
Knowledge Distillation
+2
Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching
Navigating drones through natural language commands remains challenging due to the dearth of accessible multi-modal datasets and the stringent precision requirements for aligning visual and textual data.
De-fine: Decomposing and Refining Visual Programs with Auto-Feedback
Visual programming, a modular and generalizable paradigm, integrates different modules and Python operators to solve various vision-language tasks.
NExT-Chat: An LMM for Chat, Detection and Segmentation
The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs).
Towards Robust Multi-Modal Reasoning via Model Selection
The reasoning capabilities of LLM (Large Language Model) are widely acknowledged in recent research, inspiring studies on tool learning and autonomous agents.
Domain-wise Invariant Learning for Panoptic Scene Graph Generation
Panoptic Scene Graph Generation (PSG) involves the detection of objects and the prediction of their corresponding relationships (predicates).
Towards Complex-query Referring Image Segmentation: A Novel Benchmark
Referring Image Understanding (RIS) has been extensively studied over the past decade, leading to the development of advanced algorithms.
NExT-GPT: Any-to-Any Multimodal LLM
While recently Multimodal Large Language Models (MM-LLMs) have made exciting strides, they mostly fall prey to the limitation of only input-side multimodal understanding, without the ability to produce content in multiple modalities.
Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs
In this work, we investigate strengthening the awareness of video dynamics for DMs, for high-quality T2V generation.
Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape
Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model.
Ranked #1 on
Animal Pose Estimation
on Animal3D
I3: Intent-Introspective Retrieval Conditioned on Instructions
Recent studies indicate that dense retrieval models struggle to perform well on a wide variety of retrieval tasks that lack dedicated training data, as different retrieval tasks often entail distinct search intents.
Online Distillation-enhanced Multi-modal Transformer for Sequential Recommendation
To be specific, we first introduce an ID-aware Multi-modal Transformer module in the item representation learning stage to facilitate information interaction among different features.
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task.
Panoptic Scene Graph Generation with Semantics-Prototype Learning
To promise consistency and accuracy during the transfer process, we propose to measure the invariance of representations in each predicate class, and learn unbiased prototypes of predicates with different intensities.
Ranked #3 on
Panoptic Scene Graph Generation
on PSG Dataset
In Defense of Clip-based Video Relation Detection
While recent video-based methods utilizing video tubelets have shown promising results, we argue that the effective modeling of spatial and temporal context plays a more significant role than the choice between clip tubelets and video tubelets.
Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment
Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during transfer.
Generating Visual Spatial Description via Holistic 3D Scene Understanding
With an external 3D scene extractor, we obtain the 3D objects and scene features for input images, based on which we construct a target object-centered 3D spatial scene graph (Go3D-S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes.
VPGTrans: Transfer Visual Prompt Generator across LLMs
While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm.
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation.
Ranked #2 on
Medical Image Segmentation
on Synapse multi-organ CT
Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications
Recently, Meta AI Research approaches a general, promptable Segment Anything Model (SAM) pre-trained on an unprecedentedly large segmentation dataset (SA-1B).
Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zero-shot manner.
Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer
Event camera, as an asynchronous vision sensor capturing scene dynamics, presents new opportunities for highly efficient 3D human pose tracking.
Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by learning the ``soft prompts'' to condition frozen pre-training models.
Scalable Attribution of Adversarial Attacks via Multi-Task Learning
In this paper, AAP is defined as the recognition of three signatures, i. e., {\em attack algorithm}, {\em victim model} and {\em hyperparameter}.
MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer
To effectively integrate these two cutting-edge techniques for the Medical image segmentation, we propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2.
Ranked #3 on
Medical Image Segmentation
on Synapse multi-organ CT
Are Binary Annotations Sufficient? Video Moment Retrieval via Hierarchical Uncertainty-Based Active Learning
Moreover, we treat the uncertainty score of frames in a video as a whole, and estimate the difficulty of each video, which can further relieve the burden of video selection.
Multispectral Video Semantic Segmentation: A Benchmark Dataset and Baseline
This gives rise to significantly more robust segmentation of image objects in complex scenes and under adverse conditions.
WINNER: Weakly-Supervised hIerarchical decompositioN and aligNment for Spatio-tEmporal Video gRounding
WINNER first builds the language decomposition tree in a bottom-up manner, upon which the structural attention mechanism and top-down feature backtracking jointly build a multi-modal decomposition tree, permitting a hierarchical understanding of unstructured videos.
MRTNet: Multi-Resolution Temporal Network for Video Sentence Grounding
In this work, we propose a novel multi-resolution temporal video sentence grounding network: MRTNet, which consists of a multi-modal feature encoder, a Multi-Resolution Temporal (MRT) module, and a predictor module.
Multi-queue Momentum Contrast for Microvideo-Product Retrieval
The booming development and huge market of micro-videos bring new e-commerce channels for merchants.
Driving Style Recognition at First Impression for Online Trajectory Prediction
This paper proposes a new driving style recognition approach that allows autonomous vehicles (AVs) to perform trajectory predictions for surrounding vehicles with minimal data.
Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization
The key idea underpinning the proposed method is to integrate fine- and coarse-grained retrieval as matching data points with small and large fluctuations, respectively.
Composed Image Retrieval (CoIR)
Image Retrieval with Multi-Modal Query
+1
MetaComp: Learning to Adapt for Online Depth Completion
However, facing a new environment where the test data occurs online and differs from the training data in the RGB image content and depth sparsity, the trained model might suffer severe performance drop.
Structured and Natural Responses Co-generation for Conversational Search
Existing approaches either 1) predict structured dialog acts first and then generate natural response; or 2) map conversation context to natural responses directly in an end-to-end manner.
Invariant Grounding for Video Question Answering
At its core is understanding the alignments between visual scenes in video and linguistic semantics in question to yield the answer.
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs.
Ranked #1 on
Visual Commonsense Reasoning
on VCR (Q-AR) test
Promoting Saliency From Depth: Deep Unsupervised RGB-D Saliency Detection
The laborious and time-consuming manual annotation has become a real bottleneck in various practical scenarios.
3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective
This research aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image.
Ranked #1 on
Single-View 3D Reconstruction
on CUB-200-2011
Fine-Grained Scene Graph Generation with Data Transfer
Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images.
Ranked #1 on
Unbiased Scene Graph Generation
on Visual Genome
Video Question Answering: Datasets, Algorithms and Challenges
Video Question Answering (VideoQA) aims to answer natural language questions according to the given videos.
Content-Variant Reference Image Quality Assessment via Knowledge Distillation
The comparisons of distribution differences between HQ and LQ images can help our model better assess the image quality.
Generating Diverse and Natural 3D Human Motions From Text
Automated generation of 3D human motions from text is a challenging problem.
Ranked #3 on
Motion Synthesis
on Inter-X
Exploring Denoised Cross-Video Contrast for Weakly-Supervised Temporal Action Localization
Inspired by recent success in unsupervised contrastive representation learning, we propose a novel denoised cross-video contrastive algorithm, aiming to enhance the feature discrimination ability of video snippets for accurate temporal action localization in the weakly-supervised setting.
Video as Conditional Graph Hierarchy for Multi-Granular Question Answering
To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues.
Ranked #6 on
Video Question Answering
on IntentQA
Rethinking the Two-Stage Framework for Grounded Situation Recognition
Since each verb is associated with a specific set of semantic roles, all existing GSR methods resort to a two-stage framework: predicting the verb in the first stage and detecting the semantic roles in the second stage.
Ranked #4 on
Situation Recognition
on imSitu
Joint Semantic Mining for Weakly Supervised RGB-D Salient Object Detection
As a by-product, a CapS dataset is constructed by augmenting existing benchmark training set with additional image tags and captions.
Meeting Summarization with Pre-training and Clustering Methods
Lastly, we compare the performance of our baseline models with BART, a state-of-the-art language model that is effective for summarization.
PP-PicoDet: A Better Real-Time Object Detector on Mobile Devices
We investigate the applicability of the anchor-free strategy on lightweight object detection models.
Ranked #1 on
Object Detection
on MSCOCO
Decoupling Strategy and Surface Realization for Task-oriented Dialogues
The core is to construct a latent content space for strategy optimization and disentangle the surface style from it.
Advancing biological super-resolution microscopy through deep learning: a brief review
We conclude with an outlook on how deep learning could shape the future of this new generation of light microscopy technology.
Calibrated RGB-D Salient Object Detection
Complex backgrounds and similar appearances between objects and their surroundings are generally recognized as challenging scenarios in Salient Object Detection (SOD).
Ranked #3 on
Object Detection
on PKU-DDD17-Car
Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
To our knowledge, our work is the first in producing calibrated predictions under different expertise levels for medical image segmentation.
Deconfounded Video Moment Retrieval with Causal Intervention
To fill the research gap, we propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction.
Deep Learning for Weakly-Supervised Object Detection and Object Localization: A Survey
With the success of deep neural networks in object detection, both WSOD and WSOL have received unprecedented attention.
Conditional Hyper-Network for Blind Super-Resolution with Multiple Degradations
We extract degradation prior at task-level with the proposed ConditionNet, which will be used to adapt the parameters of the basic SR network (BaseNet).
Boundary Proposal Network for Two-Stage Natural Language Video Localization
State-of-the-art NLVL methods are almost in one-stage fashion, which can be typically grouped into two categories: 1) anchor-based approach: it first pre-defines a series of video segment candidates (e. g., by sliding window), and then does classification for each candidate; 2) anchor-free approach: it directly predicts the probabilities for each video frame as a boundary or intermediate frame inside the positive segment.
Dynamic Context-Sensitive Filtering Network for Video Salient Object Detection
Our bidirectional dynamic fusion strategy encourages the interaction of spatial and temporal information in a dynamic manner.
Ranked #15 on
Video Polyp Segmentation
on SUN-SEG-Easy (Unseen)
ChemistryQA: A Complex Question Answering Dataset from Chemistry
We notice that some real-world QA tasks are more complex, which cannot be solved by end-to-end neural networks or translated to any kind of formal representations.
Table2Charts: Recommending Charts by Learning Shared Table Representations
It is common for people to create different types of charts to explore a multi-dimensional dataset (table).
Accurate RGB-D Salient Object Detection via Collaborative Learning
The explicitly extracted edge information goes together with saliency to give more emphasis to the salient regions and object boundaries.
Ranked #20 on
RGB-D Salient Object Detection
on NJU2K
An Early Study on Intelligent Analysis of Speech under COVID-19: Severity, Sleep Quality, Fatigue, and Anxiety
In particular, by analysing speech recordings from these patients, we construct audio-only-based models to automatically categorise the health state of patients from four aspects, including the severity of illness, sleep quality, fatigue, and anxiety.
Context-Aware Deep Spatio-Temporal Network for Hand Pose Estimation from Depth Images
As a fundamental and challenging problem in computer vision, hand pose estimation aims to estimate the hand joint locations from depth images.
