Search Results for author:
Found 39 papers, 15 papers with code
DOGE: Towards Versatile Visual Document Grounding and Referring
In recent years, Multimodal Large Language Models (MLLMs) have increasingly emphasized grounding and referring capabilities to achieve detailed understanding and flexible user interaction.
mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
Thus, multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge, effectively expanding the knowledge scope.
Taming Rectified Flow for Inversion and Editing
To address this issue, we propose RF-Solver, a novel training-free sampler that effectively enhances inversion precision by mitigating the errors in the ODE-solving process of rectified flow.
E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
Bench (Event-Level & Time-Sensitive Video Understanding Benchmark), a large-scale and high-quality benchmark for open-ended event-level video understanding.
CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities
However, since it is only trained on static images, the fine-tuning process of subject learning disrupts abilities of video diffusion models (VDMs) to combine concepts and generate motions.
SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses
Based on the dataset, we further introduce a more complex setting of video grounding dubbed Multi-Paragraph Video Grounding (MPVG), which takes as input multiple paragraphs and a long video for grounding each paragraph query to its temporal interval.
EA-VTR: Event-Aware Video-Text Retrieval
EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment, ultimately enhancing the comprehensive understanding of video events.
How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?
Dominant dual-encoder models enable efficient image-text retrieval but suffer from limited accuracy while the cross-encoder models offer higher accuracy at the expense of efficiency.
PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM
This system integrates our proposed layout generation method as the core component, demonstrating its effectiveness in practical scenarios.
SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model
Controllable spherical panoramic image generation holds substantial applicative potential across a variety of domains. However, it remains a challenging task due to the inherent spherical distortion and geometry characteristics, resulting in low-quality content generation. In this paper, we introduce a novel framework of SphereDiffusion to address these unique challenges, for better generating high-quality and precisely controllable spherical panoramic images. For the spherical distortion characteristic, we embed the semantics of the distorted object with text encoding, then explicitly construct the relationship with text-object correspondence to better use the pre-trained knowledge of the planar images. Meanwhile, we employ a deformable technique to mitigate the semantic deviation in latent space caused by spherical distortion. For the spherical geometry characteristic, in virtue of spherical rotation invariance, we improve the data diversity and optimization objectives in the training process, enabling the model to better learn the spherical geometry characteristic. Furthermore, we enhance the denoising process of the diffusion model, enabling it to effectively use the learned geometric characteristic to ensure the boundary continuity of the generated images. With these specific techniques, experiments on Structured3D dataset show that SphereDiffusion significantly improves the quality of controllable spherical image generation and relatively reduces around 35% FID on average.
RecDCL: Dual Contrastive Learning for Recommendation
In this work, we investigate how to employ both batch-wise CL (BCL) and feature-wise CL (FCL) for recommendation.
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Recent advances in text-to-image generation have made remarkable progress in synthesizing realistic human photos conditioned on given text prompts.
Ranked #6 on
Diffusion Personalization Tuning Free
on AgeDB
Diffusion Personalization Tuning Free
Text-to-Image Generation
CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models
As a result, our CustomNet ensures enhanced identity preservation and generates diverse, harmonious outputs.
StyleAdapter: A Unified Stylized Image Generation Model
In this work, we propose StyleAdapter, a unified stylized image generation model capable of producing a variety of stylized images that match both the content of a given prompt and the style of reference images, without the need for per-style fine-tuning.
Towards Unseen Triples: Effective Text-Image-joint Learning for Scene Graph Generation
In this work, we propose a Text-Image-joint Scene Graph Generation (TISGG) model to resolve the unseen triples and improve the generalisation capability of the SGG models.
Sticker820K: Empowering Interactive Retrieval with Stickers
Therefore, we propose StickerCLIP as a benchmark model on the Sticker820K dataset.
SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation
Experimental results on Stanford2D3D Panoramic datasets show that SGAT4PASS significantly improves performance and robustness, with approximately a 2% increase in mIoU, and when small 3D disturbances occur in the data, the stability of our performance is improved by an order of magnitude.
Ranked #4 on
Semantic Segmentation
on Stanford2D3D Panoramic
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
Despite the success in large-scale text-to-image generation and text-conditioned image editing, existing methods still struggle to produce consistent generation and editing results.
Ranked #11 on
Text-based Image Editing
on PIE-Bench
LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation
To overcome the difficult multimodal fusion of image and layout, we propose to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form.
Ranked #1 on
Layout-to-Image Generation
on Visual Genome 128x128
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
In this paper, we aim to ``dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly.
Tagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrieval
Vision-language alignment learning for video-text retrieval arouses a lot of attention in recent years.
ViLEM: Visual-Language Error Modeling for Image-Text Retrieval
ViLEM then enforces the model to discriminate the correctness of each word in the plausible negative texts and further correct the wrong words via resorting to image information.
Ranked #45 on
Visual Reasoning
on Winoground
Order-Prompted Tag Sequence Generation for Video Tagging
This paper proposes a novel generative model, Order-Prompted Tag Sequence Generation (OP-TSG), according to the above characteristics.
Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning
Accordingly, we first exclude these surely non-existent categories by a complementary learning loss.
Do we really need temporal convolutions in action segmentation?
Most state-of-the-art methods focus on designing temporal convolution-based models, but the inflexibility of temporal convolutions and the difficulties in modeling long-term temporal dependencies restrict the potential of these models.
Ranked #1 on
Action Segmentation
on 50Salads
Accelerating the Training of Video Super-Resolution Models
In this work, we show that it is possible to gradually train video models from small to large spatial/temporal sizes, i. e., in an easy-to-hard manner.
CREATE: A Benchmark for Chinese Short Video Retrieval and Title Generation
In this paper, we propose to CREATE, the first large-scale Chinese shoRt vidEo retrievAl and Title gEneration benchmark, to facilitate research and application in video titling and video retrieval in Chinese.
BTS: A Bi-Lingual Benchmark for Text Segmentation in the Wild
Current researches mainly focus on only English characters and digits, while few work studies Chinese characters due to the lack of public large-scale and high-quality Chinese datasets, which limits the practical application scenarios of text segmentation.
From Heatmaps to Structural Explanations of Image Classifiers
This paper summarizes our endeavors in the past few years in terms of explaining image classifiers, with the aim of including negative results and insights we have gained.
Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution
Unlike previous integral gradient methods, our FAIG aims at finding the most discriminative filters instead of input pixels/features for degradation removal in blind SR networks.
Stochastic Block-ADMM for Training Deep Networks
We prove the convergence of our proposed method and justify its capabilities through experiments in supervised and weakly-supervised settings.
A Generic Object Re-identification System for Short Videos
Short video applications like TikTok and Kwai have been a great hit recently.
Visualizing Point Cloud Classifiers by Curvature Smoothing
In this paper, we propose a novel approach to visualize features important to the point cloud classifiers.
Visualizing Deep Networks by Optimizing with Integrated Gradients
Understanding and interpreting the decisions made by deep learning models is valuable in many domains.
Interactive Naming for Explaining Deep Neural Networks: A Formative Study
For this purpose, we developed a user interface for "interactive naming," which allows a human annotator to manually cluster significant activation maps in a test set into meaningful groups called "visual concepts".
PointConv: Deep Convolutional Networks on 3D Point Clouds
Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure.
Ranked #2 on
3D Part Segmentation
on IntrA
Deep Air Learning: Interpolation, Prediction, and Feature Analysis of Fine-grained Air Quality
The interpolation, prediction, and feature analysis of fine-gained air quality are three important topics in the area of urban air computing.
Embedding Deep Networks into Visual Explanations
The XNN works by learning a nonlinear embedding of a high-dimensional activation vector of a deep network layer into a low-dimensional explanation space while retaining faithfulness i. e., the original deep learning predictions can be constructed from the few concepts extracted by our explanation network.
