Search Results for author:
Found 88 papers, 72 papers with code
Optimizing 4D Lookup Table for Low-light Video Enhancement via Wavelet Priori
Based on this, we propose incorporating Wavelet-priori for 4D Lookup Table (WaveLUT), which effectively enhances the color coherence between video frames and the accuracy of color mapping while maintaining low latency.
Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
Additionally, we explore the relationships among the models, attack strategies, and types of harmful content, as well as the correlations between the evaluation metrics, which proves the validity of our multifaceted evaluation framework.
ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
We benchmark ChemVLM against a range of open-source and proprietary multimodal large language models on various tasks.
MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3. 5, and manual correction.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
This long-context capability allows IXC-2. 5 to excel in tasks requiring extensive input and output contexts.
Ranked #25 on
Visual Question Answering
on MM-Vet
Iterative or Innovative? A Problem-Oriented Perspective for Code Optimization
However, LLMs have rarely been explored for code optimization.
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
It not only allows flexible transmission of task information and gradient feedback between the MLLM and multiple downstream decoders but also effectively resolves training conflicts in multi-tasking scenarios.
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
In this paper, we introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset.
Needle In A Multimodal Haystack
In this work, we present Needle In A Multimodal Haystack (MM-NIAH), the first benchmark specifically designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.
Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data.
LLMs Meet Multimodal Generation and Editing: A Survey
With the recent advancement in large language models (LLMs), there is a growing interest in combining LLMs with multimodal learning.
Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting
Large Language Models (LLMs) have exhibited remarkable proficiency in generating code.
S-Eval: Automatic and Adaptive Test Generation for Benchmarking Safety Evaluation of Large Language Models
Large Language Models have gained considerable attention for their revolutionary capabilities.
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
Compared to both open-source and proprietary models, InternVL 1. 5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks.
Ranked #10 on
Visual Question Answering
on MM-Vet v2
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution.
Ranked #19 on
Visual Question Answering
on MM-Vet
Does Knowledge Graph Really Matter for Recommender Systems?
We consider the scenarios where knowledge in a KG gets completely removed, randomly distorted and decreased, and also where recommendations are for cold-start users.
Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments
However, in the real-world where test ground truths are not provided, it is non-trivial to find out whether bounding boxes are accurate, thus preventing us from assessing the detector generalization ability.
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures
Our evaluations demonstrate that VRWKV surpasses ViT's performance in image classification and has significantly faster speeds and lower memory usage processing high-resolution inputs.
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
In addition, we design a new benchmark, termed Circular-based Relation Probing Evaluation (CRPE) for comprehensively evaluating the relation comprehension capabilities of MLLMs.
Ranked #49 on
Visual Question Answering
on MM-Vet
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI.
Ranked #112 on
Visual Question Answering
on MM-Vet
FoolSDEdit: Deceptively Steering Your Edits Towards Targeted Attribute-aware Distribution
To expose this potential vulnerability, we aim to build an adversarial attack forcing SDEdit to generate a specific data distribution aligned with a specified attribute (e. g., female), without changing the input's attribute characteristics.
MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer
Developing generative models for interleaved image-text data has both research and practical value.
Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications
The advancements in speed and efficiency of DCNv4, combined with its robust performance across diverse vision tasks, show its potential as a foundational building block for future vision models.
Low-light Image Enhancement via CLIP-Fourier Guided Wavelet Diffusion
Moreover, to further promote the effective recovery of the image details, we combine the Fourier transform based on the wavelet transform and construct a Hybrid High Frequency Perception Module (HFPM) with a significant perception of the detailed features.
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT-full
(using extra training data)
A Survey of Reasoning with Foundation Models
Reasoning, a crucial ability for complex problem-solving, plays a pivotal role in various real-world settings such as negotiation, medical diagnosis, and criminal investigation.
DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD).
Exploring ChatGPT's Capabilities on Vulnerability Management
Prior works show that ChatGPT has the capabilities of processing foundational code analysis tasks, such as abstract syntax tree generation, which indicates the potential of using ChatGPT to comprehend code syntax and static behaviors.
ControlLLM: Augment Language Models with Tools by Searching on Graphs
We present ControlLLM, a novel framework that enables large language models (LLMs) to utilize multi-modal tools for solving complex real-world tasks.
Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection
Current research is primarily dedicated to advancing the accuracy of camera-only 3D object detectors (apprentice) through the knowledge transferred from LiDAR- or multi-modal-based counterparts (expert).
Ranked #6 on
3D Object Detection
on nuScenes Camera Only
CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code
Automatically generating function summaries for binaries is an extremely valuable but challenging task, since it involves translating the execution behavior and semantics of the low-level language (assembly code) into human-readable natural language.
Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models
The revolution of artificial intelligence content generation has been rapidly accelerated with the booming text-to-image (T2I) diffusion models.
FB-BEV: BEV Representation from Forward-Backward View Transformations
Currently, the two most prominent VTM paradigms are forward projection and backward projection.
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
We present the All-Seeing (AS) project: a large-scale data and model for recognizing and understanding everything in the open world.
AVSegFormer: Audio-Visual Segmentation with Transformer
In this paper, we propose AVSegFormer, a novel framework for AVS tasks that leverages the transformer architecture.
Denoising Diffusion Semantic Segmentation with Mask Prior Modeling
In this paper, we propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a recently-developed denoising diffusion generative model.
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities.
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
We hope this model can set a new baseline for generalist vision and language models.
Tram: A Token-level Retrieval-augmented Mechanism for Source Code Summarization
In this paper, we propose a fine-grained Token-level retrieval-augmented mechanism (Tram) on the decoder side rather than the encoder side to enhance the performance of neural models and produce more low-frequency tokens in generating summaries.
VideoChat: Chat-Centric Video Understanding
In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat.
Ranked #3 on
Question Answering
on NExT-QA (Open-ended VideoQA)
Video-based Generative Performance Benchmarking (Consistency)
Video-based Generative Performance Benchmarking (Contextual Understanding)
+5
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2.
A Survey of Historical Learning: Learning Models with Learning History
The various types of elements, deposited in the training history, are a large amount of wealth for improving learning deep models.
Champion Solution for the WSDM2023 Toloka VQA Challenge
In this report, we present our champion solution to the WSDM2023 Toloka Visual Question Answering (VQA) Challenge.
Planning-oriented Autonomous Driving
Oriented at this, we revisit the key components within perception and prediction, and prioritize the tasks such that all these tasks contribute to planning.
VLG: General Video Recognition with Web Textual Knowledge
Our VLG is first pre-trained on video and language datasets to learn a shared feature space, and then devises a flexible bi-modal attention head to collaborate high-level semantic concepts under different settings.
Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
In this paper, we propose Uni-Perceiver v2, which is the first generalist model capable of handling major large-scale vision and vision-language tasks with competitive performance.
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Compared to the great progress of large-scale vision transformers (ViTs) in recent years, large-scale models based on convolutional neural networks (CNNs) are still in an early state.
Ranked #1 on
Instance Segmentation
on COCO test-dev
(AP50 metric, using extra
training data)
Demystify Transformers & Convolutions in Modern Image Deep Networks
Our experiments on various tasks and an analysis of inductive bias show a significant performance boost due to advanced network-level and block-level designs, but performance differences persist among different STMs.
On Efficient Reinforcement Learning for Full-length Game of StarCraft II
In this work, we investigate a set of RL techniques for the full-length game of StarCraft II.
Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe
As sensor configurations get more complex, integrating multi-source information from different sensors and representing features in a unified view come of vital importance.
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation
Second, to resist overfitting issues caused by few training samples, a hyper-class embedding is learned by clustering all category embeddings for initialization and aligned with category embedding of the new class for enhancement, where learned knowledge assists to learn new knowledge, thus alleviating performance dependence on training data scale.
Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs
To mitigate such interference, we introduce the Conditional Mixture-of-Experts (Conditional MoEs) to generalist models.
Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality
Masked AutoEncoder (MAE) has recently led the trends of visual self-supervision area by an elegant asymmetric encoder-decoder design, which significantly optimizes both the pre-training efficiency and fine-tuning accuracy.
Ranked #37 on
Object Detection
on COCO minival
Vision Transformer Adapter for Dense Predictions
This work investigates a simple yet powerful dense prediction task adapter for Vision Transformer (ViT).
Ranked #4 on
Semantic Segmentation
on PASCAL Context
Hybrid Cloud-Edge Collaborative Data Anomaly Detection in Industrial Sensor Networks
The sensor data detection model based on Gaussian and Bayesian algorithms can detect the anomalous sensor data in real-time and upload them to the cloud for further analysis, filtering the normal sensor data and reducing traffic load.
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers
In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries.
WegFormer: Transformers for Weakly Supervised Semantic Segmentation
Although convolutional neural networks (CNNs) have achieved remarkable progress in weakly supervised semantic segmentation (WSSS), the effective receptive field of CNN is insufficient to capture global context information, leading to sub-optimal results.
VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition
Deep learning-based models encounter challenges when processing long-tailed data in the real world.
Ranked #4 on
Long-tail Learning
on iNaturalist 2018
(using extra training data)
FAST: Faster Arbitrarily-Shaped Text Detector with Minimalist Kernel Representation
We propose an accurate and efficient scene text detection framework, termed FAST (i. e., faster arbitrarily-shaped text detector).
Ranked #2 on
Scene Text Detection
on MSRA-TD500
ARTS: Eliminating Inconsistency between Text Detection and Recognition with Auto-Rectification Text Spotter
Recent approaches for end-to-end text spotting have achieved promising results.
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner.
Ranked #4 on
Panoptic Segmentation
on COCO test-dev
An empirical evaluation of attention-based multi-head models for improved turbofan engine remaining useful life prediction
The results presented in this study demonstrate the importance of multi-head models and attention mechanisms to an improved understanding of the remaining useful life of industrial assets.
Learning Class-level Prototypes for Few-shot Learning
Few-shot learning aims to recognize new categories using very few labeled samples.
Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers
Unlike existing CNN-based methods, we adopt a transformer encoder, which learns more powerful and robust representations.
Ranked #10 on
Medical Image Segmentation
on CVC-ColonDB
PVT v2: Improved Baselines with Pyramid Vision Transformer
We hope this work will facilitate state-of-the-art Transformer researches in computer vision.
Ranked #23 on
Object Detection
on COCO-O
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders.
Ranked #1 on
Semantic Segmentation
on COCO-Stuff full
PolarMask++: Enhanced Polar Representation for Single-Shot Instance Segmentation and Beyond
Extensive experiments demonstrate the effectiveness of both PolarMask and PolarMask++, which achieve competitive results on instance segmentation in the challenging COCO dataset with single-model and single-scale training and testing, as well as new state-of-the-art results on rotate text detection and cell segmentation.
Ranked #81 on
Instance Segmentation
on COCO test-dev
(using extra training data)
PAN++: Towards Efficient and Accurate End-to-End Spotting of Arbitrarily-Shaped Text
By systematically comparing with existing scene text representations, we show that our kernel representation can not only describe arbitrarily-shaped text but also well distinguish adjacent text.
An Introduction of mini-AlphaStar
StarCraft II (SC2) is a real-time strategy game in which players produce and control multiple units to fight against opponent's units.
Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization
(1) We divide input image into small patches and adopt TIN, successfully transferring image style with arbitrary high-resolution.
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Unlike the recently-proposed Transformer model (e. g., ViT) that is specially designed for image classification, we propose Pyramid Vision Transformer~(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks.
Ranked #4 on
Semantic Segmentation
on SynPASS
DetCo: Unsupervised Contrastive Learning for Object Detection
Unlike most recent methods that focused on improving accuracy of image classification, we present a novel contrastive learning approach, named DetCo, which fully explores the contrasts between global image and local image patches to learn discriminative representations for object detection.
Segmenting Transparent Object in the Wild with Transformer
This work presents a new fine-grained transparent object segmentation dataset, termed Trans10K-v2, extending Trans10K-v1, the first large-scale transparent object segmentation dataset.
Ranked #3 on
Semantic Segmentation
on Trans10K
Polygon-free: Unconstrained Scene Text Detection with Box Annotations
For example, without using polygon annotations, PSENet achieves an 80. 5% F-score on TotalText [3] (vs. 80. 9% of fully supervised counterpart), 31. 1% better than training directly with upright bounding box annotations, and saves 80%+ labeling costs.
Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection
Such a property makes the distribution statistics of a bounding box highly correlated to its real localization quality.
Ranked #26 on
Object Detection
on COCO-O
AE TextSpotter: Learning Visual and Linguistic Representation for Ambiguous Text Spotting
Unlike previous works that merely employed visual features for text detection, this work proposes a novel text spotter, named Ambiguity Eliminating Text Spotter (AE TextSpotter), which learns both visual and linguistic features to significantly reduce ambiguity in text detection.
Differentiable Hierarchical Graph Grouping for Multi-Person Pose Estimation
The modules of HGG can be trained end-to-end with the keypoint detection network and is able to supervise the grouping process in a hierarchical manner.
Ranked #3 on
Keypoint Detection
on OCHuman
Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection
Specifically, we merge the quality estimation into the class prediction vector to form a joint representation of localization quality and classification, and use a vector to represent arbitrary distribution of box locations.
Ranked #105 on
Object Detection
on COCO test-dev
Scene Text Image Super-Resolution in the Wild
For example, it outperforms LapSRN by over 5% and 8%on the recognition accuracy of ASTER and CRNN.
Segmenting Transparent Objects in the Wild
To address this important problem, this work proposes a large-scale dataset for transparent object segmentation, named Trans10K, consisting of 10, 428 images of real scenarios with carefully manual annotations, which are 10 times larger than the existing datasets.
Ranked #4 on
Semantic Segmentation
on Trans10K
False Data Injection Attacks and the Distributed Countermeasure in DC Microgrids
In this paper, we consider a hierarchical control based DC microgrid (DCmG) equipped with unknown input observer (UIO) based detectors, where the potential false data injection (FDI) attacks and the distributed countermeasure are investigated.
PolarMask: Single Shot Instance Segmentation with Polar Representation
In this paper, we introduce an anchor-box free and single shot instance segmentation method, which is conceptually simple, fully convolutional and can be used as a mask prediction module for instance segmentation, by easily embedding it into most off-the-shelf detection methods.
Ranked #100 on
Instance Segmentation
on COCO test-dev
TextSR: Content-Aware Text Super-Resolution Guided by Recognition
Nonetheless, most of the previous methods may not work well in recognizing text with low resolution which is often seen in natural scene images.
Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network
Recently, some methods have been proposed to tackle arbitrary-shaped text detection, but they rarely take the speed of the entire pipeline into consideration, which may fall short in practical applications. In this paper, we propose an efficient and accurate arbitrary-shaped text detector, termed Pixel Aggregation Network (PAN), which is equipped with a low computational-cost segmentation head and a learnable post-processing.
Ranked #8 on
Scene Text Detection
on SCUT-CTW1500
Shape Robust Text Detection with Progressive Scale Expansion Network
Due to the fact that there are large geometrical margins among the minimal scale kernels, our method is effective to split the close text instances, making it easier to use segmentation-based methods to detect arbitrary-shaped text instances.
Ranked #12 on
Scene Text Detection
on SCUT-CTW1500
Selective Kernel Networks
A building block called Selective Kernel (SK) unit is designed, in which multiple branches with different kernel sizes are fused using softmax attention that is guided by the information in these branches.
Ranked #99 on
Image Classification
on CIFAR-100
(using extra training data)
Shape Robust Text Detection with Progressive Scale Expansion Network
To address these problems, we propose a novel Progressive Scale Expansion Network (PSENet), designed as a segmentation-based detector with multiple predictions for each text instance.
Ranked #12 on
Scene Text Detection
on ICDAR 2017 MLT
Mixed Link Networks
Basing on the analysis by revealing the equivalence of modern networks, we find that both ResNet and DenseNet are essentially derived from the same "dense topology", yet they only differ in the form of connection -- addition (dubbed "inner link") vs. concatenation (dubbed "outer link").
