Search Results for author:
Found 65 papers, 50 papers with code
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences.
Mini-Monkey: Multi-Scale Adaptive Cropping for Multimodal Large Language Models
This issue is particularly evident in lightweight MLLMs.
Multi-Prompting Decoder Helps Better Language Understanding
Despite the effectiveness of these methods, they only use a single prompt to query PLMs for decoding, leading to a heavy reliance on the quality of the adopted prompt.
MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks
The sparsely activated mixture of experts (MoE) model presents a promising alternative to traditional densely activated (dense) models, enhancing both quality and computational efficiency.
Puzzle Pieces Picker: Deciphering Ancient Chinese Characters with Radical Reconstruction
Oracle Bone Inscriptions is one of the oldest existing forms of writing in the world.
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
In this study, we investigate whether attention-based information flow inside large language models (LLMs) is aggregated through noticeable patterns for long context processing.
Deciphering Oracle Bone Language with Diffusion Models
Originating from China's Shang Dynasty approximately 3, 000 years ago, the Oracle Bone Script (OBS) is a cornerstone in the annals of linguistic history, predating many established writing systems.
Dataset and Benchmark for Urdu Natural Scenes Text Detection, Recognition and Visual Question Answering
The development of Urdu scene text detection, recognition, and Visual Question Answering (VQA) technologies is crucial for advancing accessibility, information retrieval, and linguistic diversity in digital content, facilitating better understanding and interaction with Urdu-language visual data.
MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering
Text-Centric Visual Question Answering (TEC-VQA) in its proper format not only facilitates human-machine interaction in text-centric visual environments but also serves as a de facto gold proxy to evaluate AI models in the domain of text-centric scene understanding.
The First Swahili Language Scene Text Detection and Recognition Dataset
We propose a comprehensive dataset of Swahili scene text images and evaluate the dataset on different scene text detection and recognition models.
Exploring the Capabilities of Large Multimodal Models on Dense Text
To further explore the capabilities of LMM in complex text tasks, we propose the DT-VQA dataset, with 170k question-answer pairs.
VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization
Typically, we propose a Prompt Queries Generation Module and a Tasks-aware Adapter to effectively convert the original single-task model into a multi-task model suitable for both image and video scenarios with minimal additional parameters.
TextSquare: Scaling up Text-Centric Visual Instruction Tuning
Text-centric visual question answering (VQA) has made great strides with the development of Multimodal Large Language Models (MLLMs), yet open-source models still fall short of leading models like GPT4V and Gemini, partly due to a lack of extensive, high-quality instruction tuning data.
Bridging the Gap Between End-to-End and Two-Step Text Spotting
Subsequently, we introduce a Bridge that connects the locked detector and recognizer through a zero-initialized neural network.
OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition
Recently, visually-situated text parsing (VsTP) has experienced notable advancements, driven by the increasing demand for automated document understanding and the emergence of Generative Large Language Models (LLMs) capable of processing document-based questions.
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks.
Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
In light of recent advances in multimodal Large Language Models (LLMs), there is increasing attention to scaling them from image-text data to more informative real-world videos.
Ranked #3 on
Text-to-Video Generation
on MSR-VTT
An open dataset for oracle bone script recognition and decipherment
Oracle bone script, one of the earliest known forms of ancient Chinese writing, presents invaluable research materials for scholars studying the humanities and geography of the Shang Dynasty, dating back 3, 000 years.
An open dataset for the evolution of oracle bone characters: EVOBC
We conducted validation and simulated deciphering on the constructed dataset, and the results demonstrate its high efficacy in aiding the study of oracle bone script.
SwinTextSpotter v2: Towards Better Synergy for Scene Text Spotting
In this paper, we propose a new end-to-end scene text spotting framework termed SwinTextSpotter v2, which seeks to find a better synergy between text detection and recognition.
OmniParser: A Unified Framework for Text Spotting Key Information Extraction and Table Recognition
Recently visually-situated text parsing (VsTP) has experienced notable advancements driven by the increasing demand for automated document understanding and the emergence of Generative Large Language Models (LLMs) capable of processing document-based questions.
Progressive Evolution from Single-Point to Polygon for Scene Text
We demonstrate the accuracy of the generated polygons through extensive experiments: 1) By creating polygons from ground truth points, we achieved an accuracy of 82. 0% on ICDAR 2015; 2) In training detectors with polygons generated by our method, we attained 86% of the accuracy relative to training with ground truth (GT); 3) Additionally, the proposed Point2Polygon can be seamlessly integrated to empower single-point spotters to generate polygons.
Toward Real Text Manipulation Detection: New Dataset and New Solution
With the surge in realistic text tampering, detecting fraudulent text in images has gained prominence for maintaining information security.
Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models
We contend that one main limitation of existing generation methods is the insufficient integration of foreground text with the background.
ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
Despite Large Language Models (LLMs) like GPT-4 achieving impressive results in function-level code generation, they struggle with repository-scale code understanding (e. g., coming up with the right arguments for calling routines), requiring a deeper comprehension of complex file interactions.
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
Additionally, experiments on 18 datasets further demonstrate that Monkey surpasses existing LMMs in many tasks like Image Captioning and various Visual Question Answering formats.
Ranked #13 on
MMR total
on MRR-Benchmark
(using extra training data)
KwaiYiiMath: Technical Report
Recent advancements in large language models (LLMs) have demonstrated remarkable abilities in handling a variety of natural language processing (NLP) downstream tasks, even on mathematical tasks requiring multi-step reasoning.
Ranked #87 on
Arithmetic Reasoning
on GSM8K
(using extra training data)
Turning a CLIP Model into a Scene Text Spotter
Utilizing only 10% of the supervised data, FastTCM-CR50 improves performance by an average of 26. 5% and 5. 5% for text detection and spotting tasks, respectively.
ESTextSpotter: Towards Better Scene Text Spotting with Explicit Synergy in Transformer
To this end, we introduce a new model named Explicit Synergy-based Text Spotting Transformer framework (ESTextSpotter), which achieves explicit synergy by modeling discriminative and interactive features for text detection and recognition within a single decoder.
Box-DETR: Understanding and Boxing Conditional Spatial Queries
In DAB-DETR, such queries are modulated by the so-called conditional linear projection at each decoder stage, aiming to search for positions of interest such as the four extremities of the box.
On Point Affiliation in Feature Upsampling
We introduce the notion of point affiliation into feature upsampling.
ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining
As ViTEraser implicitly integrates text localization and inpainting, we propose a novel end-to-end pretraining method, termed SegMIM, which focuses the encoder and decoder on the text box segmentation and masked image modeling tasks, respectively.
Looking and Listening: Audio Guided Text Recognition
Text recognition in the wild is a long-standing problem in computer vision.
ICDAR 2023 Competition on Structured Text Extraction from Visually-Rich Document Images
It is hoped that this competition will attract many researchers in the field of CV and NLP, and bring some new thoughts to the field of Document AI.
OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
In this paper, we conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER).
ICDAR 2023 Competition on Reading the Seal Title
To promote research in this area, we organized ICDAR 2023 competition on reading the seal title (ReST), which included two tasks: seal title text detection (Task 1) and end-to-end seal title recognition (Task 2).
Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation
This stream is subsequently fed into the decoder-based transformer to generate visual re-creations and textual feedback in the second stage.
Turning a CLIP Model into a Scene Text Detector
Recently, pretraining approaches based on vision language models have made effective progresses in the field of text detection.
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
To address these challenges, we introduce a system that can jointly optimize distributed execution and gradient checkpointing plans.
SPTS v2: Single-Point Scene Text Spotting
Within the context of our SPTS v2 framework, our experiments suggest a potential preference for single-point representation in scene text spotting when compared to other representations.
Ranked #15 on
Text Spotting
on ICDAR 2015
Towards Robust Tampered Text Detection in Document Image: New Dataset and New Solution
In this paper, we propose a novel framework to capture more fine-grained clues in complex scenarios for tampered text detection, termed as Document Tampering Detector (DTD), which consists of a Frequency Perception Head (FPH) to compensate the deficiencies caused by the inconspicuous visual features, and a Multi-view Iterative Decoder (MID) for fully utilizing the information of features in different scales.
MSDS: A Large-Scale Chinese Signature and Token Digit String Dataset for Handwriting Verification
MSDS-ChS consists of handwritten Chinese signatures, which, to the best of our knowledge, is the largest publicly available Chinese signature dataset for handwriting verification, at least eight times larger than existing online datasets.
SAPA: Similarity-Aware Point Affiliation for Feature Upsampling
We introduce point affiliation into feature upsampling, a notion that describes the affiliation of each upsampled point to a semantic cluster formed by local decoder feature points with semantic similarity.
Ranked #5 on
Feature Upsampling
on ImageNet
PageNet: Towards End-to-End Weakly Supervised Page-Level Handwritten Chinese Text Recognition
Utilizing the proposed weakly supervised learning framework, PageNet requires only transcripts to be annotated for real data; however, it can still output detection and recognition results at both the character and line levels, avoiding the labor and cost of labeling bounding boxes of characters and text lines.
Don't Forget Me: Accurate Background Recovery for Text Removal via Modeling Local-Global Context
To address this issue, we propose a Contextual-guided Text Removal Network, termed as CTRNet.
DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation
To the best of our knowledge, we are the first to make a reasonable dynamic runtime scheduler on the combination of tensor swapping and tensor recomputation without user oversight.
SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition
End-to-end scene text spotting has attracted great attention in recent years due to the success of excavating the intrinsic synergy of the scene text detection and recognition.
Ranked #3 on
Text Spotting
on Inverse-Text
SPTS: Single-Point Text Spotting
For the first time, we demonstrate that training scene text spotting models can be achieved with an extremely low-cost annotation of a single-point for each instance.
Ranked #3 on
Text Spotting
on SCUT-CTW1500
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
The success of Transformer models has pushed the deep learning model scale to billions of parameters.
ICDAR 2021 Competition on Integrated Circuit Text Spotting and Aesthetic Assessment
With hundreds of thousands of electronic chip components are being manufactured every day, chip manufacturers have seen an increasing demand in seeking a more efficient and effective way of inspecting the quality of printed texts on chip components.
ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting
Previous methods can be roughly categorized into two groups: character-based and segmentation-based, which often require character-level annotations and/or complex post-processing due to the unstructured output.
Ranked #7 on
Text Spotting
on Inverse-Text
Structured Multimodal Attentions for TextVQA
In this paper, we propose an end-to-end structured multimodal attention (SMA) neural network to mainly solve the first two issues above.
On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
Visual Question Answering (VQA) methods have made incredible progress, but suffer from a failure to generalize.
ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network
Our contributions are three-fold: 1) For the first time, we adaptively fit arbitrarily-shaped text by a parameterized Bezier curve.
Ranked #9 on
Text Spotting
on Inverse-Text
Separating Content from Style Using Adversarial Learning for Recognizing Text in the Wild
Furthermore, to tackle the issue of lacking paired training samples, we design an interactive joint training scheme, which shares attention masks from the recognizer to the discriminator, and enables the discriminator to extract the features of each character for further adversarial training.
Exploring the Capacity of an Orderless Box Discretization Network for Multi-orientation Scene Text Detection
More importantly, based on OBD, we provide a detailed analysis of the impact of a collection of refinements, which may inspire others to build state-of-the-art text detectors.
Ranked #3 on
Scene Text Detection
on ICDAR 2017 MLT
ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVT
Robust text reading from street view images provides valuable information for various applications.
ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)
This paper reports the ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT) that consists of three major challenges: i) scene text detection, ii) scene text recognition, and iii) scene text spotting.
Omnidirectional Scene Text Detection with Sequential-free Box Discretization
Scene text in the wild is commonly presented with high variant characteristics.
Ranked #1 on
Scene Text Detection
on IC19-ReCTs
(using extra training data)
Aggregation Cross-Entropy for Sequence Recognition
In this paper, we propose a novel method, aggregation cross-entropy (ACE), for sequence recognition from a brand new perspective.
Tightness-aware Evaluation Protocol for Scene Text Detection
Evaluation protocols play key role in the developmental progress of text detection methods.
EnsNet: Ensconce Text in the Wild
The feature of the former is first enhanced by a novel lateral connection structure and then refined by four carefully designed losses: multiscale regression loss and content loss, which capture the global discrepancy of different level features; texture loss and total variation loss, which primarily target filling the text region and preserving the reality of the background.
DeRPN: Taking a further step toward more general object detection
However, the detection performance is sensitive to the setting of the anchor boxes.
Feature Enhancement Network: A Refined Scene Text Detector
In this paper, we propose a refined scene text detector with a \textit{novel} Feature Enhancement Network (FEN) for Region Proposal and Text Detection Refinement.
Deep Matching Prior Network: Toward Tighter Multi-oriented Text Detection
The effectiveness of our approach is evaluated on a public word-level, multi-oriented scene text database, ICDAR 2015 Robust Reading Competition Challenge 4 "Incidental scene text localization".
