Chunking
67 papers with code • 6 benchmarks • 5 datasets
Chunking, also known as shallow parsing, identifies continuous spans of tokens that form syntactic units such as noun phrases or verb phrases.
Example:
| Vinken | , | 61 | years | old |
|---|---|---|---|---|
| B-NLP | I-NP | I-NP | I-NP | I-NP |
Benchmarks
These leaderboards are used to track progress in Chunking
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Chunking models and implementations
Penn Treebank
CoNLL 2003
Latest papers with no code
Improving Retrieval for RAG based Question Answering Models on Financial Documents
The effectiveness of Large Language Models (LLMs) in generating accurate responses relies heavily on the quality of input provided, particularly when employing Retrieval Augmented Generation (RAG) techniques.
BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models
In this work, we proposeExtensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness.
Grounding Language Model with Chunking-Free In-Context Retrieval
These strategies not only improve the efficiency of the retrieval process but also ensure that the fidelity of the generated grounding text evidence is maintained.
Punctuation Restoration Improves Structure Understanding without Supervision
Unsupervised learning objectives like language modeling and de-noising constitute a significant part in producing pre-trained models that perform various downstream applications from natural language understanding to conversational tasks.
Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT
Retrieval pipelines-an integral component of many machine learning systems-perform poorly in domains where documents are long (e. g., 10K tokens or more) and where identifying the relevant document requires synthesizing information across the entire text.
Financial Report Chunking for Effective Retrieval Augmented Generation
We introduce a novel framework that evaluates how chunking based on element types annotated by document understanding models contributes to the overall context and accuracy of the information retrieved.
Releasing the CRaQAn (Coreference Resolution in Question-Answering): An open-source dataset and dataset creation methodology using instruction-following models
Instruction-following language models demand robust methodologies for information retrieval to augment instructions for question-answering applications.
A recurrent connectionist model of melody perception : An exploration using TRACX2
We address this question by exploring how TRACX2, (French et al., 2011; French \& Cottrell, 2014; Mareschal \& French, 2017), a recognition-based, recursive connectionist autoencoder model of chunking and sequence segmentation, which has successfully simulated speech and serial-image processing, might be applied to elementary melody perception.
Breaking the Token Barrier: Chunking and Convolution for Efficient Long Text Classification with BERT
Various complex methods have claimed to overcome this limit, but recent research questions the efficacy of these models across different classification tasks.
Symmetrical SyncMap for Imbalanced General Chunking Problems
The main idea is to apply equal updates from negative and positive feedback loops by symmetrical activation.
