Long-range modeling
45 papers with code • 2 benchmarks • 4 datasets
A new task for testing the long-sequence modeling capabilities and efficiency of language models.
Image credit: SCROLLS: Standardized CompaRison Over Long Language Sequences
Libraries
Use these libraries to find Long-range modeling models and implementations
LRA
SCROLLS
MuLD
Pathfinder-X2
Most implemented papers
T-former: An Efficient Transformer for Image Inpainting
dengyecode/t-former_image_inpainting
•
•
And based on this attention, a network called $T$-former is designed for image inpainting.
MambaMorph: a Mamba-based Framework for Medical MR-CT Deformable Registration
Capturing voxel-wise spatial correspondence across distinct modalities is crucial for medical image analysis.
VM-UNet: Vision Mamba UNet for Medical Image Segmentation
To our best knowledge, this is the first medical image segmentation model constructed based on the pure SSM-based model.
V4D:4D Convolutional Neural Networks for Video-level Representation Learning
Most existing 3D CNNs for video representation learning are clip-based methods, and thus do not consider video-level temporal evolution of spatio-temporal features.
DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning
whwu95/DSANet
•
•
Long-range and short-range temporal modeling are two complementary and crucial aspects of video recognition.
Image Super-Resolution With Non-Local Sparse Attention
HarukiYqM/Non-Local-Sparse-Attention
•
•
CVPR 2021
NLSA is designed to retain long-range modeling capability from NL operation while enjoying robustness and high-efficiency of sparse representation.
Sparse Factorization of Large Square Matrices
ruslankhalitov/sparsefactorization
•
•
The sparse factorization method is tested for a variety of synthetic and real-world square matrices.
LongT5: Efficient Text-To-Text Transformer for Long Sequences
google-research/longt5
•
•
Findings (NAACL) 2022
Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models.
Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks
Recurrent Neural Networks, Transformers, and Convolutional Neural Networks are three major techniques for learning from sequential data.
Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention
wiedersehne/paramixer
•
•
CVPR 2022
The overall computing cost of the new building block is as low as $O(N \log N)$.
