Trending Research
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
However, key-value (KV) cache, which is stored to avoid re-computation, has emerged as a critical bottleneck by growing linearly in size with the sequence length.
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT-full
(using extra training data)
Rethinking Inductive Biases for Surface Normal Estimation
Despite the growing demand for accurate surface normal estimation models, existing methods use general-purpose dense prediction models, adopting the same inductive biases as other tasks.
Efficient Multimodal Learning from Data-centric Perspective
baai-dcai/bunny
•
•
Multimodal Large Language Models (MLLMs) have demonstrated notable capabilities in general visual understanding and reasoning tasks.
LongEmbed: Extending Embedding Models for Long Context Retrieval
This paper explores context window extension of existing embedding models, pushing the limit to 32k without requiring additional training.
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy.
BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models
ledzy/badam
•
•
This work presents BAdam, an optimizer that leverages the block coordinate optimization framework with Adam as the inner solver.
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation.
State Space Model for New-Generation Network Alternative to Transformers: A Survey
event-ahu/mamba_state_space_model_paper_list
•
•
In this paper, we give the first comprehensive review of these works and also provide experimental comparisons and analysis to better demonstrate the features and advantages of SSM.
LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models
We present LongQLoRA, an efficient and effective method to extend context length of large language models with less training resources.
