Search Results for author:
Found 11 papers, 2 papers with code
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
The application of transformer-based models on time series forecasting (TSF) tasks has long been popular to study.
Unlocking the Theory Behind Scaling 1-Bit Neural Networks
Research by Wang et al. (2023); Ma et al. (2024) indicates that the performance of these 1-bit LLMs progressively improves as the number of parameters increases, hinting at the potential existence of a Scaling Law for 1-bit Neural Networks.
Towards Infinite-Long Prefix in Transformer
Prompting and context-based fine-tuning methods, which we call Prefix Learning, have been proposed to enhance the performance of language models on various downstream tasks.
Attention is Naturally Sparse with Gaussian Distributed Input
The computational intensity of Large Language Models (LLMs) is a critical bottleneck, primarily due to the $O(n^2)$ complexity of the attention mechanism in transformer architectures.
Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence
Based on SGD, previous works have proposed many algorithms that have improved convergence speed and generalization in stochastic optimization, such as SGDm, AdaGrad, Adam, etc.
One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space
Considering a single-layer self-attention with Query, Key, and Value matrices $Q, K, V \in \mathbb{R}^{n \times d}$, the polynomial method approximates the attention output $T \in \mathbb{R}^{n \times d}$.
A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer
The Deep Leakage from Gradient (DLG) attack has emerged as a prevalent and highly effective method for extracting sensitive training data by inspecting exchanged gradients.
Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights
In the realm of deep learning, transformers have emerged as a dominant architecture, particularly in natural language processing tasks.
An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent
The delta-bar-delta algorithm is recognized as a learning rate adaptation technique that enhances the convergence speed of the training process in optimization by dynamically scheduling the learning rate based on the difference between the current and previous weight updates.
Fine-tune Language Models to Approximate Unbiased In-context Learning
To address this issue, we introduce a reweighted algorithm called RICL (Reweighted In-context Learning).
How to Protect Copyright Data in Optimization of Large Language Models?
Large language models (LLMs) and generative AI have played a transformative role in computer research and applications.
