Search Results for author:
Found 14 papers, 0 papers with code
Attention is Naturally Sparse with Gaussian Distributed Input
The computational intensity of Large Language Models (LLMs) is a critical bottleneck, primarily due to the $O(n^2)$ complexity of the attention mechanism in transformer architectures.
Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence
Based on SGD, previous works have proposed many algorithms that have improved convergence speed and generalization in stochastic optimization, such as SGDm, AdaGrad, Adam, etc.
Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights
In the realm of deep learning, transformers have emerged as a dominant architecture, particularly in natural language processing tasks.
Superiority of Softmax: Unveiling the Performance Edge Over Linear Attention
Large transformer models have achieved state-of-the-art results in numerous natural language processing tasks.
Clustered Linear Contextual Bandits with Knapsacks
Thus, maximizing the total reward requires learning not only models about the reward and the resource consumption, but also cluster memberships.
Convergence of Two-Layer Regression with Nonlinear Units
Softmax unit and ReLU unit are the key structure in attention computation.
Zero-th Order Algorithm for Softmax Attention Optimization
We demonstrate the convergence of our algorithm, highlighting its effectiveness in efficiently computing gradients for large-scale LLMs.
Faster Robust Tensor Power Method for Arbitrary Order
Tensor decomposition is a fundamental method used in various areas to deal with high-dimensional data.
Attention Scheme Inspired Softmax Regression
One of the key computation in LLMs is the softmax unit.
Solving Tensor Low Cycle Rank Approximation
For the tensor classical rank, tucker rank and train rank, it has been well studied in [Song, Woodruff, Zhong SODA 2019].
Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimension
It runs in $\widetilde{O}(\mathrm{nnz}(X) + n^{\omega} ) $ time, has $1-\delta$ succeed probability, and chooses $m = O(n \log(n/\delta))$.
Streaming Kernel PCA Algorithm With Small Space
The kernel method, which is commonly used in learning algorithms such as Support Vector Machines (SVMs), has also been applied in PCA algorithms.
Training Overparametrized Neural Networks in Sublinear Time
The success of deep learning comes at a tremendous computational and energy cost, and the scalability of training massively overparametrized neural networks is becoming a real barrier to the progress of artificial intelligence (AI).
SODA: Site Object Detection dAtaset for Deep Learning in Construction
In this manner, this research contributes a large-scale image dataset for the development of deep learning-based object detection methods in the construction industry and sets up a performance benchmark for further evaluation of corresponding algorithms in this area.
