Search Results for author:
Found 13 papers, 8 papers with code
Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering
However, they often struggle to fully comprehend and effectively utilize their input contexts, resulting in responses that are unfaithful or hallucinated.
Robust Reinforcement Learning from Corrupted Human Feedback
To tackle this challenge, we propose a robust RLHF approach -- $R^3M$, which models the potentially corrupted preference label as sparse outliers.
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference.
Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
In human-written articles, we often leverage the subtleties of text style, such as bold and italics, to guide the attention of readers.
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
These models leverage the attention mechanism to capture long- and short-range dependencies in the sequence.
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Pruning enhances the diversity of low-rank approximations, and low-rank approximation prevents pruning from losing too many expressive neurons.
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e. g., low-rank increments.
Less is More: Task-aware Layer-wise Distillation for Language Model Compression
As such, TED reduces the knowledge gap between the two models and helps the student to fit better on the target task.
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
Large Transformer-based models have exhibited superior performance in various natural language processing and computer vision tasks.
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
We propose MoEBERT, which uses a Mixture-of-Experts structure to increase model capacity and inference speed.
A Biased Graph Neural Network Sampler with Near-Optimal Regret
Graph neural networks (GNN) have recently emerged as a vehicle for applying deep network architectures to graph and relational data.
A Non-asymptotic comparison of SVRG and SGD: tradeoffs between compute and speed
In this paper, we present a non-asymptotic analysis of SVRG under a noisy least squares regression problem.
AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
Adam is shown not being able to converge to the optimal solution in certain cases.
