Search Results for author:
Found 12 papers, 4 papers with code
PALBERT”:" Teaching ALBERT to Ponder
Currently, pre-trained models can be considered the default choice for a wide range of NLP tasks.
Learn Your Reference Model for Real Good Alignment
For instance, in the fundamental Reinforcement Learning From Human Feedback (RLHF) technique of Language Model alignment, in addition to reward maximization, the Kullback-Leibler divergence between the trainable policy and the SFT policy is minimized.
Linear Transformers with Learnable Kernel Functions are Better In-Context Models
Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing.
Diffusion Language Models Generation Can Be Halted Early
We evaluate our methods on Plaid, SSD, and CDCD DLMs and create a cohesive perspective on their generation workflows.
Ahead-of-Time P-Tuning
In this paper, we propose Ahead-of-Time (AoT) P-Tuning, a novel parameter-efficient fine-tuning method for pre-trained Language Models (LMs) that adds input-dependent bias before each Transformer layer.
Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Models
The simplest way to obtain continuous interpolation between two points in high dimensional space is to draw a line between them.
Classifiers are Better Experts for Controllable Text Generation
This paper proposes a simple method for controllable text generation based on weighting logits with a free-form classifier, namely CAIF sampling.
PALBERT: Teaching ALBERT to Ponder
Recently proposed PonderNet may be a promising solution for performing an early exit by treating the exit layer's index as a latent variable.
FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks
In this paper, we explore Linear Transformer models, rethinking their two core components.
Implicit Unlikelihood Training: Improving Neural Text Generation with Reinforcement Learning
Likelihood training and maximization-based decoding result in dull and repetitive generated texts even when using powerful language models (Holtzman et al., 2019).
Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
We also proposed a variant of Weight Squeezing called Gated Weight Squeezing, for which we combined fine-tuning of BERT-Medium model and learning mapping from BERT-Base weights.
Self-Attentive Model for Headline Generation
Headline generation is a special type of text summarization task.
