Search Results for author:
Found 8 papers, 3 papers with code
Tradeoffs Between Alignment and Helpfulness in Language Models
Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks.
Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play Framework
Here we propose $\textit{Align With Purpose}$, a $\textbf{general Plug-and-Play framework}$ for enhancing a desired property in models trained with the CTC criterion.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Fundamental Limitations of Alignment in Large Language Models
An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users.
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks
Recently, several works have demonstrated high gains by taking a straightforward approach for incorporating intermediate supervision in compounded natural language problems: the sequence-to-sequence LM is fed with an augmented input, in which the decomposed tasks' labels are simply concatenated to the original input.
The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design
We highlight a bias introduced by this common practice: we prove that the pretrained NLM can model much stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples.
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
After their successful debut in natural language processing, Transformer architectures are now becoming the de-facto standard in many domains.
The Depth-to-Width Interplay in Self-Attention
Our guidelines elucidate the depth-to-width trade-off in self-attention networks of sizes up to the scale of GPT3 (which we project to be too deep for its size), and beyond, marking an unprecedented width of 30K as optimal for a 1-Trillion parameter network.
Deep autoregressive models for the efficient variational simulation of many-body quantum systems
Artificial Neural Networks were recently shown to be an efficient representation of highly-entangled many-body quantum states.
