Search Results for author:
Found 7 papers, 4 papers with code
RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval
This paper investigates the gap in representation powers of Recurrent Neural Networks (RNNs) and Transformers in the context of solving algorithmic problems.
Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars
Interpretability methods aim to understand the algorithm implemented by a trained model (e. g., a Transofmer) by examining various aspects of the model, such as the weight matrices or the attention patterns.
Practically Solving LPN in High Noise Regimes Faster Using Neural Networks
For some settings we are also able to provide theories that explain the rationale of the design of our models.
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Furthermore, we demonstrate the skill neurons are most likely generated in pre-training rather than fine-tuning by showing that the skill neurons found with prompt tuning are also crucial for other fine-tuning methods freezing neuron weights, such as the adapter-based tuning and BitFit.
How Does Sharpness-Aware Minimization Minimize Sharpness?
SAM intends to penalize a notion of sharpness of the model but implements a computationally efficient variant; moreover, a third notion of sharpness was used for proving generalization guarantees.
Benign Overfitting in Classification: Provably Counter Label Noise with Larger Models
Studies on benign overfitting provide insights for the success of overparameterized deep learning models.
On Transferability of Prompt Tuning for Natural Language Processing
To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work.
