Search Results for author:
Found 17 papers, 6 papers with code
AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Models
In-context learning (ICL) allows a language model to improve its problem-solving capability when provided with suitable information in context.
On the Power of Context-Enhanced Learning in LLMs
We formalize a new concept for LLMs, context-enhanced learning.
Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
We seek strategies for training on the SIMPLE version of the tasks that improve performance on the corresponding HARD task, i. e., S2H generalization.
Progressive distillation induces an implicit curriculum
Our theoretical and empirical findings on sparse parity, complemented by empirical observations on more complex tasks, highlight the benefit of progressive distillation via implicit curriculum across setups.
Representing Rule-based Chatbots with Transformers
Next, we train Transformers on a dataset of synthetically generated ELIZA conversations and investigate the mechanisms the models learn.
Efficient Stagewise Pretraining via Progressive Subnetworks
We propose an instantiation of this framework - Random Part Training (RAPTR) - that selects and trains only a random subnetwork (e. g. depth-wise, width-wise) of the network at each step, progressively increasing the size in stages.
Trainable Transformer in Transformer
In this work, we propose an efficient construction, Transformer in Transformer (in short, TinT), that allows a transformer to simulate and fine-tune complex models internally during inference (e. g., pre-trained language models).
Do Transformers Parse while Predicting the Masked Word?
We also show that the Inside-Outside algorithm is optimal for masked language modeling loss on the PCFG-generated data.
Task-Specific Skill Localization in Fine-tuned Language Models
Given the downstream task and a model fine-tuned on that task, a simple optimization is used to identify a very small subset of parameters ($\sim0. 01$% of model parameters) responsible for ($>95$%) of the model's performance, in the sense that grafting the fine-tuned values for just this tiny subset onto the pre-trained model gives performance almost as well as the fine-tuned model.
On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
Approximating Stochastic Gradient Descent (SGD) as a Stochastic Differential Equation (SDE) has allowed researchers to enjoy the benefits of studying a continuous optimization trajectory while carefully preserving the stochasticity of SGD.
Understanding Gradient Descent on Edge of Stability in Deep Learning
The current paper mathematically analyzes a new mechanism of implicit regularization in the EoS phase, whereby GD updates due to non-smooth loss landscape turn out to evolve along some deterministic flow on the manifold of minimum loss.
Learning and Generalization in RNNs
In contrast to the previous work that could only deal with functions of sequences that are sums of functions of individual tokens in the sequence, we allow general functions.
Non-Gaussianity of Stochastic Gradient Noise
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training?
Effect of Activation Functions on the Training of Overparametrized Neural Nets
In the present paper, we provide theoretical results about the effect of activation function on the training of highly overparametrized 2-layer neural networks.
Word2Sense: Sparse Interpretable Word Embeddings
We present an unsupervised method to generate Word2Sense word embeddings that are interpretable {---} each dimension of the embedding space corresponds to a fine-grained sense, and the non-negative value of the embedding along the j-th dimension represents the relevance of the j-th sense to the word.
DeepTagRec: A Content-cum-User based Tag Recommendation Framework for Stack Overflow
In this paper, we develop a content-cum-user based deep learning framework DeepTagRec to recommend appropriate question tags on Stack Overflow.
Analysis on Gradient Propagation in Batch Normalized Residual Networks
We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work.
