Search Results for author:
Found 17 papers, 7 papers with code
Efficient Stagewise Pretraining via Progressive Subnetworks
RaPTr achieves better pre-training loss for BERT and UL2 language models while requiring 20-33% fewer FLOPs compared to standard training, and is competitive or better than other efficient training methods.
Reasoning in Large Language Models Through Symbolic Math Word Problems
Large language models (LLMs) have revolutionized NLP by solving downstream tasks with little to no labeled data.
Task-Specific Skill Localization in Fine-tuned Language Models
Given the downstream task and a model fine-tuned on that task, a simple optimization is used to identify a very small subset of parameters ($\sim0. 01$% of model parameters) responsible for ($>95$%) of the model's performance, in the sense that grafting the fine-tuned values for just this tiny subset onto the pre-trained model gives performance almost as well as the fine-tuned model.
New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Saliency methods compute heat maps that highlight portions of an input that were most {\em important} for the label assigned to it by a deep net.
Understanding Influence Functions and Datamodels via Harmonic Analysis
Influence functions estimate effect of individual data points on predictions of the model on test data and were adapted to deep learning in Koh and Liang [2017].
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs.
On Predicting Generalization using GANs
Research on generalization bounds for deep networks seeks to give ways to predict test error using just the training dataset and the network parameters.
New Definitions and Evaluations for Saliency Methods: Staying Intrinsic and Sound
Saliency methods seek to provide human-interpretable explanations for the output of machine learning model on a given input.
A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning
An effective approach in meta-learning is to utilize multiple "train tasks" to learn a good initialization for model parameters that can help solve unseen "test tasks" with very few samples by fine-tuning from this initialization.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This paper initiates a mathematical study of this phenomenon for the downstream task of text classification by considering the following questions: (1) What is the intuitive connection between the pretraining task of next word prediction and text classification?
Predicting What You Already Know Helps: Provable Self-Supervised Learning
Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data to learn useful semantic representations.
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
In contrast, for the non-convex formulation of a two layer linear network on the same instance, we show that both Reptile and multi-task representation learning can have new task sample complexity of $\mathcal{O}(1)$, demonstrating a separation from convex meta-learning.
Provable Representation Learning for Imitation Learning via Bi-level Optimization
We formulate representation learning as a bi-level optimization problem where the "outer" optimization tries to learn the joint representation and the "inner" optimization encodes the imitation learning setup and tries to learn task-specific parameters.
A Theoretical Analysis of Contrastive Unsupervised Representation Learning
This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes.
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
Motivations like domain adaptation, transfer learning, and feature learning have fueled interest in inducing embeddings for rare or unseen words, n-grams, synsets, and other textual features.
Ranked #3 on
Sentiment Analysis
on MPQA
A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs
We also show a surprising new property of embeddings such as GloVe and word2vec: they form a good sensing matrix for text that is more efficient than random matrices, the standard sparse recovery tool, which may explain why they lead to better representations in practice.
A Large Self-Annotated Corpus for Sarcasm
We introduce the Self-Annotated Reddit Corpus (SARC), a large corpus for sarcasm research and for training and evaluating systems for sarcasm detection.
Ranked #1 on
Sarcasm Detection
on SARC (pol-unbal)
