Search Results for author:
Found 13 papers, 6 papers with code
Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning
The unusual properties of in-context learning (ICL) have prompted investigations into the internal mechanisms of large language models.
A statistical theory of overfitting for imbalanced classification
Classification with imbalanced data is a common challenge in data analysis, where certain classes (minority classes) account for a small fraction of the training data compared with other classes (majority classes).
Assessing and improving reliability of neighbor embedding methods: a map-continuity perspective
We find that the manifold learning interpretations from many prior works are inaccurate and that the misuse stems from a lack of data-independent notions of embedding maps, which project high-dimensional data into a lower-dimensional space.
Out-of-distribution generalization via composition: a lens through induction heads in Transformers
We empirically examined the training dynamics of Transformers on a synthetic example and conducted extensive experiments on a variety of pretrained LLMs, focusing on a type of components known as induction heads.
Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity
Furthermore, these tuning-based methods require large-scale preference data for training and are susceptible to noisy preference data.
How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes
Large language models (LLM) have recently shown the extraordinary ability to perform unseen tasks based on few-shot examples provided as text, also known as in-context learning (ICL).
Uncovering hidden geometry in Transformers via disentangling position and context
Given embedding vector $\boldsymbol{h}_{c, t} \in \mathbb{R}^d$ at sequence position $t \le T$ in a sequence (or context) $c \le C$, extracting the mean effects yields the decomposition \[ \boldsymbol{h}_{c, t} = \boldsymbol{\mu} + \mathbf{pos}_t + \mathbf{ctx}_c + \mathbf{resid}_{c, t} \] where $\boldsymbol{\mu}$ is the global mean vector, $\mathbf{pos}_t$ and $\mathbf{ctx}_c$ are the mean vectors across contexts and across positions respectively, and $\mathbf{resid}_{c, t}$ is the residual vector.
Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkage
Firstly, through empirical and theoretical analysis, we identify two crucial effects -- expansion and shrinkage -- induced by the contrastive loss on the projectors.
Tractability from overparametrization: The example of the negative perceptron
In the negative perceptron problem we are given $n$ data points $({\boldsymbol x}_i, y_i)$, where ${\boldsymbol x}_i$ is a $d$-dimensional vector and $y_i\in\{+1,-1\}$ is a binary label.
The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training
We assume that both the sample size $n$ and the dimension $d$ are large, and they are polynomially related.
A Selective Overview of Deep Learning
Deep learning has arguably achieved tremendous success in recent years.
Robust high dimensional factor models with applications to statistical machine learning
Factor models are a class of powerful statistical models that have been widely used to deal with dependent measurements that arise frequently from various applications from genomics and neuroscience to economics and finance.
Differentially Private Data Releasing for Smooth Queries with Synthetic Database Output
We develop an $\epsilon$-differentially private mechanism for the class of $K$-smooth queries.
