Search Results for author:
Found 5 papers, 2 papers with code
A phase transition between positional and semantic learning in a solvable model of dot-product attention
We investigate how a dot-product attention layer learns a positional attention matrix (with tokens attending to each other based on their respective positions) and a semantic attention matrix (with tokens attending to each other based on their meaning).
Unveiling the Hessian's Connection to the Decision Boundary
Based on this finding, we provide a new and straightforward approach to studying the complexity of a high-dimensional decision boundary; show that this connection naturally inspires a new generalization measure; and finally, we develop a novel margin estimation technique which, in combination with the generalization measure, precisely identifies minima with simple wide-margin boundaries.
Bandits for Learning to Explain from Explanations
We introduce Explearn, an online algorithm that learns to jointly output predictions and explanations for those predictions.
Towards Neurally Augmented ALISTA
It is well-established that many iterative sparse reconstruction algorithms such as ISTA can be unrolled to yield a learnable neural network for improved empirical performance.
Neurally Augmented ALISTA
A prime example is learned ISTA (LISTA) where weights, step sizes and thresholds are learned from training data.
