Search Results for author:
Found 15 papers, 7 papers with code
Towards Understanding Inductive Bias in Transformers: A View From Infinity
We study inductive bias in Transformers in the infinitely over-parameterized Gaussian process limit and argue transformers tend to be biased towards more permutation symmetric functions in sequence space.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Ranked #1 on
Question Answering
on StrategyQA
Exploring Length Generalization in Large Language Models
The ability to extrapolate from short problem instances to longer ones is an important form of out-of-distribution generalization in reasoning tasks, and is crucial when learning from datasets where longer problem instances are rare.
Solving Quantitative Reasoning Problems with Language Models
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding.
Ranked #21 on
Math Word Problem Solving
on MATH
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Coreference Resolution
on Winograd Schema Challenge
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs.
The large learning rate phase of deep learning
In the small learning rate phase, training can be understood using the existing theory of infinitely wide neural networks.
Are wider nets better given the same number of parameters?
Empirical studies demonstrate that the performance of neural networks improves with increasing number of parameters.
On the training dynamics of deep networks with $L_2$ regularization
Finally, we show that these empirical relations can be understood theoretically in the context of infinitely wide networks.
On the asymptotics of wide networks with polynomial activations
We consider an existing conjecture addressing the asymptotic behavior of neural networks in the large width limit.
The large learning rate phase of deep learning: the catapult mechanism
In the small learning rate phase, training can be understood using the existing theory of infinitely wide neural networks.
Wider Networks Learn Better Features
Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task.
Asymptotics of Wide Networks from Feynman Diagrams
Understanding the asymptotic behavior of wide networks is of considerable interest.
Gradient Descent Happens in a Tiny Subspace
We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training.
