Search Results for author:
Found 8 papers, 2 papers with code
What makes a good feedforward computational graph?
As implied by the plethora of literature on graph rewiring, the choice of computational graph employed by a neural network can make a significant impact on its downstream performance.
On the consistency of hyper-parameter selection in value-based deep reinforcement learning
This paper conducts an extensive empirical study focusing on the reliability of hyper-parameter selection for value-based deep reinforcement learning agents, including the introduction of a new score to quantify the consistency and reliability of various hyper-parameters.
Transformers need glasses! Information over-squashing in language tasks
We rely on a theoretical signal propagation analysis -- specifically, we analyse the representations of the last token in the final layer of the Transformer, as this is the representation used for next-token prediction.
Position: Categorical Deep Learning is an Algebraic Theory of All Architectures
We present our position on the elusive quest for a general-purpose framework for specifying and studying deep learning architectures.
Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
As a result, it is usually necessary to reproduce the experiments from scratch, which can be time-consuming and error-prone.
Scalable Training of Language Models using JAX pjit and TPUv4
Modern large language models require distributed training strategies due to their size.
No News is Good News: A Critique of the One Billion Word Benchmark
The One Billion Word Benchmark is a dataset derived from the WMT 2011 News Crawl, commonly used to measure language modeling ability in natural language processing.
Mitigating harm in language models with conditional-likelihood filtration
Language models trained on large-scale unfiltered datasets curated from the open web acquire systemic biases, prejudices, and harmful views from their training data.
