Search Results for author:
Found 82 papers, 20 papers with code
Why is prompting hard? Understanding prompts on binary sequence predictors
Large language models (LLMs) can be prompted to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts.
Exponential Speedups by Rerooting Levin Tree Search
We prove that the number of search steps that $\sqrt{\text{LTS}}$ takes is competitive with the best decomposition into subtasks, at the price of a factor that relates to the uncertainty of the rerooter.
RL, but don't do anything I wouldn't do
When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do").
Evaluating Frontier Models for Dangerous Capabilities
To understand the risks posed by a new AI system, we must understand what it can and cannot do.
Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models
We consider the problem of online fine tuning the parameters of a language model at test time, also known as dynamic evaluation.
Learning Universal Predictors
Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data.
Dynamic Knowledge Injection for AIXI Agents
Prior approximations of AIXI, a Bayesian optimality notion for general reinforcement learning, can only approximate AIXI's Bayesian environment model using an a-priori defined set of models.
Distributional Bellman Operators over Mean Embeddings
We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions.
Bridging Algorithmic Information Theory and Machine Learning: A New Approach to Kernel Learning
Machine Learning (ML) and Algorithmic Information Theory (AIT) look at Complexity from different points of view.
Language Modeling Is Compression
We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning.
Line Search for Convex Minimization
However, to the best of our knowledge, there is no principled exact line search algorithm for general convex functions -- including piecewise-linear and max-compositions of convex functions -- that takes advantage of convexity.
Combining a Meta-Policy and Monte-Carlo Planning for Scalable Type-Based Reasoning in Partially Observable Environments
The design of autonomous agents that can interact effectively with other agents without prior coordination is a core problem in multi-agent systems.
Levin Tree Search with Context Models
Levin Tree Search (LTS) is a search algorithm that makes use of a policy (a probability distribution over actions) and comes with a theoretical guarantee on the number of expansions before reaching a goal node, depending on the quality of the policy.
Evaluating Representations with Readout Model Switching
Although much of the success of Deep Learning builds on learning good representations, a rigorous method to evaluate their quality is lacking.
Universal Agent Mixtures and the Geometry of Intelligence
Thus, if RL agent intelligence is quantified in terms of performance across environments, the weighted mixture's intelligence is the weighted average of the original agents' intelligences.
Multi-agent Reinforcement Learning
Reinforcement Learning (RL)
U-Clip: On-Average Unbiased Stochastic Gradient Clipping
U-Clip is a simple amendment to gradient clipping that can be applied to any iterative gradient optimization algorithm.
Memory-Based Meta-Learning on Non-Stationary Distributions
Memory-based meta-learning is a technique for approximating Bayes-optimal predictors.
Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes.
Testing Independence of Exchangeable Random Variables
Given well-shuffled data, can we determine whether the data items are statistically (in)dependent?
Sequential Learning Of Neural Networks for Prequential MDL
In the prequential formulation of MDL, the objective is to minimize the cumulative next-step log-loss when sequentially going through the data and using previous observations for parameter estimation.
Atari-5: Distilling the Arcade Learning Environment down to Five Games
The Arcade Learning Environment (ALE) has become an essential benchmark for assessing the performance of reinforcement learning algorithms.
Beyond Bayes-optimality: meta-learning what you know you don't know
This is in contrast to risk-sensitive agents, which additionally exploit the higher-order moments of the return, and ambiguity-sensitive agents, which act differently when recognizing situations in which they lack knowledge.
Formal Algorithms for Transformers
This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results).
Neural Networks and the Chomsky Hierarchy
Reliable generalization lies at the heart of safe ML and AI.
Uniqueness and Complexity of Inverse MDP Models
In the traditional "forward" view, transition "matrix" p(s'|sa) and policy {\pi}(a|s) uniquely determine "everything": the whole dynamics p(as'a's"a"...|s), and with it, the action-conditional state process p(s's"...|saa'a"), the multi-step inverse models p(aa'a"...|ss^i), etc.
On the Role of Neural Collapse in Transfer Learning
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes.
Isotuning With Applications To Scale-Free Online Learning
We extend and combine several tools of the literature to design fast, adaptive, anytime and scale-free online learning algorithms.
Reducing Planning Complexity of General Reinforcement Learning with Non-Markovian Abstractions
A distinguishing feature of ESA is that it proves an upper bound of $O\left(\varepsilon^{-A} \cdot (1-\gamma)^{-2A}\right)$ on the number of states required for the surrogate MDP (where $A$ is the number of actions, $\gamma$ is the discount-factor, and $\varepsilon$ is the optimality-gap) which holds \emph{uniformly} for \emph{all} domains.
Shaking the foundations: delusions in sequence models for interaction and control
The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models such as transformers are being applied to a variety of domains.
Reinforcement Learning with Information-Theoretic Actuation
In this work we explore and formalize a contrasting view, namely that actions are best thought of as the output of a sequence of internal choices with respect to an action model.
Intelligence and Unambitiousness Using Algorithmic Information Theory
Algorithmic Information Theory has inspired intractable constructions of general intelligence (AGI), and undiscovered tractable approximations are likely feasible.
Fully General Online Imitation Learning
If we run an imitator, we probably want events to unfold similarly to the way they would have if the demonstrator had been acting the whole time.
Learning Curve Theory
Theoretical understanding of this phenomenon is largely lacking, except in finite-dimensional models for which error typically decreases with $n^{-1/2}$ or $n^{-1}$, where $n$ is the sample size.
Model-Free Counterfactual Credit Assignment
Credit assignment in reinforcement learning is the problem of measuring an action’s influence on future rewards.
Exact Reduction of Huge Action Spaces in General Reinforcement Learning
In this work we show how action-binarization in the non-MDP case can significantly improve Extreme State Aggregation (ESA) bounds.
Counterfactual Credit Assignment in Model-Free Reinforcement Learning
Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards.
A Combinatorial Perspective on Transfer Learning
Our main postulate is that the combination of task segmentation, modular learning and memory-based ensembling can give rise to generalization on an exponentially growing number of unseen tasks.
On Representing (Anti)Symmetric Functions
Permutation-invariant, -equivariant, and -covariant functions and anti-symmetric functions are important in quantum physics, computer vision, and other disciplines.
Logarithmic Pruning is All You Need
The Lottery Ticket Hypothesis is a conjecture that every large neural network contains a subnetwork that, when trained in isolation, achieves comparable performance to the large network.
Pessimism About Unknown Unknowns Inspires Conservatism
Our other main contribution is that the agent's policy's value approaches at least that of the mentor, while the probability of deferring to the mentor goes to 0.
Curiosity Killed or Incapacitated the Cat and the Asymptotically Optimal Agent
Much work in reinforcement learning uses an ergodicity assumption to avoid this problem.
Online Learning in Contextual Bandits using Gated Linear Networks
We introduce a new and completely online contextual bandit algorithm called Gated Linear Contextual Bandits (GLCB).
Gated Linear Networks
This paper presents a new family of backpropagation-free neural architectures, Gated Linear Networks (GLNs).
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective
Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding?
Fairness without Regret
A popular approach of achieving fairness in optimization problems is by constraining the solution space to "fair" solutions, which unfortunately typically reduces solution quality.
Asymptotically Unambitious Artificial General Intelligence
General intelligence, the ability to solve arbitrary solvable problems, is supposed by many to be artificially constructible.
Conditions on Features for Temporal Difference-Like Methods to Converge
The convergence of many reinforcement learning (RL) algorithms with linear function approximation has been investigated extensively but most proofs assume that these methods converge to a unique solution.
A Strongly Asymptotically Optimal Agent in General Environments
This is known as strong asymptotic optimality, and it was previously unknown whether it was possible for a policy to be strongly asymptotically optimal in the class of all computable probabilistic environments.
Performance Guarantees for Homomorphisms Beyond Markov Decision Processes
However, we show that near-optimal performance is sometimes guaranteed even if the homomorphism is non-Markovian.
AGI Safety Literature Review
The development of Artificial General Intelligence (AGI) promises to be a major event.
Count-Based Exploration in Feature Space for Reinforcement Learning
We present a new method for computing a generalised state visit-count, which allows the agent to estimate the uncertainty associated with any state.
Ranked #13 on
Atari Games
on Atari 2600 Montezuma's Revenge
Universal Reinforcement Learning Algorithms: Survey and Experiments
Many state-of-the-art reinforcement learning (RL) algorithms typically assume that the environment is an ergodic Markov Decision Process (MDP).
Reinforcement Learning with a Corrupted Reward Channel
Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards.
Generalised Discount Functions applied to a Monte-Carlo AImu Implementation
We have added to the GRL simulation platform AIXIjs the functionality to assign an agent arbitrary discount functions, and an environment which can be used to determine the effect of discounting on an agent's policy.
Free Lunch for Optimisation under the Universal Distribution
Function optimisation is a major challenge in computer science.
Death and Suicide in Universal Artificial Intelligence
Reinforcement learning (RL) is a general paradigm for studying intelligent behaviour, with applications ranging from artificial intelligence to psychology and economics.
Discriminative Hierarchical Rank Pooling for Activity Recognition
We present hierarchical rank pooling, a video sequence encoding method for activity recognition.
Avoiding Wireheading with Value Reinforcement Learning
Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward -- the so-called wireheading problem.
Self-Modification of Policy and Utility Function in Rational Agents
As we continue to create more and more intelligent agents, chances increase that they will learn about this ability.
Loss Bounds and Time Complexity for Speed Priors
On a polynomial time computable sequence our speed prior is computable in exponential time.
Thompson Sampling is Asymptotically Optimal in General Environments
We discuss a variant of Thompson sampling for nonparametric reinforcement learning in a countable classes of general stochastic environments.
On the Computability of AIXI
Solomonoff induction and the reinforcement learning agent AIXI are proposed answers to this question.
Bad Universal Priors and Notions of Optimality
A big open question of algorithmic information theory is the choice of the universal Turing machine (UTM).
A Topological Approach to Meta-heuristics: Analytical Results on the BFS vs. DFS Algorithm Selection Problem
In this paper we derive estimates for average BFS and DFS runtime.
Solomonoff Induction Violates Nicod's Criterion
Nicod's criterion states that observing a black raven is evidence for the hypothesis H that all ravens are black.
On the Computability of Solomonoff Induction and Knowledge-Seeking
Solomonoff induction is held as a gold standard for learning, but it is known to be incomputable.
Sequential Extensions of Causal and Evidential Decision Theory
Moving beyond the dualistic view in AI where agent and environment are separated incurs new challenges for decision making, as calculation of expected utility is no longer straightforward.
Compress and Control
This paper describes a new information-theoretic policy evaluation technique for reinforcement learning.
Indefinitely Oscillating Martingales
We construct a class of nonnegative martingale processes that oscillate indefinitely with high probability.
Robust Feature Selection by Mutual Information Distributions
Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables.
Offline to Online Conversion
We consider the problem of converting offline estimators into an online predictor or estimator with small extra regret.
Extreme State Aggregation Beyond MDPs
We consider a Reinforcement Learning setup where an agent interacts with an environment in observation-reward-action cycles without any (esp.\ MDP) assumptions on the environment.
Online Learning of k-CNF Boolean Functions
This paper revisits the problem of learning a k-CNF Boolean function from examples in the context of online learning under the logarithmic loss.
A Novel Illumination-Invariant Loss for Monocular 3D Pose Estimation
Our proposed method of registering a 3D model of a known object on a given 2D photo of the object has numerous advantages over existing methods.
The Sample-Complexity of General Reinforcement Learning
We present a new algorithm for general reinforcement learning where the true environment is known to belong to a finite class of N arbitrary models.
Concentration and Confidence for Discrete Bayesian Sequence Predictors
We prove tight high-probability bounds on the cumulative error, which is measured in terms of the Kullback-Leibler (KL) divergence.
Context Tree Switching
This paper describes the Context Tree Switching technique, a modification of Context Tree Weighting for the prediction of binary, stationary, n-Markov sources.
Information Theory Information Theory
Reinforcement Learning via AIXI Approximation
This paper introduces a principled approach for the design of a scalable general reinforcement learning agent.
General Reinforcement Learning
Open-Ended Question Answering
+3
Discrete MDL Predicts in Total Variation
The Minimum Description Length (MDL) principle selects the model that has the shortest code for data plus model.
A Monte Carlo AIXI Approximation
This paper introduces a principled approach for the design of a scalable general reinforcement learning agent.
General Reinforcement Learning
Open-Ended Question Answering
+3
Universal Intelligence: A Definition of Machine Intelligence
Finally, we survey the many other tests and definitions of intelligence that have been proposed for machines.
