Search Results for author:
Found 99 papers, 30 papers with code
Improved Sleeping Bandits with Stochastic Action Sets and Adversarial Rewards
The best existing efficient (i. e., polynomial-time) algorithms for this problem only guarantee a $O(T^{2/3})$ upper-bound on the regret.
Budgeted Online Influence Maximization
We introduce a new budgeted framework for online influence maximization, considering the total cost of an advertising campaign instead of the common cardinality constraint on a chosen influencer set.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Multi-task Language Understanding
on MMLU
Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
(b) When using an LLM to solve the test questions, we present it with the full list of skill labels and ask it to identify the skill needed.
Understanding the performance gap between online and offline alignment algorithms
However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF.
Human Alignment of Large Language Models through Online Preference Optimisation
Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm.
Generalized Preference Optimization: A Unified Approach to Offline Alignment
The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss.
Decoding-time Realignment of Language Models
Aligning language models with human preferences is crucial for reducing errors and biases in these models.
Demonstration-Regularized RL
In particular, we study the demonstration-regularized reinforcement learning that leverages the expert demonstrations by KL-regularization for a policy learned by behavior cloning.
A General Theoretical Paradigm to Understand Learning from Human Preferences
In particular we derive a new general objective called $\Psi$PO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations.
Local and adaptive mirror descents in extensive-form games
We study how to learn $\epsilon$-optimal strategies in zero-sum imperfect information games (IIG) with trajectory feedback.
Half-Hop: A graph upsampling approach for slowing down message passing
Message passing neural networks have shown a lot of success on graph-structured data.
Ranked #1 on
Node Classification
on AMZ Comp
DoMo-AC: Doubly Multi-step Off-policy Actor-Critic Algorithm
Multi-step learning applies lookahead over multiple time steps and has proved valuable in policy evaluation settings.
VA-learning as a more efficient alternative to Q-learning
In reinforcement learning, the advantage function is critical for policy improvement, but is often extracted from a learned Q-function.
Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice
Mirror descent value iteration (MDVI), an abstraction of Kullback-Leibler (KL) and entropy-regularized reinforcement learning (RL), has served as the basis for recent high-performing practical RL algorithms.
Unlocking the Power of Representations in Long-term Novelty-based Exploration
We introduce Robust Exploration via Clustering-based Online Density Estimation (RECODE), a non-parametric method for novelty-based exploration that estimates visitation counts for clusters of states based on their similarity in a chosen embedding space.
Sharp Deviations Bounds for Dirichlet Weighted Sums with Application to analysis of Bayesian algorithms
These bounds are based on a novel integral representation of the density of a weighted Dirichlet sum.
Fast Rates for Maximum Entropy Exploration
Finally, we apply developed regularization techniques to reduce sample complexity of visitation entropy maximization to $\widetilde{\mathcal{O}}(H^2SA/\varepsilon^2)$, yielding a statistical separation between maximum entropy exploration and reward-free exploration.
Adapting to game trees in zero-sum imperfect information games
Imperfect information games (IIG) are games in which each player only partially observes the current game state.
Understanding Self-Predictive Learning for Reinforcement Learning
We identify that a faster paced optimization of the predictor and semi-gradient updates on the representation, are crucial to preventing the representation collapse.
Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments
In this work, we study a natural solution derived from structural causal models of the world: Our key idea is to learn representations of the future that capture precisely the unpredictable aspects of each outcome -- which we use as additional input for predictions, such that intrinsic rewards only reflect the predictable aspects of world dynamics.
Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guarantees
We consider reinforcement learning in an environment modeled by an episodic, finite, stage-dependent Markov decision process of horizon $H$ with $S$ states, and $A$ actions.
BYOL-Explore: Exploration by Bootstrapped Prediction
We present BYOL-Explore, a conceptually simple yet general approach for curiosity-driven exploration in visually-complex environments.
KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal
In this work, we consider and analyze the sample complexity of model-free reinforcement learning with a generative model.
From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses
We propose the Bayes-UCBVI algorithm for reinforcement learning in tabular, stage-dependent, episodic Markov decision process: a natural extension of the Bayes-UCB algorithm by Kaufmann et al. (2012) for multi-armed bandits.
Marginalized Operators for Off-policy Reinforcement Learning
We show that the estimates for marginalized operators can be computed in a scalable way, which also generalizes prior results on marginalized importance sampling as special cases.
Retrieval-Augmented Reinforcement Learning
In this paper we explore an alternative paradigm in which we train a network to map a dataset of past experiences to optimal behavior.
Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Times
Computing a Gaussian process (GP) posterior has a computational cost cubical in the number of historical points.
Learning in two-player zero-sum partially observable Markov games with perfect recall
We study the problem of learning a Nash equilibrium (NE) in an extensive game with imperfect information (EGII) through self-play.
Adaptive Multi-Goal Exploration
We introduce a generic strategy for provably efficient multi-goal exploration.
Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
Our approach combines a generative modeling framework with an instance-specific alignment loss that tries to maximize the representational similarity between transformed views of the input (brain state).
Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions.
Density-Based Bonuses on Learned Representations for Reward-Free Exploration in Deep Reinforcement Learning
In this paper, we study the problem of representation learning and exploration in reinforcement learning.
Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall
We study the problem of learning a Nash equilibrium (NE) in an imperfect information game (IIG) through self-play.
Taylor Expansion of Discount Factors
In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective.
Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret
We study the problem of learning in the stochastic shortest path (SSP) setting, where an agent seeks to minimize the expected cost accumulated before reaching a goal state.
Broaden Your Views for Self-Supervised Video Learning
Most successful self-supervised learning methods are trained to align the representations of two independent views from the data.
Bootstrapped Representation Learning on Graphs
Current state-of-the-art self-supervised learning methods for graph neural networks are based on contrastive learning.
UCB Momentum Q-learning: Correcting the bias without forgetting
We propose UCBMQ, Upper Confidence Bound Momentum Q-learning, a new algorithm for reinforcement learning in tabular and possibly stage-dependent, episodic Markov decision process.
Revisiting Peng's Q($λ$) for Modern Reinforcement Learning
These results indicate that Peng's Q($\lambda$), which was thought to be unsafe, is a theoretically-sound and practically effective algorithm.
Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
State-of-the-art methods for self-supervised learning (SSL) build representations by maximizing the similarity between different transformed "views" of a sample.
Large-Scale Representation Learning on Graphs via Bootstrapping
To address these challenges, we introduce Bootstrapped Graph Latents (BGRL) - a graph representation learning method that learns by predicting alternative augmentations of the input.
Geometric Entropic Exploration
Exploration is essential for solving complex Reinforcement Learning (RL) tasks.
On the Approximation Relationship between Optimizing Ratio of Submodular (RS) and Difference of Submodular (DS) Functions
We demonstrate that from an algorithm guaranteeing an approximation factor for the ratio of submodular (RS) optimization problem, we can build another algorithm having a different kind of approximation guarantee -- weaker than the classical one -- for the difference of submodular (DS) optimization problem, and vice versa.
Data Structures and Algorithms
Improved Sample Complexity for Incremental Autonomous Exploration in MDPs
We investigate the exploration of an unknown environment when no reward function is provided.
Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view.
Sampling from a k-DPP without looking at all items
Determinantal point processes (DPPs) are a useful probabilistic model for selecting a small diverse subset out of a large collection of items, with applications in summarization, recommendation, stochastic optimization, experimental design and more.
Game Plan: What AI can do for Football, and What Football can do for AI
The rapid progress in artificial intelligence (AI) and machine learning has opened unprecedented analytics possibilities in various team and individual sports, including baseball, basketball, and tennis.
BYOL works even without batch statistics
Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for image representation.
Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisited
In this paper, we propose new problem-independent lower bounds on the sample complexity and regret in episodic MDPs, with a particular focus on the non-stationary case in which the transition kernel is allowed to change in each stage of the episode.
Fast active learning for pure exploration in reinforcement learning
Realistic environments often provide agents with very limited feedback.
Monte-Carlo Tree Search as Regularized Policy Optimization
The combination of Monte-Carlo tree search (MCTS) with deep reinforcement learning has led to significant advances in artificial intelligence.
A Provably Efficient Sample Collection Strategy for Reinforcement Learning
One of the challenges in online reinforcement learning (RL) is that the agent needs to trade off the exploration of the environment and the exploitation of the samples to optimize its behavior.
A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spaces
In this work, we propose KeRNS: an algorithm for episodic reinforcement learning in non-stationary Markov Decision Processes (MDPs) whose state-action set is endowed with a metric.
Gamification of Pure Exploration for Linear Bandits
We investigate an active pure-exploration setting, that includes best-arm identification, in the context of linear stochastic bandits.
Sampling from a $k$-DPP without looking at all items
Determinantal point processes (DPPs) are a useful probabilistic model for selecting a small diverse subset out of a large collection of items, with applications in summarization, stochastic optimization, active learning and more.
Stochastic bandits with arm-dependent delays
Significant work has been recently dedicated to the stochastic delayed bandit setting because of its relevance in applications.
Bootstrap your own latent: A new approach to self-supervised Learning
From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view.
Ranked #2 on
Self-Supervised Person Re-Identification
on SYSU-30k
Adaptive Reward-Free Exploration
Reward-free exploration is a reinforcement learning setting studied by Jin et al. (2020), who address it by running several algorithms with regret guarantees in parallel.
Statistical Efficiency of Thompson Sampling for Combinatorial Semi-Bandits
In CMAB, the question of the existence of an efficient policy with an optimal asymptotic regret (up to a factor poly-logarithmic with the action size) is still open for many families of distributions, including mutually independent outcomes, and more generally the multivariate sub-Gaussian family.
Planning in Markov Decision Processes with Gap-Dependent Sample Complexity
We propose MDP-GapE, a new trajectory-based Monte-Carlo Tree Search algorithm for planning in a Markov Decision Process in which transitions have a finite support.
Improved Sleeping Bandits with Stochastic Actions Sets and Adversarial Rewards
We then study the most general version of the problem where at each round available sets are generated from some unknown arbitrary distribution (i. e., without the independence assumption) and propose an efficient algorithm with $O(\sqrt {2^K T})$ regret guarantee.
Kernel-Based Reinforcement Learning: A Finite-Time Analysis
We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning problems whose state-action space is endowed with a metric.
Taylor Expansion Policy Optimization
In this work, we investigate the application of Taylor expansions in reinforcement learning.
Near-linear Time Gaussian Process Optimization with Adaptive Batching and Resparsification
Gaussian processes (GP) are one of the most successful frameworks to model uncertainty.
No-Regret Exploration in Goal-Oriented Reinforcement Learning
Many popular reinforcement learning problems (e. g., navigation in a maze, some Atari games, mountain car) are instances of the episodic setting under its stochastic shortest path (SSP) formulation, where an agent has to achieve a goal state while minimizing the cumulative cost.
On two ways to use determinantal point processes for Monte Carlo integration
In the absence of DPP machinery to derive an efficient sampler and analyze their estimator, the idea of Monte Carlo integration with DPPs was stored in the cellar of numerical integration.
Planning in entropy-regularized Markov decision processes and games
We propose SmoothCruiser, a new planning algorithm for estimating the value function in entropy-regularized Markov decision processes and two-player games, given a generative model of the SmoothCruiser.
Fixed-Confidence Guarantees for Bayesian Best-Arm Identification
We investigate and provide new insights on the sampling rule called Top-Two Thompson Sampling (TTTS).
Derivative-Free & Order-Robust Optimisation
In this paper, we formalise order-robust optimisation as an instance of online learning minimising simple regret, and propose Vroom, a zero'th order optimisation algorithm capable of achieving vanishing regret in non-stationary environments, while recovering favorable rates under stochastic reward-generating processes.
Multiagent Evaluation under Incomplete Information
This paper investigates the evaluation of learned multiagent strategies in the incomplete information setting, which plays a critical role in ranking and training of agents.
Online A-Optimal Design and Active Linear Regression
By trying to minimize the $\ell^2$-loss $\mathbb{E} [\lVert\hat{\beta}-\beta^{\star}\rVert^2]$ the decision maker is actually minimizing the trace of the covariance matrix of the problem, which corresponds then to online A-optimal design.
Exact sampling of determinantal point processes with sublinear time preprocessing
For this purpose, we propose a new algorithm which, given access to $\mathbf{L}$, samples exactly from a determinantal point process while satisfying the following two properties: (1) its preprocessing cost is $n \cdot \text{poly}(k)$, i. e., sublinear in the size of $\mathbf{L}$, and (2) its sampling cost is $\text{poly}(k)$, i. e., independent of the size of $\mathbf{L}$.
Gaussian Process Optimization with Adaptive Sketching: Scalable and No Regret
Moreover, we show that our procedure selects at most $\tilde{O}(d_{eff})$ points, where $d_{eff}$ is the effective dimension of the explored space, which is typically much smaller than both $d$ and $t$.
Exploiting Structure of Uncertainty for Efficient Matroid Semi-Bandits
We improve the efficiency of algorithms for stochastic \emph{combinatorial semi-bandits}.
Optimistic optimization of a Brownian
Given $W$, our goal is to return an $\epsilon$-approximation of its maximum using the smallest possible number of function evaluations, the sample complexity of the algorithm.
Rotting bandits are not harder than stochastic ones
In stochastic multi-armed bandits, the reward distribution of each arm is assumed to be stationary.
A simple parameter-free and adaptive approach to optimization under a minimal local smoothness assumption
The difficulty of optimization is measured in terms of 1) the amount of \emph{noise} $b$ of the function evaluation and 2) the local smoothness, $d$, of the function.
Compressing the Input for CNNs with the First-Order Scattering Transform
We study the first-order scattering transform as a candidate for reducing the signal processed by a convolutional neural network (CNN).
DPPy: Sampling DPPs with Python
Determinantal point processes (DPPs) are specific probability distributions over clouds of points that are used as models and computational tools across physics, probability, statistics, and more recently machine learning.
Improved large-scale graph learning through ridge spectral sparsification
By constructing a spectrally-similar graph, we are able to bound the error induced by the sparsification for a variety of downstream tasks (e. g., SSL).
Finding the bandit in a graph: Sequential search-and-stop
We consider the problem where an agent wants to find a hidden object that is randomly located in some vertex of a directed acyclic graph (DAG) according to a fixed but possibly unknown distribution.
Distributed Adaptive Sampling for Kernel Matrix Approximation
In this paper, we introduce SQUEAK, a new algorithm for kernel approximation based on RLS sampling that sequentially processes the dataset, storing a dictionary which creates accurate kernel matrix approximations with a number of points that only depends on the effective dimension $d_{eff}(\gamma)$ of the dataset.
Efficient Second-Order Online Kernel Learning with Adaptive Embedding
The embedded space is continuously updated to guarantee that the embedding remains accurate, and we show that the per-step cost only grows with the effective dimension of the problem and not with $T$.
Second-Order Kernel Online Convex Optimization with Adaptive Sketching
First-order KOCO methods such as functional gradient descent require only $\mathcal{O}(t)$ time and space per iteration, and, when the only information on the losses is their convexity, achieve a minimax optimal $\mathcal{O}(\sqrt{T})$ regret.
Zonotope hit-and-run for efficient sampling from projection DPPs
Previous theoretical results yield a fast mixing time of our chain when targeting a distribution that is close to a projection DPP, but not a DPP in general.
Blazing the trails before beating the path: Sample-efficient Monte-Carlo planning
We study the sampling-based planning problem in Markov decision processes (MDPs) that we can access only through a generative model, usually referred to as Monte-Carlo planning.
Analysis of Kelner and Levin graph sparsification algorithm for a streaming setting
We derive a new proof to show that the incremental resparsification algorithm proposed by Kelner and Levin (2013) produces a spectral sparsifier in high probability.
Online Influence Maximization under Independent Cascade Model with Semi-Bandit Feedback
Specifically, we aim to learn the set of "best influencers" in a social network online while repeatedly interacting with it.
Incremental Spectral Sparsification for Large-Scale Graph-Based Semi-Supervised Learning
While the harmonic function solution performs well in many semi-supervised learning (SSL) tasks, it is known to scale poorly with the number of samples.
Black-box optimization of noisy functions with unknown smoothness
We study the problem of black-box optimization of a function $f$ of any dimension, given function evaluations perturbed by noise.
Cheap Bandits
We consider stochastic sequential learning problems where the learner can observe the \textit{average reward of several actions}.
Simple regret for infinitely many armed bandits
As in the cumulative regret setting of infinitely many armed bandits, the rate of the simple regret will depend on a parameter $\beta$ characterizing the distribution of the near-optimal arms.
Extreme bandits
In many areas of medicine, security, and life sciences, we want to allocate limited resources to different sources in order to detect extreme values.
Efficient learning by implicit exploration in bandit problems with side observations
As the predictions of our first algorithm cannot be always computed efficiently in this setting, we propose another algorithm with similar properties and with the benefit of always being computationally efficient, at the price of a slightly more complicated tuning mechanism.
Online combinatorial optimization with stochastic decision sets and adversarial losses
Most work on sequential learning assumes a fixed set of actions that are available all the time.
Learning to Act Greedily: Polymatroid Semi-Bandits
Many important optimization problems, such as the minimum spanning tree and minimum-cost flow, can be solved optimally by a greedy method.
Finite-Time Analysis of Kernelised Contextual Bandits
For contextual bandits, the related algorithm GP-UCB turns out to be a special case of our algorithm, and our finite-time analysis improves the regret bound of GP-UCB for the agnostic case, both in the terms of the kernel-dependent quantity and the RKHS norm of the reward function.
