Search Results for author: Leon Bottou

Found 20 papers, 5 papers with code

Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need

1 code implementation ICCV 2023 Vivien Cabannes, Leon Bottou, Yann Lecun, Randall Balestriero

Third, it provides a proper active learning framework yielding low-cost solutions to annotate datasets, arguably bringing the gap between theory and practice of active learning that is based on simple-to-answer-by-non-experts queries of semantic relationships between inputs.

Active Learning Self-Supervised Learning

The Effects of Regularization and Data Augmentation are Class Dependent

no code implementations7 Apr 2022 Randall Balestriero, Leon Bottou, Yann Lecun

The optimal amount of DA or weight decay found from cross-validation leads to disastrous model performances on some classes e. g. on Imagenet with a resnet50, the "barn spider" classification test accuracy falls from $68\%$ to $46\%$ only by introducing random crop DA during training.

Data Augmentation

Match Prediction Using Learned History Embeddings

no code implementations29 Sep 2021 Maxwell Goldstein, Leon Bottou, Rob Fergus

Contemporary ranking systems that are based on win/loss history, such as Elo or TrueSkill represent each player using a scalar estimate of ability (plus variance, in the latter case).

Pseudo-Euclidean Attract-Repel Embeddings for Undirected Graphs

no code implementations17 Jun 2021 Alexander Peysakhovich, Anna Klimovskaia Susmel, Leon Bottou

Dot product embeddings take a graph and construct vectors for nodes such that dot products between two vectors give the strength of the edge.

Link Prediction Representation Learning

Linear unit-tests for invariance discovery

2 code implementations22 Feb 2021 Benjamin Aubin, Agnieszka Słowik, Martin Arjovsky, Leon Bottou, David Lopez-Paz

There is an increasing interest in algorithms to learn invariant correlations across training environments.

Out-of-Distribution Generalization

Scaling Laws for the Principled Design, Initialization, and Preconditioning of ReLU Networks

no code implementations ICLR 2020 Aaron Defazio, Leon Bottou

Abstract In this work, we describe a set of rules for the design and initialization of well-conditioned neural networks, guided by the goal of naturally balancing the diagonal blocks of the Hessian at the start of training.

Mean Replacement Pruning

no code implementations ICLR 2019 Utku Evci, Nicolas Le Roux, Pablo Castro, Leon Bottou

Finally, we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the stages of training.

AdaGrad stepsizes: Sharp convergence over nonconvex landscapes

1 code implementation5 Jun 2018 Rachel Ward, Xiaoxia Wu, Leon Bottou

Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule.

Stochastic Optimization

Geometrical Insights for Implicit Generative Modeling

no code implementations21 Dec 2017 Leon Bottou, Martin Arjovsky, David Lopez-Paz, Maxime Oquab

Learning algorithms for implicit generative models can optimize a variety of criteria that measure how the data distribution differs from the implicit model distribution, including the Wasserstein distance, the Energy distance, and the Maximum Mean Discrepancy criterion.

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

no code implementations ICLR 2018 Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, Leon Bottou

In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

no code implementations22 Nov 2016 Levent Sagun, Leon Bottou, Yann Lecun

We look at the eigenvalues of the Hessian of a loss function before and after training.

A Lower Bound for the Optimization of Finite Sums

no code implementations2 Oct 2014 Alekh Agarwal, Leon Bottou

This paper presents a lower bound for optimizing a finite sum of $n$ functions, where each function is $L$-smooth and the sum is $\mu$-strongly convex.

Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks

1 code implementation CVPR 2014 Maxime Oquab, Leon Bottou, Ivan Laptev, Josef Sivic

We show that despite differences in image statistics and tasks in the two datasets, the transferred representation leads to significantly improved results for object and action classification, outperforming the current state of the art on Pascal VOC 2007 and 2012 datasets.

Action Classification Action Localization +4

An efficient distributed learning algorithm based on effective local functional approximations

no code implementations31 Oct 2013 Dhruv Mahajan, Nikunj Agrawal, S. Sathiya Keerthi, S. Sundararajan, Leon Bottou

In this paper we give a novel approach to the distributed training of linear classifiers (involving smooth losses and L2 regularization) that is designed to reduce the total communication costs.

L2 Regularization

Para-active learning

no code implementations30 Oct 2013 Alekh Agarwal, Leon Bottou, Miroslav Dudik, John Langford

We leverage the same observation to build a generic strategy for parallelizing learning algorithms.

Active Learning

Natural Language Processing (almost) from Scratch

2 code implementations2 Mar 2011 Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling.

Chunking named-entity-recognition +4

From Machine Learning to Machine Reasoning

no code implementations9 Feb 2011 Leon Bottou

This observation suggests a conceptual continuity between algebraically rich inference systems, such as logical or probabilistic inference, and simple manipulations, such as the mere concatenation of trainable learning systems.

BIG-bench Machine Learning Language Modelling +2

Cannot find the paper you are looking for? You can Submit a new open access paper.