Search Results for author: Angelos Katharopoulos

Found 10 papers, 6 papers with code

Specialized Language Models with Cheap Inference from Limited Domain Data

no code implementations2 Feb 2024 David Grangier, Angelos Katharopoulos, Pierre Ablin, Awni Hannun

Large language models have emerged as a versatile tool but are challenging to apply to tasks lacking large inference budgets and large in-domain training sets.

 Ranked #1 on Language Modelling on The Pile (Test perplexity metric)

Language Modelling

Controllable Music Production with Diffusion Models and Guidance Gradients

no code implementations1 Nov 2023 Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, Tom Nickson

We demonstrate how conditional generation from diffusion models can be used to tackle a variety of realistic tasks in the production of music in 44. 1kHz stereo audio with sampling-time guidance.

Masked Autoencoding Does Not Help Natural Language Supervision at Scale

no code implementations CVPR 2023 Floris Weers, Vaishaal Shankar, Angelos Katharopoulos, Yinfei Yang, Tom Gunter

Self supervision and natural language supervision have emerged as two exciting ways to train general purpose image encoders which excel at a variety of downstream tasks.

Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks

1 code implementation CVPR 2021 Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, Sanja Fidler

The INN allows us to compute the inverse mapping of the homeomorphism, which in turn, enables the efficient computation of both the implicit surface function of a primitive and its mesh, without any additional post-processing.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

6 code implementations ICML 2020 Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences.

D4RL Language Modelling +1

Processing Megapixel Images with Deep Attention-Sampling Models

2 code implementations3 May 2019 Angelos Katharopoulos, François Fleuret

We show that sampling from the attention distribution results in an unbiased estimator of the full model with minimal variance, and we derive an unbiased estimator of the gradient that we use to train our model end-to-end with a normal SGD procedure.

Deep Attention

Not All Samples Are Created Equal: Deep Learning with Importance Sampling

2 code implementations ICML 2018 Angelos Katharopoulos, François Fleuret

Deep neural network training spends most of the computation on examples that are properly handled, and could be ignored.

Image Classification

Learning Local Feature Aggregation Functions with Backpropagation

no code implementations26 Jun 2017 Angelos Katharopoulos, Despoina Paschalidou, Christos Diou, Anastasios Delopoulos

This paper introduces a family of local feature aggregation functions and a novel method to estimate their parameters, such that they generate optimal representations for classification (or any task that can be expressed as a cost function minimization problem).

General Classification

Biased Importance Sampling for Deep Neural Network Training

1 code implementation31 May 2017 Angelos Katharopoulos, François Fleuret

Importance sampling has been successfully used to accelerate stochastic optimization in many convex problems.

Image Classification Language Modelling +1

Cannot find the paper you are looking for? You can Submit a new open access paper.