Search Results for author:
Found 34 papers, 17 papers with code
Understanding Factual Recall in Transformers via Associative Memories
We begin by proving that the storage capacities of both linear and MLP associative memories scale linearly with parameter count.
How Truncating Weights Improves Reasoning in Language Models
In addition to the ability to generate fluent text in various languages, large language models have been successful at tasks that involve basic forms of logical "reasoning" over their context.
Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task
Transformers have revolutionized machine learning across diverse domains, yet understanding their behavior remains crucial, particularly in high-stakes applications.
Level Set Teleportation: An Optimization Perspective
We study level set teleportation, an optimization sub-routine which seeks to accelerate gradient methods by maximizing the gradient norm on a level-set of the objective function.
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
We show that a key factor in this performance gap is the heavy-tailed class imbalance found in language tasks.
Learning Associative Memories with Gradient Descent
This work focuses on the training dynamics of one associative memory module storing outer products of token embeddings.
On Learning Gaussian Multi-index Models with Gradient Flow
We study gradient flow on the multi-index regression problem for high-dimensional Gaussian data.
Scaling Laws for Associative Memories
Learning arguably involves the discovery and memorization of abstract rules.
AstroCLIP: A Cross-Modal Foundation Model for Galaxies
These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification.
xVal: A Continuous Numerical Tokenization for Scientific Language Models
Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets.
Multiple Physics Pretraining for Physical Surrogate Models
We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers.
The SSL Interplay: Augmentations, Inductive Bias, and Generalization
Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision.
On minimal variations for unsupervised representation learning
Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks.
Learning Single-Index Models with Shallow Neural Networks
Single-index models are a class of functions given by an unknown univariate ``link'' function applied to an unknown one-dimensional projection of the input.
When does return-conditioned supervised learning work for offline reinforcement learning?
Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL).
On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes
To better understand these factors, we provide a precise study of the adversarial robustness in different scenarios, from initialization to the end of training in different regimes, as well as intermediate scenarios, where initialization still plays a role due to "lazy" training.
Efficient Kernel UCB for Contextual Bandits
While standard methods require a O(CT^3) complexity where T is the horizon and the constant C is related to optimizing the UCB rule, we propose an efficient contextual algorithm for large-scale problems.
Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning
Large-scale machine learning systems often involve data distributed across a collection of users.
On the Sample Complexity of Learning under Geometric Stability
Many supervised learning problems involve high-dimensional data such as images, text, or graphs.
Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
In the feature-learning regime, this dual formulation justifies using a two time-scale gradient ascent-descent (GDA) training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy.
On the Sample Complexity of Learning under Invariance and Geometric Stability
Many supervised learning problems involve high-dimensional data such as images, text, or graphs.
On the Universality of Graph Neural Networks on Large Random Graphs
In the large graph limit, GNNs are known to converge to certain "continuous" models known as c-GNNs, which directly enables a study of their approximation power on random graph models.
On Energy-Based Models with Overparametrized Shallow Neural Networks
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling.
Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
The empirical success of deep convolutional networks on tasks involving high-dimensional data such as images or audio suggests that they can efficiently approximate certain functions that are well-suited for such tasks.
Deep Equals Shallow for ReLU Networks in Kernel Regimes
Deep networks are often considered to be more expressive than shallow ones in terms of approximation.
Convergence and Stability of Graph Convolutional Networks on Large Random Graphs
We study properties of Graph Convolutional Networks (GCNs) by analyzing their behavior on standard models of random graphs, where nodes are represented by random latent variables and edges are drawn according to a similarity kernel.
Counterfactual Learning of Stochastic Policies with Continuous Actions: from Models to Offline Evaluation
Counterfactual reasoning from logged data has become increasingly important for many applications such as web advertising or healthcare.
On the Inductive Bias of Neural Tangent Kernels
State-of-the-art neural networks are heavily over-parameterized, making the optimization algorithm a crucial ingredient for learning predictive models with good generalization properties.
A Kernel Perspective for Regularizing Deep Neural Networks
We propose a new point of view for regularizing deep neural networks by using the norm of a reproducing kernel Hilbert space (RKHS).
A Contextual Bandit Bake-off
Contextual bandit algorithms are essential for solving many real-world interactive machine learning problems.
Invariance and Stability of Deep Convolutional Representations
In this paper, we study deep signal representations that are near-invariant to groups of transformations and stable to the action of diffeomorphisms without losing signal information.
Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite Sum Structure
Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions.
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations
The success of deep convolutional architectures is often attributed in part to their ability to learn multiscale and invariant representations of natural signals.
Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure
Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions.
