Search Results for author:
Found 65 papers, 36 papers with code
Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training
Our results show that the infinite learning rate schedule remains effective at scale, surpassing repeated cosine decay for both MAE pre-training and zero-shot LM benchmarks.
Learning Versatile Optimizers on a Compute Diet
Learned optimization has emerged as a promising alternative to hand-crafted optimizers, with the potential to discover stronger learned update rules that enable faster, hyperparameter-free training of neural networks.
Non-Uniform Parameter-Wise Model Merging
Combining multiple machine learning models has long been a technique for enhancing performance, particularly in distributed settings.
Sketch-guided Cage-based 3D Gaussian Splatting Deformation
3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision.
Towards motion from video diffusion models
Specifically we propose to synthesize human motion by deforming an SMPL-X body representation guided by Score distillation sampling (SDS) calculated using a video diffusion model.
Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting
It has been recently shown that a powerful approach to combat spurious correlations is to re-train the last layer on a balanced validation dataset, isolating robust features for the predictor.
Accelerating Training with Neuron Interaction and Nowcasting Networks
Neural network training can be accelerated when a learnable update rule is used in lieu of classic adaptive optimizers (e. g. Adam).
Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis
We propose a new model merging algorithm, CCA Merge, which is based on Canonical Correlation Analysis and aims to maximize the correlations between linear combinations of the model features.
Controlling Forgetting with Test-Time Data in Continual Learning
Foundational vision-language models have shown impressive performance on various downstream tasks.
PETRA: Parallel End-to-end Training with Reversible Architectures
Reversible architectures have been shown to be capable of performing on par with their non-reversible architectures, being applied in deep learning for memory savings and generative modeling.
ACCO: Accumulate while you Communicate, Hiding Communications in Distributed LLM Training
Our method relies on a novel technique to mitigate the one-step delay inherent in parallel execution of gradient computations and communications, eliminating the need for warmup steps and aligning with the training dynamics of standard distributed optimization while converging faster in terms of wall-clock time.
From Feature Visualization to Visual Circuits: Effect of Adversarial Model Manipulation
Understanding the inner working functionality of large-scale deep neural networks is challenging yet crucial in several high-stakes applications.
$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
We extend $\mu$P theory to learned optimizers, treating the meta-training problem as finding the learned optimizer under $\mu$P.
WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
The performance of deep neural networks is enhanced by ensemble methods, which average the output of several models.
AdaFisher: Adaptive Second Order Optimization via Fisher Information
However, their practicality in training DNNs is still limited due to increased per-iteration computations compared to the first-order methods.
Simple and Scalable Strategies to Continually Pre-train Large Language Models
In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks.
Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation
However, in many practical applications this technique is vulnerable to label distribution shifts, sometimes producing catastrophic failure.
Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
These breadcrumbs are constructed by subtracting the weights from a pre-trained model before and after fine-tuning, followed by a sparsification process that eliminates weight outliers and negligible perturbations.
Can We Learn Communication-Efficient Optimizers?
Although many variants of these approaches have been proposed, they can sometimes lag behind state-of-the-art adaptive optimizers for deep learning.
DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priors
Thus, our deformation method achieves globally realistic shape deformation which is not restricted to any class of objects.
Continual Pre-Training of Large Language Models: How to (re)warm your model?
We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule.
$\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning
Distributed training of Deep Learning models has been critical to many recent successes in the field.
Can Forward Gradient Match Backpropagation?
Forward Gradients - the idea of using directional derivatives in forward differentiation mode - have recently been shown to be utilizable for neural network training while avoiding problems generally associated with backpropagation gradient computation, such as locking and memorization requirements.
Adversarial Attacks on the Interpretation of Neuron Activation Maximization
These inputs can be selected from a data set or obtained by optimization.
Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning
In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes, to reduce communication costs multiple gradient steps are performed at each node prior to aggregation.
Simulated Annealing in Early Layers Leads to Better Generalization
Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers.
Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning
Class prototypes are evolved continually in the same latent space, enabling learning and prediction at any point.
Imitation from Observation With Bootstrapped Contrastive Learning
In this paper, we adopt a challenging, but more realistic problem formulation, learning control policies that operate on a learned latent space with access only to visual demonstrations of an expert completing a task.
Local Learning with Neuron Groups
Local learning is an approach to model-parallelism that removes the standard end-to-end learning setup and utilizes local objective functions to permit parallel learning amongst model components in a deep network.
Reliability of CKA as a Similarity Measure in Deep Learning
Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways.
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
We present a technique for zero-shot generation of a 3D model using only a target text prompt.
Tackling Online One-Class Incremental Learning by Removing Negative Contrasts
Recent work studies the supervised online continual learning setting where a learner receives a stream of data whose class distribution changes over time.
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
Continual Learning research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks.
New Insights on Reducing Abrupt Representation Change in Online Continual Learning
In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones.
Towards Scaling Difference Target Propagation by Learning Backprop Targets
As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks.
Gradient Masked Averaging for Federated Learning
A major challenge in federated learning is the heterogeneity of data across client, which can degrade the performance of standard FL algorithms.
Revisiting Learnable Affines for Batch Norm in Few-Shot Transfer Learning
Batch Normalization is a staple of computer vision models, including those employed in few-shot learning.
Exploring the Optimality of Tight-Frame Scattering Networks
The wavelet filters used in the scattering transform are typically selected to create a tight frame via a parameterized mother wavelet.
Parametric Scattering Networks
The wavelet scattering transform creates geometric invariants and deformation stability.
Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning
With the use of a replay buffer we show that this approach can be extended to asynchronous settings, where modules can operate and continue to update with possibly large communication delays.
Learning Compositional Shape Priors for Few-Shot 3D Reconstruction
The impressive performance of deep convolutional neural networks in single-view 3D reconstruction suggests that these models perform non-trivial reasoning about the 3D structure of the output space.
New Insights on Reducing Abrupt Representation Change in Online Continual Learning
In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones.
The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
A recent line of work showed that various forms of convolutional kernel methods can be competitive with standard supervised deep convolutional networks on datasets like CIFAR-10, obtaining accuracies in the range of 87-90% while being more amenable to theoretical analysis.
The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods.
A recent line of work showed that various forms of convolutional kernel methods can be competitive with standard supervised deep convolutional networks on datasets like CIFAR-10, obtaining accuracies in the range of 87-90% while being more amenable to theoretical analysis.
Generative Compositional Augmentations for Scene Graph Prediction
However, test images might contain zero- and few-shot compositions of objects and relationships, e. g. <cup, on, surfboard>.
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation
We show that such models can suffer the most in their ability to generalize to rare compositions, evaluating two different models on the Visual Genome dataset and its more recent, improved version, GQA.
A Simple and Scalable Shape Representation for 3D Reconstruction
Deep learning applied to the reconstruction of 3D shapes has seen growing interest.
Few-Shot Single-View 3-D Object Reconstruction with Compositional Priors
In this work we demonstrate experimentally that naive baselines do not apply when the goal is to learn to reconstruct novel objects using very few examples, and that in a \emph{few-shot} learning setting, the network must learn concepts that can be applied to new categories, avoiding rote memorization.
Online Continual Learning with Maximal Interfered Retrieval
Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks.
Online Learned Continual Compression with Adaptive Quantization Modules
We show how to use discrete auto-encoders to effectively address this challenge and introduce Adaptive Quantization Modules (AQM) to control variation in the compression ability of the module at any given stage of learning.
Online Learned Continual Compression with Stacked Quantization Modules
We first replace the episodic memory used in Experience Replay with SQM, leading to significant gains on standard continual learning benchmarks using a fixed memory budget.
VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering
The goal of this dataset is to assess question-answering performance from nearly-ideal navigation paths, while considering a much more complete variety of questions than current instantiations of the EQA task.
Online Continual Learning with Maximally Interfered Retrieval
Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks.
Decoupled Greedy Learning of CNNs
It is based on a greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification.
Greedy Layerwise Learning Can Scale to ImageNet
Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks.
Kymatio: Scattering Transforms in Python
The wavelet scattering transform is an invariant signal representation suitable for many signal processing and machine learning applications.
Blindfold Baselines for Embodied QA
We explore blindfold (question-only) baselines for Embodied Question Answering.
Shallow Learning For Deep Networks
Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks.
Compressing the Input for CNNs with the First-Order Scattering Transform
We study the first-order scattering transform as a candidate for reducing the signal processed by a convolutional neural network (CNN).
Scattering Networks for Hybrid Representation Learning
In particular, by working in scattering space, we achieve competitive results both for supervised and unsupervised learning tasks, while making progress towards constructing more interpretable CNNs.
Scaling the Scattering Transform: Deep Hybrid Networks
Combining scattering networks with a modern ResNet, we achieve a single-crop top 5 error of 11. 4% on imagenet ILSVRC2012, comparable to the Resnet-18 architecture, while utilizing only 10 layers.
Ranked #72 on
Image Classification
on STL-10
Fast Non-Parametric Tests of Relative Dependency and Similarity
The second test, called the relative test of similarity, is use to determine which of the two samples from arbitrary distributions is significantly closer to a reference sample of interest and the relative measure of similarity is based on the Maximum Mean Discrepancy (MMD).
Learning to Discover Sparse Graphical Models
Learning this function brings two benefits: it implicitly models the desired structure or sparsity properties to form suitable priors, and it can be tailored to the specific problem of edge structure discovery, rather than maximizing data likelihood.
Testing for Differences in Gaussian Graphical Models: Applications to Brain Connectivity
We characterize the uncertainty of differences with confidence intervals obtained using a parametric distribution on parameters of a sparse estimator.
A Test of Relative Similarity For Model Selection in Generative Models
Probabilistic generative models provide a powerful framework for representing data that avoids the expense of manual annotation typically needed by discriminative approaches.
