Search Results for author:
Found 50 papers, 29 papers with code
Simple and Scalable Strategies to Continually Pre-train Large Language Models
In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks.
Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation
However, in many practical applications this technique is vulnerable to label distribution shifts, sometimes producing catastrophic failure.
Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
These breadcrumbs are constructed by subtracting the weights from a pre-trained model before and after fine-tuning, followed by a sparsification process that eliminates weight outliers and negligible perturbations.
Can We Learn Communication-Efficient Optimizers?
Although many variants of these approaches have been proposed, they can sometimes lag behind state-of-the-art adaptive optimizers for deep learning.
DragD3D: Vertex-based Editing for Realistic Mesh Deformations using 2D Diffusion Priors
Direct mesh editing and deformation are key components in the geometric modeling and animation pipeline.
Continual Pre-Training of Large Language Models: How to (re)warm your model?
We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule.
$\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning
Distributed training of Deep Learning models has been critical to many recent successes in the field.
Adversarial Attacks on the Interpretation of Neuron Activation Maximization
These inputs can be selected from a data set or obtained by optimization.
Can Forward Gradient Match Backpropagation?
Forward Gradients - the idea of using directional derivatives in forward differentiation mode - have recently been shown to be utilizable for neural network training while avoiding problems generally associated with backpropagation gradient computation, such as locking and memorization requirements.
Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning
In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes, to reduce communication costs multiple gradient steps are performed at each node prior to aggregation.
Simulated Annealing in Early Layers Leads to Better Generalization
Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers.
Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning
Class prototypes are evolved continually in the same latent space, enabling learning and prediction at any point.
Imitation from Observation With Bootstrapped Contrastive Learning
In this paper, we adopt a challenging, but more realistic problem formulation, learning control policies that operate on a learned latent space with access only to visual demonstrations of an expert completing a task.
Local Learning with Neuron Groups
Local learning is an approach to model-parallelism that removes the standard end-to-end learning setup and utilizes local objective functions to permit parallel learning amongst model components in a deep network.
Reliability of CKA as a Similarity Measure in Deep Learning
Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways.
Tackling Online One-Class Incremental Learning by Removing Negative Contrasts
Recent work studies the supervised online continual learning setting where a learner receives a stream of data whose class distribution changes over time.
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
Continual Learning research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks.
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
We present a technique for zero-shot generation of a 3D model using only a target text prompt.
New Insights on Reducing Abrupt Representation Change in Online Continual Learning
In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones.
Towards Scaling Difference Target Propagation by Learning Backprop Targets
As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks.
Gradient Masked Averaging for Federated Learning
A major challenge in federated learning is the heterogeneity of data across client, which can degrade the performance of standard FL algorithms.
Revisiting Learnable Affines for Batch Norm in Few-Shot Transfer Learning
Batch Normalization is a staple of computer vision models, including those employed in few-shot learning.
Exploring the Optimality of Tight-Frame Scattering Networks
The wavelet filters used in the scattering transform are typically selected to create a tight frame via a parameterized mother wavelet.
Parametric Scattering Networks
The wavelet scattering transform creates geometric invariants and deformation stability.
Learning Compositional Shape Priors for Few-Shot 3D Reconstruction
The impressive performance of deep convolutional neural networks in single-view 3D reconstruction suggests that these models perform non-trivial reasoning about the 3D structure of the output space.
Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning
With the use of a replay buffer we show that this approach can be extended to asynchronous settings, where modules can operate and continue to update with possibly large communication delays.
New Insights on Reducing Abrupt Representation Change in Online Continual Learning
In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones.
The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
A recent line of work showed that various forms of convolutional kernel methods can be competitive with standard supervised deep convolutional networks on datasets like CIFAR-10, obtaining accuracies in the range of 87-90% while being more amenable to theoretical analysis.
The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods.
A recent line of work showed that various forms of convolutional kernel methods can be competitive with standard supervised deep convolutional networks on datasets like CIFAR-10, obtaining accuracies in the range of 87-90% while being more amenable to theoretical analysis.
Generative Compositional Augmentations for Scene Graph Prediction
However, test images might contain zero- and few-shot compositions of objects and relationships, e. g. <cup, on, surfboard>.
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation
We show that such models can suffer the most in their ability to generalize to rare compositions, evaluating two different models on the Visual Genome dataset and its more recent, improved version, GQA.
A Simple and Scalable Shape Representation for 3D Reconstruction
Deep learning applied to the reconstruction of 3D shapes has seen growing interest.
Few-Shot Single-View 3-D Object Reconstruction with Compositional Priors
In this work we demonstrate experimentally that naive baselines do not apply when the goal is to learn to reconstruct novel objects using very few examples, and that in a \emph{few-shot} learning setting, the network must learn concepts that can be applied to new categories, avoiding rote memorization.
Online Continual Learning with Maximal Interfered Retrieval
Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks.
Online Learned Continual Compression with Adaptive Quantization Modules
We show how to use discrete auto-encoders to effectively address this challenge and introduce Adaptive Quantization Modules (AQM) to control variation in the compression ability of the module at any given stage of learning.
Online Learned Continual Compression with Stacked Quantization Modules
We first replace the episodic memory used in Experience Replay with SQM, leading to significant gains on standard continual learning benchmarks using a fixed memory budget.
VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering
The goal of this dataset is to assess question-answering performance from nearly-ideal navigation paths, while considering a much more complete variety of questions than current instantiations of the EQA task.
Online Continual Learning with Maximally Interfered Retrieval
Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks.
Decoupled Greedy Learning of CNNs
It is based on a greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification.
Greedy Layerwise Learning Can Scale to ImageNet
Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks.
Kymatio: Scattering Transforms in Python
The wavelet scattering transform is an invariant signal representation suitable for many signal processing and machine learning applications.
Blindfold Baselines for Embodied QA
We explore blindfold (question-only) baselines for Embodied Question Answering.
Shallow Learning For Deep Networks
Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks.
Compressing the Input for CNNs with the First-Order Scattering Transform
We study the first-order scattering transform as a candidate for reducing the signal processed by a convolutional neural network (CNN).
Scattering Networks for Hybrid Representation Learning
In particular, by working in scattering space, we achieve competitive results both for supervised and unsupervised learning tasks, while making progress towards constructing more interpretable CNNs.
Scaling the Scattering Transform: Deep Hybrid Networks
Combining scattering networks with a modern ResNet, we achieve a single-crop top 5 error of 11. 4% on imagenet ILSVRC2012, comparable to the Resnet-18 architecture, while utilizing only 10 layers.
Ranked #73 on
Image Classification
on STL-10
Fast Non-Parametric Tests of Relative Dependency and Similarity
The second test, called the relative test of similarity, is use to determine which of the two samples from arbitrary distributions is significantly closer to a reference sample of interest and the relative measure of similarity is based on the Maximum Mean Discrepancy (MMD).
Learning to Discover Sparse Graphical Models
Learning this function brings two benefits: it implicitly models the desired structure or sparsity properties to form suitable priors, and it can be tailored to the specific problem of edge structure discovery, rather than maximizing data likelihood.
Testing for Differences in Gaussian Graphical Models: Applications to Brain Connectivity
We characterize the uncertainty of differences with confidence intervals obtained using a parametric distribution on parameters of a sparse estimator.
A Test of Relative Similarity For Model Selection in Generative Models
Probabilistic generative models provide a powerful framework for representing data that avoids the expense of manual annotation typically needed by discriminative approaches.
