Search Results for author:
Found 13 papers, 8 papers with code
Diffusion-based Neural Network Weights Generation
By learning the distribution of a neural network on a variety pretrained models, our approach enables adaptive sampling weights for unseen datasets achieving faster convergence and reaching competitive performance.
Co-training and Co-distillation for Quality Improvement and Compression of Language Models
2) The enhanced performance of the larger model further boosts the performance of the smaller model.
Meta-prediction Model for Distillation-Aware NAS on Unseen Datasets
Previous DaNAS methods have mostly tackled the search for the neural architecture for fixed datasets and the teacher, which are not generalized well on a new task consisting of an unseen dataset and an unseen teacher, thus need to perform a costly search for any new combination of the datasets and the teachers.
A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models
Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance.
DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models
To tackle such limitations of existing NAS methods, we propose a paradigm shift from NAS to a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG.
A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search
The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization.
Hardware-adaptive Efficient Latency Prediction for NAS via Meta-Learning
To overcome such limitations, we propose Hardware-adaptive Efficient Latency Predictor (HELP), which formulates the device-specific latency estimation problem as a meta-learning problem, such that we can estimate the latency of a model's performance for a given task on an unseen device with a few samples.
Online Hyperparameter Meta-Learning with Hypergradient Distillation
Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters.
Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets
Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform human-designed networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets).
HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning
To overcome such limitations, we propose Hardware-adaptive Efficient Latency Predictor (HELP), which formulates the device-specific latency estimation problem as a meta-learning problem, such that we can estimate the latency of a model's performance for a given task on an unseen device with a few samples.
Task-Adaptive Neural Network Search with Meta-Contrastive Learning
To address such limitations, we introduce a novel problem of \emph{Neural Network Search} (NNS), whose goal is to search for the optimal pretrained network for a novel dataset and constraints (e. g. number of parameters), from a model zoo.
Learning to Generalize to Unseen Tasks with Bilevel Optimization
To tackle this issue, we propose a simple yet effective meta-learning framework for metricbased approaches, which we refer to as learning to generalize (L2G), that explicitly constrains the learning on a sampled classification task to reduce the classification error on a randomly sampled unseen classification task with a bilevel optimization scheme.
Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks
While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that the number of instances per task and class is fixed.
