Search Results for author:
Found 25 papers, 10 papers with code
Robust Cross-Modal Knowledge Distillation for Unconstrained Videos
However, such semantic consistency from the synchronization is hard to guarantee in unconstrained videos, due to the irrelevant modality noise and differentiated semantic correlation.
Large-scale Knowledge Distillation with Elastic Heterogeneous Computing Resources
Although more layers and more parameters generally improve the accuracy of the models, such big models generally have high computational complexity and require big memory, which exceed the capacity of small devices for inference and incurs long training time.
Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
The advent of large-scale pre-trained language models has contributed greatly to the recent progress in natural language processing.
Deep Active Learning with Noise Stability
We provide theoretical analyses by leveraging the small Gaussian noise theory and demonstrate that our method favors a subset with large and diverse gradients.
Towards Inadequately Pre-trained Models in Transfer Learning
Based on the contradictory phenomenon between FE and FT that better feature extractor fails to be fine-tuned better accordingly, we conduct comprehensive analyses on features before softmax layer to provide insightful explanations.
Boosting Active Learning via Improving Test Performance
In this work, we explore such an impact by theoretically proving that selecting unlabeled data of higher gradient norm leads to a lower upper-bound of test loss, resulting in better test performance.
Noise Stability Regularization for Improving BERT Fine-tuning
The brittleness of this process is often reflected by the sensitivity to random seeds.
SMILE: Self-Distilled MIxup for Efficient Transfer LEarning
Performing mixup for transfer learning with pre-trained models however is not that simple, a high capacity pre-trained model with a large fully-connected (FC) layer could easily overfit to the target dataset even with samples-to-labels mixed up.
Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond
Then, to understand the interpretation results, we also survey the performance metrics for evaluating interpretation algorithms.
Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization that consists of two complementary components: Adaptive Knowledge Consistency (AKC) on the examples between the source and target model, and Adaptive Representation Consistency (ARC) on the target model between labeled and unlabeled examples.
Can We Use Gradient Norm as a Measure of Generalization Error for Model Selection in Practice?
The recent theoretical investigation (Li et al., 2020) on the upper bound of generalization error of deep neural networks (DNNs) demonstrates the potential of using the gradient norm as a measure that complements validation accuracy for model selection in practice.
Model information as an analysis tool in deep learning
We demonstrate using model information as a general analysis tool to gain insight into problems that arise in deep learning.
Temporal Relational Modeling with Self-Supervision for Action Segmentation
The main reason is that large number of nodes (i. e., video frames) makes GCNs hard to capture and model temporal relations in videos.
Ranked #23 on
Action Segmentation
on Breakfast
A Neural Network Approach Applied to Multi-Agent Optimal Control
Our approach is grid-free and scales efficiently to dimensions where grids become impractical or infeasible.
Optimization and Control
Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement
Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples.
Measuring Information Transfer in Neural Networks
We propose a practical measure of the generalizable information in a neural network model based on prequential coding, which we term Information Transfer ($L_{IT}$).
XMixup: Efficient Transfer Learning with Auxiliary Samples by Cross-domain Mixup
While the existing multitask learning algorithms need to run backpropagation over both the source and target datasets and usually consume a higher gradient complexity, XMixup transfers the knowledge from source to target tasks more efficiently: for every class of the target task, XMixup selects the auxiliary samples from the source dataset and augments training samples via the simple mixup strategy.
RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr
RIFLE brings meaningful updates to the weights of deep CNN layers and improves low-level feature learning, while the effects of randomization can be easily converged throughout the overall learning procedure.
OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport
On five high-dimensional density estimation and generative modeling tasks, OT-Flow performs competitively to state-of-the-art CNFs while on average requiring one-fourth of the number of weights with an 8x speedup in training time and 24x speedup in inference.
Pay Attention to Features, Transfer Learn Faster CNNs
Deep convolutional neural networks are now widely deployed in vision applications, but a limited size of training data can restrict their task performance.
COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization
Softening labels of training datasets with respect to data representations has been frequently used to improve the training of deep neural networks (DNNs).
Towards Making Deep Transfer Learning Never Hurt
The empirical results show that the proposed descent direction estimation strategy DTNH can always improve the performance of deep transfer learning tasks based on all above regularizers, even when transferring pre-trained weights from inappropriate networks.
Improving Adversarial Robustness via Attention and Adversarial Logit Pairing
Finally, we conduct extensive experiments using a wide range of datasets and the experiment results show that our AT+ALP achieves the state of the art defense performance.
An Empirical Study on Regularization of Deep Neural Networks by Local Rademacher Complexity
In this paper, we investigate the role of Rademacher complexity in improving generalization of DNNs and propose a novel regularizer rooted in Local Rademacher Complexity (LRC).
DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks
Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network.
