Search Results for author: Xingjian Li

Found 25 papers, 10 papers with code

Robust Cross-Modal Knowledge Distillation for Unconstrained Videos

1 code implementation16 Apr 2023 Wenke Xia, Xingjian Li, Andong Deng, Haoyi Xiong, Dejing Dou, Di Hu

However, such semantic consistency from the synchronization is hard to guarantee in unconstrained videos, due to the irrelevant modality noise and differentiated semantic correlation.

Action Recognition Audio Tagging +3

Large-scale Knowledge Distillation with Elastic Heterogeneous Computing Resources

1 code implementation14 Jul 2022 Ji Liu, daxiang dong, Xi Wang, An Qin, Xingjian Li, Patrick Valduriez, Dejing Dou, dianhai yu

Although more layers and more parameters generally improve the accuracy of the models, such big models generally have high computational complexity and require big memory, which exceed the capacity of small devices for inference and incurs long training time.

Knowledge Distillation

Fine-tuning Pre-trained Language Models with Noise Stability Regularization

no code implementations12 Jun 2022 Hang Hua, Xingjian Li, Dejing Dou, Cheng-Zhong Xu, Jiebo Luo

The advent of large-scale pre-trained language models has contributed greatly to the recent progress in natural language processing.

Domain Generalization Language Modelling +3

Towards Inadequately Pre-trained Models in Transfer Learning

no code implementations9 Mar 2022 Andong Deng, Xingjian Li, Di Hu, Tianyang Wang, Haoyi Xiong, Chengzhong Xu

Based on the contradictory phenomenon between FE and FT that better feature extractor fails to be fine-tuned better accordingly, we conduct comprehensive analyses on features before softmax layer to provide insightful explanations.

Transfer Learning

Boosting Active Learning via Improving Test Performance

1 code implementation10 Dec 2021 Tianyang Wang, Xingjian Li, Pengkun Yang, Guosheng Hu, Xiangrui Zeng, Siyu Huang, Cheng-Zhong Xu, Min Xu

In this work, we explore such an impact by theoretically proving that selecting unlabeled data of higher gradient norm leads to a lower upper-bound of test loss, resulting in better test performance.

Active Learning Electron Tomography +2

SMILE: Self-Distilled MIxup for Efficient Transfer LEarning

no code implementations25 Mar 2021 Xingjian Li, Haoyi Xiong, Chengzhong Xu, Dejing Dou

Performing mixup for transfer learning with pre-trained models however is not that simple, a high capacity pre-trained model with a large fully-connected (FC) layer could easily overfit to the target dataset even with samples-to-labels mixed up.

Transfer Learning

Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond

1 code implementation19 Mar 2021 Xuhong LI, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Ji Liu, Jiang Bian, Dejing Dou

Then, to understand the interpretation results, we also survey the performance metrics for evaluating interpretation algorithms.

Adversarial Robustness

Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

1 code implementation CVPR 2021 Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, Dejing Dou

To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization that consists of two complementary components: Adaptive Knowledge Consistency (AKC) on the examples between the source and target model, and Adaptive Representation Consistency (ARC) on the target model between labeled and unlabeled examples.

Pseudo Label Transfer Learning

Model information as an analysis tool in deep learning

no code implementations1 Jan 2021 Xiao Zhang, Di Hu, Xingjian Li, Dejing Dou, Ji Wu

We demonstrate using model information as a general analysis tool to gain insight into problems that arise in deep learning.

Can We Use Gradient Norm as a Measure of Generalization Error for Model Selection in Practice?

no code implementations1 Jan 2021 Haozhe An, Haoyi Xiong, Xuhong LI, Xingjian Li, Dejing Dou, Zhanxing Zhu

The recent theoretical investigation (Li et al., 2020) on the upper bound of generalization error of deep neural networks (DNNs) demonstrates the potential of using the gradient norm as a measure that complements validation accuracy for model selection in practice.

Model Selection

Temporal Relational Modeling with Self-Supervision for Action Segmentation

1 code implementation14 Dec 2020 Dong Wang, Di Hu, Xingjian Li, Dejing Dou

The main reason is that large number of nodes (i. e., video frames) makes GCNs hard to capture and model temporal relations in videos.

Action Recognition Action Segmentation +1

A Neural Network Approach Applied to Multi-Agent Optimal Control

1 code implementation9 Nov 2020 Derek Onken, Levon Nurbekyan, Xingjian Li, Samy Wu Fung, Stanley Osher, Lars Ruthotto

Our approach is grid-free and scales efficiently to dimensions where grids become impractical or infeasible.

Optimization and Control

Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement

no code implementations16 Oct 2020 Xingjian Li, Di Hu, Xuhong LI, Haoyi Xiong, Zhi Ye, Zhipeng Wang, Chengzhong Xu, Dejing Dou

Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples.

Disentanglement Transfer Learning

Measuring Information Transfer in Neural Networks

no code implementations16 Sep 2020 Xiao Zhang, Xingjian Li, Dejing Dou, Ji Wu

We propose a practical measure of the generalizable information in a neural network model based on prequential coding, which we term Information Transfer ($L_{IT}$).

Continual Learning Transfer Learning

XMixup: Efficient Transfer Learning with Auxiliary Samples by Cross-domain Mixup

no code implementations20 Jul 2020 Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhong Xu, Dejing Dou

While the existing multitask learning algorithms need to run backpropagation over both the source and target datasets and usually consume a higher gradient complexity, XMixup transfers the knowledge from source to target tasks more efficiently: for every class of the target task, XMixup selects the auxiliary samples from the source dataset and augments training samples via the simple mixup strategy.

Transfer Learning

RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr

1 code implementation ICML 2020 Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhong Xu, Dejing Dou

RIFLE brings meaningful updates to the weights of deep CNN layers and improves low-level feature learning, while the effects of randomization can be easily converged throughout the overall learning procedure.

Transfer Learning

OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport

1 code implementation29 May 2020 Derek Onken, Samy Wu Fung, Xingjian Li, Lars Ruthotto

On five high-dimensional density estimation and generative modeling tasks, OT-Flow performs competitively to state-of-the-art CNFs while on average requiring one-fourth of the number of weights with an 8x speedup in training time and 24x speedup in inference.

Density Estimation

Pay Attention to Features, Transfer Learn Faster CNNs

no code implementations ICLR 2020 Kafeng Wang, Xitong Gao, Yiren Zhao, Xingjian Li, Dejing Dou, Cheng-Zhong Xu

Deep convolutional neural networks are now widely deployed in vision applications, but a limited size of training data can restrict their task performance.

Transfer Learning

COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization

no code implementations26 Apr 2020 Xingjian Li, Haoyi Xiong, Haozhe An, Dejing Dou, Chengzhong Xu

Softening labels of training datasets with respect to data representations has been frequently used to improve the training of deep neural networks (DNNs).

General Classification

Towards Making Deep Transfer Learning Never Hurt

no code implementations18 Nov 2019 Ruosi Wan, Haoyi Xiong, Xingjian Li, Zhanxing Zhu, Jun Huan

The empirical results show that the proposed descent direction estimation strategy DTNH can always improve the performance of deep transfer learning tasks based on all above regularizers, even when transferring pre-trained weights from inappropriate networks.

Knowledge Distillation Transfer Learning

Improving Adversarial Robustness via Attention and Adversarial Logit Pairing

no code implementations23 Aug 2019 Dou Goodman, Xingjian Li, Ji Liu, Dejing Dou, Tao Wei

Finally, we conduct extensive experiments using a wide range of datasets and the experiment results show that our AT+ALP achieves the state of the art defense performance.

Adversarial Robustness

An Empirical Study on Regularization of Deep Neural Networks by Local Rademacher Complexity

no code implementations3 Feb 2019 Yingzhen Yang, Jiahui Yu, Xingjian Li, Jun Huan, Thomas S. Huang

In this paper, we investigate the role of Rademacher complexity in improving generalization of DNNs and propose a novel regularizer rooted in Local Rademacher Complexity (LRC).

Neural Architecture Search

Cannot find the paper you are looking for? You can Submit a new open access paper.