Search Results for author: Dongsheng Li

Found 140 papers, 46 papers with code

Social Bot-Aware Graph Neural Network for Early Rumor Detection

1 code implementation COLING 2022 Zhen Huang, Zhilong Lv, Xiaoyun Han, Binyang Li, Menglong Lu, Dongsheng Li

SBAG firstly pre-trains a multi-layer perception network to capture social bot features, and then constructs multiple graph neural networks by embedding the features to model the early propagation of posts, which is further used to detect rumors.

PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction

1 code implementation1 Dec 2023 Lei Guan, Dongsheng Li, Jiye Liang, Wenjian Wang, Xicheng Lu

The key insight of our proposal is that we employ a weight prediction strategy in the forward pass to ensure that each mini-batch uses consistent and staleness-free weights to compute the forward pass.

Image Classification Machine Translation +1

AdaMedGraph: Adaboosting Graph Neural Networks for Personalized Medicine

no code implementations24 Nov 2023 Jie Lian, Xufang Luo, Caihua Shan, Dongqi Han, Varut Vardhanabhuti, Dongsheng Li

However, selecting the appropriate edge feature to define patient similarity and construct the graph is challenging, given that each patient is depicted by high-dimensional features from diverse sources.

Unified Medical Image Pre-training in Language-Guided Common Semantic Space

no code implementations24 Nov 2023 Xiaoxuan He, Yifan Yang, Xinyang Jiang, Xufang Luo, Haoji Hu, Siyun Zhao, Dongsheng Li, Yuqing Yang, Lili Qiu

To overcome the aforementioned challenges, we propose an Unified Medical Image Pre-training framework, namely UniMedI, which utilizes diagnostic reports as common semantic space to create unified representations for diverse modalities of medical images (especially for 2D and 3D images).

TVT: Training-Free Vision Transformer Search on Tiny Datasets

no code implementations24 Nov 2023 Zimian Wei, Hengyue Pan, Lujun Li, Peijie Dong, Zhiliang Tian, Xin Niu, Dongsheng Li

In this paper, for the first time, we investigate how to search in a training-free manner with the help of teacher models and devise an effective Training-free ViT (TVT) search framework.

Online Video Quality Enhancement with Spatial-Temporal Look-up Tables

no code implementations22 Nov 2023 Zefan Qu, Xinyang Jiang, Yifan Yang, Dongsheng Li, Cairong Zhao

To the best of our knowledge, we are the first to exploit the LUT structure to extract temporal information in video tasks.

A Comprehensive Summarization and Evaluation of Feature Refinement Modules for CTR Prediction

1 code implementation8 Nov 2023 Fangye Wang, Hansu Gu, Dongsheng Li, Tun Lu, Peng Zhang, Li Shang, Ning Gu

In addition, we present a new architecture of assigning independent FR modules to separate sub-networks for parallel CTR models, as opposed to the conventional method of inserting a shared FR module on top of the embedding layer.

Benchmarking Click-Through Rate Prediction

Label Propagation for Graph Label Noise

no code implementations25 Oct 2023 Yao Cheng, Caihua Shan, Yifei Shen, Xiang Li, Siqiang Luo, Dongsheng Li

In this paper, we study graph label noise in the context of arbitrary heterophily, with the aim of rectifying noisy labels and assigning labels to previously unlabeled nodes.

Denoising Node Classification

Rethinking SIGN Training: Provable Nonconvex Acceleration without First- and Second-Order Gradient Lipschitz

no code implementations23 Oct 2023 Tao Sun, Congliang Chen, Peng Qiao, Li Shen, Xinwang Liu, Dongsheng Li

Sign-based stochastic methods have gained attention due to their ability to achieve robust performance despite using only the sign information for parameter updates.

Learning To Teach Large Language Models Logical Reasoning

1 code implementation13 Oct 2023 Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang, Dongsheng Li

Large language models (LLMs) have gained enormous attention from both academia and industry, due to their exceptional ability in language generation and extremely powerful generalization.

counterfactual Event Relation Extraction +3

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

1 code implementation10 Oct 2023 Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu

Inspired by these findings, we propose LongLLMLingua for prompt compression towards improving LLMs' perception of the key information to simultaneously address the three challenges.

Code Completion Few-Shot Learning

GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence

no code implementations9 Oct 2023 Zhihua Wen, Zhiliang Tian, Wei Wu, Yuxin Yang, Yanqi Shi, Zhen Huang, Dongsheng Li

Finally, we select the most fitting chains of evidence from the evidence forest and integrate them into the generated story, thereby enhancing the narrative's complexity and credibility.

Retrieval Story Generation

Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent

no code implementations18 Aug 2023 Xiaoge Deng, Li Shen, Shengwei Li, Tao Sun, Dongsheng Li, DaCheng Tao

Stochastic gradient descent (SGD) performed in an asynchronous manner plays a crucial role in training large-scale machine learning models.

AutoSeqRec: Autoencoder for Efficient Sequential Recommendation

1 code implementation14 Aug 2023 Sijia Liu, Jiahao Liu, Hansu Gu, Dongsheng Li, Tun Lu, Peng Zhang, Ning Gu

Sequential recommendation demonstrates the capability to recommend items by modeling the sequential behavior of users.

Collaborative Filtering Sequential Recommendation

DaMSTF: Domain Adversarial Learning Enhanced Meta Self-Training for Domain Adaptation

no code implementations5 Aug 2023 Menglong Lu, Zhen Huang, Yunxiang Zhao, Zhiliang Tian, Yang Liu, Dongsheng Li

To this end, we employ domain adversarial learning as a heuristic neural network initialization method, which can help the meta-learning module converge to a better optimal.

Domain Adaptation Meta-Learning +2

Meta-Tsallis-Entropy Minimization: A New Self-Training Approach for Domain Adaptation on Text Classification

no code implementations4 Aug 2023 Menglong Lu, Zhen Huang, Zhiliang Tian, Yunxiang Zhao, Xuanyu Fei, Dongsheng Li

Theoretically, we prove the convergence of the meta-learning algorithm in MTEM and analyze the effectiveness of MTEM in achieving domain adaptation.

Domain Adaptation Meta-Learning +2

Recommendation Unlearning via Matrix Correction

no code implementations29 Jul 2023 Jiahao Liu, Dongsheng Li, Hansu Gu, Tun Lu, Jiongran Wu, Peng Zhang, Li Shang, Ning Gu

We conducted comprehensive experiments to validate the effectiveness of IMCorrect and the results demonstrate that IMCorrect is superior in completeness, utility, and efficiency, and is applicable in many recommendation unlearning scenarios.

Collaborative Filtering Recommendation Systems

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

no code implementations27 Jul 2023 Yu-Ting Lan, Kan Ren, Yansen Wang, Wei-Long Zheng, Dongsheng Li, Bao-liang Lu, Lili Qiu

Seeing is believing, however, the underlying mechanism of how human visual perceptions are intertwined with our cognitions is still a mystery.

EEG Image Reconstruction +1

A Dynamic Equivalent Method for PMSG Based Wind Farms Under Asymmetrical Faults

no code implementations7 Jul 2023 Dongsheng Li, Chen Shen

In this paper, a three-machine equivalent method applicable to asymmetrical faults is proposed considering the operating wind speed and fault severity.

Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance

no code implementations6 Jul 2023 Yuchen Fang, Zhenggang Tang, Kan Ren, Weiqing Liu, Li Zhao, Jiang Bian, Dongsheng Li, Weinan Zhang, Yong Yu, Tie-Yan Liu

Order execution is a fundamental task in quantitative finance, aiming at finishing acquisition or liquidation for a number of trading orders of the specific assets.

Reinforcement Learning (RL)

Large Language Models Empowered Autonomous Edge AI for Connected Intelligence

no code implementations6 Jul 2023 Yifei Shen, Jiawei Shao, Xinjie Zhang, Zehong Lin, Hao Pan, Dongsheng Li, Jun Zhang, Khaled B. Letaief

The evolution of wireless networks gravitates towards connected intelligence, a concept that envisions seamless interconnectivity among humans, objects, and intelligence in a hyper-connected cyber-physical world.

Code Generation Federated Learning +3

Is Risk-Sensitive Reinforcement Learning Properly Resolved?

no code implementations2 Jul 2023 Ruiwen Zhou, Minghuan Liu, Kan Ren, Xufang Luo, Weinan Zhang, Dongsheng Li

Due to the nature of risk management in learning applicable policies, risk-sensitive reinforcement learning (RSRL) has been realized as an important direction.

Distributional Reinforcement Learning Management +2

Protecting the Future: Neonatal Seizure Detection with Spatial-Temporal Modeling

no code implementations2 Jul 2023 Ziyue Li, Yuchen Fang, You Li, Kan Ren, Yansen Wang, Xufang Luo, Juanyong Duan, Congrui Huang, Dongsheng Li, Lili Qiu

A timely detection of seizures for newborn infants with electroencephalogram (EEG) has been a common yet life-saving practice in the Neonatal Intensive Care Unit (NICU).

EEG Electroencephalogram (EEG) +1

End-to-End Word-Level Pronunciation Assessment with MASK Pre-training

no code implementations5 Jun 2023 Yukang Liang, Kaitao Song, Shaoguang Mao, Huiqiang Jiang, Luna Qiu, Yuqing Yang, Dongsheng Li, Linli Xu, Lili Qiu

Pronunciation assessment is a major challenge in the computer-aided pronunciation training system, especially at the word (phoneme)-level.

XGrad: Boosting Gradient-Based Optimizers With Weight Prediction

1 code implementation26 May 2023 Lei Guan, Dongsheng Li, Jian Meng, Yanqi Shi

the future weights to update the DNN parameters, making the gradient-based optimizer achieve better convergence and generalization compared to the original optimizer without weight prediction.

Simulating News Recommendation Ecosystem for Fun and Profit

no code implementations23 May 2023 Guangping Zhang, Dongsheng Li, Hansu Gu, Tun Lu, Li Shang, Ning Gu

In this work, we propose SimuLine, a simulation platform to dissect the evolution of news recommendation ecosystems and present a detailed analysis of the evolutionary process and underlying mechanisms.

News Recommendation Recommendation Systems

DiffusionNER: Boundary Diffusion for Named Entity Recognition

2 code implementations22 May 2023 Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang

In this paper, we propose DiffusionNER, which formulates the named entity recognition task as a boundary-denoising diffusion process and thus generates named entities from noisy spans.

Chinese Named Entity Recognition Denoising +4

Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks

no code implementations22 May 2023 Haoqi Zheng, Qihuang Zhong, Liang Ding, Zhiliang Tian, Xin Niu, Dongsheng Li, DaCheng Tao

However, most of the mixup methods do not consider the varying degree of learning difficulty in different stages of training and generate new samples with one hot labels, resulting in the model over confidence.

Data Augmentation Few-Shot Text Classification +2

EA-HAS-Bench:Energy-Aware Hyperparameter and Architecture Search Benchmark

1 code implementation The Eleventh International Conference on Learning Representations 2023 Shuguang Dou, Xinyang Jiang, Cai Rong Zhao, Dongsheng Li

The energy consumption for training deep learning models is increasing at an alarming rate due to the growth of training data and model scale, resulting in a negative impact on carbon neutrality.


SIMPLE: Specialized Model-Sample Matching for Domain Generalization

1 code implementation International Conference on Learning Representations 2023 Ziyue Li, Kan Ren, Xinyang Jiang, Yifei Shen, Haipeng Zhang, Dongsheng Li

Moreover, our method is highly efficient and achieves more than 1000 times training speedup compared to the conventional DG methods with fine-tuning a pretrained model.

Domain Generalization

MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks

no code implementations28 Apr 2023 Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, Yuqing Yang

In contrast, though human engineers have the incredible ability to understand tasks and reason about solutions, their experience and knowledge are often sparse and difficult to utilize by quantitative approaches.


Triple Structural Information Modelling for Accurate, Explainable and Interactive Recommendation

no code implementations23 Apr 2023 Jiahao Liu, Dongsheng Li, Hansu Gu, Tun Lu, Peng Zhang, Li Shang, Ning Gu

Specifically, TriSIM4Rec consists of 1) a dynamic ideal low-pass graph filter to dynamically mine co-occurrence information in user-item interactions, which is implemented by incremental singular value decomposition (SVD); 2) a parameter-free attention module to capture sequential information of user interactions effectively and efficiently; and 3) an item transition matrix to store the transition probabilities of item pairs.

Collaborative Filtering

Biological Factor Regulatory Neural Network

1 code implementation11 Apr 2023 Xinnan Dai, Caihua Shan, Jie Zheng, Xiaoxiao Li, Dongsheng Li

BFReg-NN starts from gene expression data and is capable of merging most existing biological knowledge into the model, including the regulatory relations among genes or proteins (e. g., gene regulatory networks (GRN), protein-protein interaction networks (PPI)) and the hierarchical relations among genes, proteins and pathways (e. g., several genes/proteins are contained in a pathway).

Habits and goals in synergy: a variational Bayesian framework for behavior

1 code implementation11 Apr 2023 Dongqi Han, Kenji Doya, Dongsheng Li, Jun Tani

The habitual behavior is generated by using prior distribution of intention, which is goal-less; and the goal-directed behavior is generated by the posterior distribution of intention, which is conditioned on the goal.

HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

1 code implementation NeurIPS 2023 Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang

Solving complicated AI tasks with different domains and modalities is a key step toward artificial general intelligence.


A Hierarchical Regression Chain Framework for Affective Vocal Burst Recognition

1 code implementation14 Mar 2023 Jinchao Li, Xixin Wu, Kaitao Song, Dongsheng Li, Xunying Liu, Helen Meng

Experimental results based on the ACII Challenge 2022 dataset demonstrate the superior performance of the proposed system and the effectiveness of considering multiple relationships using hierarchical regression chain models.

A-VB Culture A-VB High +6

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

no code implementations14 Mar 2023 Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, Jing Jiang

In this paper, we consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online, and propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data.

Continuous Control Offline RL +2

Rotation-Invariant Transformer for Point Cloud Matching

1 code implementation CVPR 2023 Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, Slobodan Ilic

To this end, we introduce RoITr, a Rotation-Invariant Transformer to cope with the pose variations in the point cloud matching task.

Data Augmentation

Online Streaming Video Super-Resolution with Convolutional Look-Up Table

1 code implementation1 Mar 2023 Guanghao Yin, Zefan Qu, Xinyang Jiang, Shan Jiang, Zhenhua Han, Ningxin Zheng, Xiaohong Liu, Huan Yang, Yuqing Yang, Dongsheng Li, Lili Qiu

To facilitate the research on this problem, a new benchmark dataset named LDV-WebRTC is constructed based on a real-world online streaming system.

Video Super-Resolution

Unsupervised Video Anomaly Detection for Stereotypical Behaviours in Autism

no code implementations27 Feb 2023 Jiaqi Gao, Xinyang Jiang, Yuqing Yang, Dongsheng Li, Lili Qiu

Correspondingly, we propose a Dual Stream deep model for Stereotypical Behaviours Detection, DS-SBD, based on the temporal trajectory of human poses and the repetition patterns of human actions.

Activity Recognition Anomaly Detection +1

Personalized Graph Signal Processing for Collaborative Filtering

no code implementations4 Feb 2023 Jiahao Liu, Dongsheng Li, Hansu Gu, Tun Lu, Peng Zhang, Li Shang, Ning Gu

However, the interaction signal may not be sufficient to accurately characterize user interests and the low-pass filters may ignore the useful information contained in the high-frequency component of the observed signals, resulting in suboptimal accuracy.

Collaborative Filtering

Towards Inference Efficient Deep Ensemble Learning

no code implementations29 Jan 2023 Ziyue Li, Kan Ren, Yifan Yang, Xinyang Jiang, Yuqing Yang, Dongsheng Li

Ensemble methods can deliver surprising performance gains but also bring significantly higher computational costs, e. g., can be up to 2048X in large-scale ensemble tasks.

Ensemble Learning

SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking

no code implementations29 Jan 2023 Xiang Li, Tiandi Ye, Caihua Shan, Dongsheng Li, Ming Gao

In this paper, to comprehensively enhance the performance of generative graph SSL against other GCL models on both unsupervised and supervised learning tasks, we propose the SeeGera model, which is based on the family of self-supervised variational graph auto-encoder (VGAE).

Contrastive Learning Self-Supervised Learning +1

Progressive Meta-Pooling Learning for Lightweight Image Classification Model

no code implementations24 Jan 2023 Peijie Dong, Xin Niu, Zhiliang Tian, Lujun Li, Xiaodong Wang, Zimian Wei, Hengyue Pan, Dongsheng Li

Practical networks for edge devices adopt shallow depth and small convolutional kernels to save memory and computational cost, which leads to a restricted receptive field.

Classification Image Classification

RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies

1 code implementation24 Jan 2023 Peijie Dong, Xin Niu, Lujun Li, Zhiliang Tian, Xiaodong Wang, Zimian Wei, Hengyue Pan, Dongsheng Li

In this paper, we propose Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies as the cheap teacher and adopts the margin ranking loss to distill the ranking knowledge.

Neural Architecture Search

OVO: One-shot Vision Transformer Search with Online distillation

no code implementations28 Dec 2022 Zimian Wei, Hengyue Pan, Xin Niu, Dongsheng Li

OVO samples sub-nets for both teacher and student networks for better distillation results.

Learning Domain Invariant Prompt for Vision-Language Models

no code implementations8 Dec 2022 Cairong Zhao, Yubin Wang, Xinyang Jiang, Yifei Shen, Kaitao Song, Dongsheng Li, Duoqian Miao

Prompt learning is one of the most effective and trending ways to adapt powerful vision-language foundation models like CLIP to downstream datasets by tuning learnable prompt vectors with very few samples.

Domain Generalization Language Modelling +1

SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems

no code implementations7 Dec 2022 Jiangsu Du, Dongsheng Li, Yingpeng Wen, Jiazhi Jiang, Dan Huang, Xiangke Liao, Yutong Lu

In this paper, we propose a scalable evaluation methodology (SAIH) for analyzing the AI performance trend of HPC systems with scaling the problem sizes of customized AI applications.

CL4CTR: A Contrastive Learning Framework for CTR Prediction

1 code implementation1 Dec 2022 Fangye Wang, Yingxu Wang, Dongsheng Li, Hansu Gu, Tun Lu, Peng Zhang, Ning Gu

Many Click-Through Rate (CTR) prediction works focused on designing advanced architectures to model complex feature interactions but neglected the importance of feature representation learning, e. g., adopting a plain embedding layer for each feature, which results in sub-optimal feature representations and thus inferior CTR prediction performance.

Click-Through Rate Prediction Contrastive Learning +3

A Dynamic Equivalent Method for PMSG-WTG Based Wind Farms Considering wind Speeds and Fault Severities

no code implementations23 Nov 2022 Dongsheng Li, Chen Shen, Ye Liu, Ying Chen, Shaowei Huang

In order to reduce the complexity of simulation of power systems including large-scale wind farms, it is critical to develop dynamic equivalent methods for wind farms which are applicable to the expected contingency analysis.


Invisible Backdoor Attack with Dynamic Triggers against Person Re-identification

no code implementations20 Nov 2022 Wenli Sun, Xinyang Jiang, Shuguang Dou, Dongsheng Li, Duoqian Miao, Cheng Deng, Cairong Zhao

Instead of learning fixed triggers for the target classes from the training set, DT-IBA can dynamically generate new triggers for any unknown identities.

Backdoor Attack Image Steganography +2

Towards Understanding Omission in Dialogue Summarization

1 code implementation14 Nov 2022 Yicheng Zou, Kaitao Song, Xu Tan, Zhongkai Fu, Qi Zhang, Dongsheng Li, Tao Gui

By analyzing this dataset, we find that a large improvement in summarization quality can be achieved by providing ground-truth omission labels for the summarization model to recover omission information, which demonstrates the importance of omission detection for omission mitigation in dialogue summarization.

Parameter-free Dynamic Graph Embedding for Link Prediction

1 code implementation15 Oct 2022 Jiahao Liu, Dongsheng Li, Hansu Gu, Tun Lu, Peng Zhang, Ning Gu

Dynamic interaction graphs have been widely adopted to model the evolution of user-item interactions over time.

Dynamic graph embedding Link Prediction

A Unified Generative Framework based on Prompt Learning for Various Information Extraction Tasks

no code implementations23 Sep 2022 Zhigang Kan, Linhui Feng, Zhangyue Yin, Linbo Qiao, Xipeng Qiu, Dongsheng Li

In this paper, we propose a novel composable prompt-based generative framework, which could be applied to a wide range of tasks in the field of Information Extraction.

Relation Extraction

Improving Hypernasality Estimation with Automatic Speech Recognition in Cleft Palate Speech

no code implementations10 Aug 2022 Kaitao Song, Teng Wan, Bixia Wang, Huiqiang Jiang, Luna Qiu, Jiahang Xu, Liping Jiang, Qun Lou, Yuqing Yang, Dongsheng Li, Xudong Wang, Lili Qiu

Specifically, we first pre-train an encoder-decoder framework in an automatic speech recognition (ASR) objective by using speech-to-text dataset, and then fine-tune ASR encoder on the cleft palate dataset for hypernasality estimation.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +1

Online Video Super-Resolution with Convolutional Kernel Bypass Graft

no code implementations4 Aug 2022 Jun Xiao, Xinyang Jiang, Ningxin Zheng, Huan Yang, Yifan Yang, Yuqing Yang, Dongsheng Li, Kin-Man Lam

Then, our proposed CKBG method enhances this lightweight base model by bypassing the original network with ``kernel grafts'', which are extra convolutional kernels containing the prior knowledge of external pretrained image SR models.

Transfer Learning Video Super-Resolution

Towards Privacy-Preserving Person Re-identification via Person Identify Shift

no code implementations15 Jul 2022 Shuguang Dou, Xinyang Jiang, Qingsong Zhao, Dongsheng Li, Cairong Zhao

In this paper, we aim to develop a technique that can achieve a good trade-off between privacy protection and data usability for person ReID.

De-identification Person Re-Identification +1

RendNet: Unified 2D/3D Recognizer With Latent Space Rendering

no code implementations CVPR 2022 Ruoxi Shi, Xinyang Jiang, Caihua Shan, Yansen Wang, Dongsheng Li

Instead of looking at one format, it is a good solution to utilize the formats of VG and RG together to avoid these shortcomings.

3D Object Recognition Vector Graphics

Bootstrapped Transformer for Offline Reinforcement Learning

no code implementations17 Jun 2022 Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, Dongsheng Li

Offline reinforcement learning (RL) aims at learning policies from previously collected static trajectory data without interacting with the real environment.

Offline RL reinforcement-learning +1

Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

1 code implementation10 Jun 2022 Zhiquan Lai, Shengwei Li, Xudong Tang, Keshi Ge, Weijie Liu, Yabo Duan, Linbo Qiao, Dongsheng Li

These features make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism and tensor model parallelism, to achieve high training efficiency.

Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling

1 code implementation25 May 2022 Kaitao Song, Yichong Leng, Xu Tan, Yicheng Zou, Tao Qin, Dongsheng Li

Previous works on sentence scoring mainly adopted either causal language modeling (CLM) like GPT or masked language modeling (MLM) like BERT, which have some limitations: 1) CLM only utilizes unidirectional information for the probability estimation of a sentence without considering bidirectional context, which affects the scoring quality; 2) MLM can only estimate the probability of partial tokens at a time and thus requires multiple forward passes to estimate the probability of the whole sentence, which incurs large computation and time cost.

Causal Language Modeling Language Modelling +1

Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble

no code implementations19 May 2022 Zhengyu Yang, Kan Ren, Xufang Luo, Minghuan Liu, Weiqing Liu, Jiang Bian, Weinan Zhang, Dongsheng Li

Considering the great performance of ensemble methods on both accuracy and generalization in supervised learning (SL), we design a robust and applicable method named Ensemble Proximal Policy Optimization (EPPO), which learns ensemble policies in an end-to-end manner.

reinforcement-learning Reinforcement Learning (RL)

Enhancing CTR Prediction with Context-Aware Feature Representation Learning

1 code implementation19 Apr 2022 Fangye Wang, Yingxu Wang, Dongsheng Li, Hansu Gu, Tun Lu, Peng Zhang, Ning Gu

However, most methods only learn a fixed representation for each feature without considering the varying importance of each feature under different contexts, resulting in inferior performance.

Click-Through Rate Prediction Representation Learning

Learning Convolutional Neural Networks in the Frequency Domain

1 code implementation14 Apr 2022 Hengyue Pan, Yixin Chen, Xin Niu, Wenbo Zhou, Dongsheng Li

The most important motivation of this research is that we can use the straightforward element-wise multiplication operation to replace the image convolution in the frequency domain based on the Cross-Correlation Theorem, which obviously reduces the computation complexity.

DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation

1 code implementation30 Mar 2022 Yu Tang, Chenyu Wang, Yufan Zhang, Yuliang Liu, Xingcheng Zhang, Linbo Qiao, Zhiquan Lai, Dongsheng Li

To the best of our knowledge, we are the first to make a reasonable dynamic runtime scheduler on the combination of tensor swapping and tensor recomputation without user oversight.

CMMD: Cross-Metric Multi-Dimensional Root Cause Analysis

no code implementations30 Mar 2022 Shifu Yan, Caihua Shan, Wenyi Yang, Bixiong Xu, Dongsheng Li, Lili Qiu, Jie Tong, Qi Zhang

To this end, we propose a cross-metric multi-dimensional root cause analysis method, named CMMD, which consists of two key components: 1) relationship modeling, which utilizes graph neural network (GNN) to model the unknown complex calculation among metrics and aggregation function among dimensions from historical data; 2) root cause localization, which adopts the genetic algorithm to efficiently and effectively dive into the raw data and localize the abnormal dimension(s) once the KPI anomalies are detected.

VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning

1 code implementation17 Feb 2022 Che Wang, Xufang Luo, Keith Ross, Dongsheng Li

We propose VRL3, a powerful data-driven framework with a simple design for solving challenging visual deep reinforcement learning (DRL) tasks.

Offline RL reinforcement-learning +1

Neural Piecewise-Constant Delay Differential Equations

no code implementations4 Jan 2022 Qunxi Zhu, Yifei Shen, Dongsheng Li, Wei Lin

Continuous-depth neural networks, such as the Neural Ordinary Differential Equations (ODEs), have aroused a great deal of interest from the communities of machine learning and data science in recent years, which bridge the connection between deep neural networks and dynamical systems.

Reinforcement Learning Enhanced Explainer for Graph Neural Networks

no code implementations NeurIPS 2021 Caihua Shan, Yifei Shen, Yao Zhang, Xiang Li, Dongsheng Li

To address these issues, we propose a RL-enhanced GNN explainer, RG-Explainer, which consists of three main components: starting point selection, iterative graph generation and stopping criteria learning.

Combinatorial Optimization Graph Generation +2

Towards Generating Real-World Time Series Data

1 code implementation16 Nov 2021 Hengzhi Pei, Kan Ren, Yuqing Yang, Chang Liu, Tao Qin, Dongsheng Li

In this paper, we propose a novel generative framework for RTS data - RTSGAN to tackle the aforementioned challenges.

Time Series Time Series Analysis

EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks

no code implementations18 Oct 2021 Shengwei Li, Zhiquan Lai, Dongsheng Li, Yiming Zhang, Xiangyu Ye, Yabo Duan

EmbRace introduces Sparsity-aware Hybrid Communication, which integrates AlltoAll and model parallelism into data-parallel training, so as to reduce the communication overhead of highly sparse parameters.

Image Classification Scheduling

Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization

no code implementations18 Oct 2021 Tao Sun, Huaming Ling, Zuoqiang Shi, Dongsheng Li, Bao Wang

In this paper, to eliminate the effort for tuning the momentum-related hyperparameter, we propose a new adaptive momentum inspired by the optimal choice of the heavy ball momentum for quadratic optimization.

BIG-bench Machine Learning Image Classification +3

S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning

no code implementations5 Oct 2021 Keshi Ge, Yongquan Fu, Zhiquan Lai, Xiaoge Deng, Dongsheng Li

Distributed stochastic gradient descent (SGD) approach has been widely used in large-scale deep learning, and the gradient collective method is vital to ensure the training scalability of the distributed deep learning system.

Vocal Bursts Intensity Prediction

SANE: Specialization-Aware Neural Network Ensemble

no code implementations29 Sep 2021 Ziyue Li, Kan Ren, Xinyang Jiang, Mingzhe Han, Haipeng Zhang, Dongsheng Li

Real-world data is often generated by some complex distribution, which can be approximated by a composition of multiple simpler distributions.

Ensemble Learning

Deep Ensemble Policy Learning

no code implementations29 Sep 2021 Zhengyu Yang, Kan Ren, Xufang Luo, Weiqing Liu, Jiang Bian, Weinan Zhang, Dongsheng Li

Ensemble learning, which can consistently improve the prediction performance in supervised learning, has drawn increasing attentions in reinforcement learning (RL).

Ensemble Learning Reinforcement Learning (RL)

Adaptive Q-learning for Interaction-Limited Reinforcement Learning

no code implementations29 Sep 2021 Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, Jing Jiang

Specifically, we explicitly consider the difference between the online and offline data and apply an adaptive update scheme accordingly, i. e., a pessimistic update strategy for the offline dataset and a greedy or no pessimistic update scheme for the online dataset.

Offline RL Q-Learning +2

AARL: Automated Auxiliary Loss for Reinforcement Learning

no code implementations29 Sep 2021 Tairan He, Yuge Zhang, Kan Ren, Che Wang, Weinan Zhang, Dongsheng Li, Yuqing Yang

A good state representation is crucial to reinforcement learning (RL) while an ideal representation is hard to learn only with signals from the RL objective.

reinforcement-learning Reinforcement Learning (RL)

Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision

no code implementations30 Aug 2021 Bo Li, Xinyang Jiang, Donglin Bai, Yuge Zhang, Ningxin Zheng, Xuanyi Dong, Lu Liu, Yuqing Yang, Dongsheng Li

The energy consumption of deep learning models is increasing at a breathtaking rate, which raises concerns due to potential negative effects on carbon neutrality in the context of global warming and climate change.

Model Compression

How Powerful is Graph Convolution for Recommendation?

1 code implementation17 Aug 2021 Yifei Shen, Yongji Wu, Yao Zhang, Caihua Shan, Jun Zhang, Khaled B. Letaief, Dongsheng Li

In this paper, we endeavor to obtain a better understanding of GCN-based CF methods via the lens of graph signal processing.

Collaborative Filtering

Energy-Based Open-World Uncertainty Modeling for Confidence Calibration

no code implementations ICCV 2021 Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, Dongsheng Li

Confidence calibration is of great importance to the reliability of decisions made by machine learning systems.

Invariant Information Bottleneck for Domain Generalization

no code implementations11 Jun 2021 Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Colorado J. Reed, Jun Zhang, Dongsheng Li, Kurt Keutzer, Han Zhao

IIB significantly outperforms IRM on synthetic datasets, where the pseudo-invariant features and geometric skews occur, showing the effectiveness of proposed formulation in overcoming failure modes of IRM.

Domain Generalization

Deep Tiny Network for Recognition-Oriented Face Image Quality Assessment

no code implementations9 Jun 2021 Baoyun Peng, Min Liu, Heng Yang, Zhaoning Zhang, Dongsheng Li

Based on the proposed quality measurement, we propose a deep Tiny Face Quality network (tinyFQnet) to learn a quality prediction function from data.

Face Image Quality Face Image Quality Assessment +1

Scalable and Explainable 1-Bit Matrix Completion via Graph Signal Learning

1 code implementation AAAI 2021 Chao Chen, Dongsheng Li, Junchi Yan, Hanchi Huang, Xiaokang Yang

One-bit matrix completion is an important class of positiveunlabeled (PU) learning problems where the observations consist of only positive examples, eg, in top-N recommender systems.

Collaborative Ranking Matrix Completion +1

Graph Pooling via Coarsened Graph Infomax

1 code implementation4 May 2021 Yunsheng Pang, Yunxiang Zhao, Dongsheng Li

Graph pooling that summaries the information in a large graph into a compact form is essential in hierarchical graph representation learning.

Contrastive Learning Graph Representation Learning

Decentralized Federated Averaging

no code implementations23 Apr 2021 Tao Sun, Dongsheng Li, Bao Wang

In FedAvg, clients keep their data locally for privacy protection; a central parameter server is used to communicate between clients.

NeuSE: A Neural Snapshot Ensemble Method for Collaborative Filtering

no code implementations15 Apr 2021 Dongsheng Li, Haodong Liu, Chao Chen, Yingying Zhao, Stephen M. Chu, Bo Yang

In collaborative filtering (CF) algorithms, the optimal models are usually learned by globally minimizing the empirical risks averaged over all the observed data.

Collaborative Filtering Ensemble Learning

Hierarchical Adaptive Pooling by Capturing High-order Dependency for Graph Representation Learning

no code implementations13 Apr 2021 Ning Liu, Songlei Jian, Dongsheng Li, Yiming Zhang, Zhiquan Lai, Hongzuo Xu

Graph neural networks (GNN) have been proven to be mature enough for handling graph-structured data on node-level graph representation learning tasks.

Graph Classification Graph Matching +2

A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

no code implementations9 Apr 2021 Yingying Zhao, Mingzhi Dong, Yujiang Wang, Da Feng, Qin Lv, Robert P. Dick, Dongsheng Li, Tun Lu, Ning Gu, Li Shang

By monitoring the impact of varying resolution on the quality of high-dimensional video analytics features, hence the accuracy of video analytics results, the proposed end-to-end optimization framework learns the best non-myopic policy for dynamically controlling the resolution of input video streams to globally optimize energy efficiency.

Instance Segmentation Optical Flow Estimation +4

Decentralized Statistical Inference with Unrolled Graph Neural Networks

1 code implementation4 Apr 2021 He Wang, Yifei Shen, Ziyuan Wang, Dongsheng Li, Jun Zhang, Khaled B. Letaief, Jie Lu

In this paper, we investigate the decentralized statistical inference problem, where a network of agents cooperatively recover a (structured) vector from private noisy samples without centralized coordination.

Stability and Generalization of the Decentralized Stochastic Gradient Descent

no code implementations2 Feb 2021 Tao Sun, Dongsheng Li, Bao Wang

The stability and generalization of stochastic gradient-based methods provide valuable insights into understanding the algorithmic performance of machine learning models.

BIG-bench Machine Learning

Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training

no code implementations30 Jan 2021 Linbo Qiao, Tao Sun, Hengyue Pan, Dongsheng Li

In recent years, the Deep Learning Alternating Minimization (DLAM), which is actually the alternating minimization applied to the penalty form of the deep neutral networks training, has been developed as an alternative algorithm to overcome several drawbacks of Stochastic Gradient Descent (SGD) algorithms.

Learning content and context with language bias for Visual Question Answering

1 code implementation21 Dec 2020 Chao Yang, Su Feng, Dongsheng Li, HuaWei Shen, Guoqing Wang, Bin Jiang

Many works concentrate on how to reduce language bias which makes models answer questions ignoring visual content and language context.

Question Answering Visual Question Answering

Meta-Learning for Neural Relation Classification with Distant Supervision

no code implementations26 Oct 2020 Zhenzhen Li, Jian-Yun Nie, Benyou Wang, Pan Du, Yuhan Zhang, Lixin Zou, Dongsheng Li

Distant supervision provides a means to create a large number of weakly labeled data at low cost for relation classification.

Classification General Classification +2

Dynamic Knowledge Distillation for Black-box Hypothesis Transfer Learning

no code implementations24 Jul 2020 Yiqin Yu, Xu Min, Shiwan Zhao, Jing Mei, Fei Wang, Dongsheng Li, Kenney Ng, Shaochun Li

In real world applications like healthcare, it is usually difficult to build a machine learning prediction model that works universally well across different institutions.

Knowledge Distillation Transfer Learning

Adaptive Temporal Difference Learning with Linear Function Approximation

no code implementations20 Feb 2020 Tao Sun, Han Shen, Tianyi Chen, Dongsheng Li

Typically, the performance of TD(0) and TD($\lambda$) is very sensitive to the choice of stepsizes.

OpenAI Gym reinforcement-learning +1

Towards Precise End-to-end Weakly Supervised Object Detection Network

1 code implementation ICCV 2019 Ke Yang, Dongsheng Li, Yong Dou

It is challenging for weakly supervised object detection network to precisely predict the positions of the objects, since there are no instance-level category annotations.

Multiple Instance Learning object-detection +2

XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

no code implementations24 Oct 2019 Lei Guan, Wotao Yin, Dongsheng Li, Xicheng Lu

It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches.

General Proximal Incremental Aggregated Gradient Algorithms: Better and Novel Results under General Scheme

no code implementations NeurIPS 2019 Tao Sun, Yuejiao Sun, Dongsheng Li, Qing Liao

In this paper, we propose a general proximal incremental aggregated gradient algorithm, which contains various existing algorithms including the basic incremental aggregated gradient method.

Decentralized Markov Chain Gradient Descent

no code implementations23 Sep 2019 Tao Sun, Dongsheng Li

Decentralized stochastic gradient method emerges as a promising solution for solving large-scale machine learning problems.

A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning

1 code implementation IJCNLP 2019 Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li

Rapid progress has been made in the field of reading comprehension and question answering, where several systems have achieved human parity in some simplified settings.

Question Answering Reading Comprehension +1

Heavy-ball Algorithms Always Escape Saddle Points

no code implementations23 Jul 2019 Tao Sun, Dongsheng Li, Zhe Quan, Hao Jiang, Shengguo Li, Yong Dou

In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoid saddle points?

Exploring Pre-trained Language Models for Event Extraction and Generation

no code implementations ACL 2019 Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan, Dongsheng Li

Traditional approaches to the task of ACE event extraction usually depend on manually annotated data, which is often laborious to create and limited in size.

Event Extraction General Classification

Correlation Congruence for Knowledge Distillation

2 code implementations ICCV 2019 Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yi-Chao Wu, Yu Liu, Dongsheng Li, Zhaoning Zhang

Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level.

Face Recognition Image Classification +3

IF-TTN: Information Fused Temporal Transformation Network for Video Action Recognition

no code implementations26 Feb 2019 Ke Yang, Peng Qiao, Dongsheng Li, Yong Dou

Focusing on discriminate spatiotemporal feature learning, we propose Information Fused Temporal Transformation Network (IF-TTN) for action recognition on top of popular Temporal Segment Network (TSN) framework.

Action Recognition Optical Flow Estimation +1

Exploring Frame Segmentation Networks for Temporal Action Localization

no code implementations14 Feb 2019 Ke Yang, Xiaolong Shen, Peng Qiao, Shijie Li, Dongsheng Li, Yong Dou

The proposed FSN can make dense predictions at frame-level for a video clip using both spatial and temporal context information.

Open-Ended Question Answering Temporal Action Localization

Iteratively reweighted penalty alternating minimization methods with continuation for image deblurring

no code implementations9 Feb 2019 Tao Sun, Dongsheng Li, Hao Jiang, Zhe Quan

In this paper, we consider a class of nonconvex problems with linear constraints appearing frequently in the area of image processing.

Deblurring Image Deblurring

Collaborative Filtering with Stability

no code implementations6 Nov 2018 Dongsheng Li, Chao Chen, Qin Lv, Junchi Yan, Li Shang, Stephen M. Chu

Collaborative filtering (CF) is a popular technique in today's recommender systems, and matrix approximation-based CF methods have achieved great success in both rating prediction and top-N recommendation tasks.

Collaborative Filtering Recommendation Systems +1

Non-ergodic Convergence Analysis of Heavy-Ball Algorithms

no code implementations5 Nov 2018 Tao Sun, Penghang Yin, Dongsheng Li, Chun Huang, Lei Guan, Hao Jiang

For objective functions satisfying a relaxed strongly convex condition, the linear convergence is established under weaker assumptions on the step size and inertial parameter than made in the existing literature.

An Efficient ADMM-Based Algorithm to Nonconvex Penalized Support Vector Machines

no code implementations11 Sep 2018 Lei Guan, Linbo Qiao, Dongsheng Li, Tao Sun, Keshi Ge, Xicheng Lu

Support vector machines (SVMs) with sparsity-inducing nonconvex penalties have received considerable attentions for the characteristics of automatic classification and variable selection.

General Classification Variable Selection

Attention-Guided Answer Distillation for Machine Reading Comprehension

no code implementations EMNLP 2018 Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, Ming Zhou

Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models.

Knowledge Distillation Machine Reading Comprehension +1

Loss Rank Mining: A General Hard Example Mining Method for Real-time Detectors

no code implementations10 Apr 2018 Hao Yu, Zhaoning Zhang, Zheng Qin, Hao Wu, Dongsheng Li, Jun Zhao, Xicheng Lu

LRM is a general method for real-time detectors, as it utilizes the final feature map which exists in all real-time detectors to mine hard examples.

Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions

3 code implementations27 Mar 2018 Zheng Qin, Zhaoning Zhang, Dongsheng Li, Yiming Zhang, Yuxing Peng

Depthwise convolutions provide significant performance benefits owing to the reduction in both parameters and mult-adds.

Non-ergodic Complexity of Convex Proximal Inertial Gradient Descents

no code implementations23 Jan 2018 Tao Sun, Linbo Qiao, Dongsheng Li

The non-ergodic O(1/k) rate is proved for proximal inertial gradient descent with constant stepzise when the objective function is coercive.

Mixture-Rank Matrix Approximation for Collaborative Filtering

1 code implementation NeurIPS 2017 Dongsheng Li, Chao Chen, Wei Liu, Tun Lu, Ning Gu, Stephen Chu

However, our studies show that submatrices with different ranks could coexist in the same user-item rating matrix, so that approximations with fixed ranks cannot perfectly describe the internal structures of the rating matrix, therefore leading to inferior recommendation accuracy.

Collaborative Filtering

Exploring Temporal Preservation Networks for Precise Temporal Action Localization

no code implementations10 Aug 2017 Ke Yang, Peng Qiao, Dongsheng Li, Shaohe Lv, Yong Dou

A newly proposed work exploits Convolutional-Deconvolutional-Convolutional (CDC) filters to upsample the predictions of 3D ConvNets, making it possible to perform per-frame action predictions and achieving promising performance in terms of temporal action localization.

Open-Ended Question Answering Temporal Action Localization +1

S-OHEM: Stratified Online Hard Example Mining for Object Detection

no code implementations5 May 2017 Minne Li, Zhaoning Zhang, Hao Yu, Xinyuan Chen, Dongsheng Li

S-OHEM exploits OHEM with stratified sampling, a widely-adopted sampling technique, to choose the training examples according to this influence during hard example mining, and thus enhance the performance of object detectors.

object-detection Object Detection

Weakly supervised object detection using pseudo-strong labels

no code implementations16 Jul 2016 Ke Yang, Dongsheng Li, Yong Dou, Shaohe Lv, Qiang Wang

Object detection is an import task of computer vision. A variety of methods have been proposed, but methods using the weak labels still do not have a satisfactory result. In this paper, we propose a new framework that using the weakly supervised method's output as the pseudo-strong labels to train a strongly supervised model. One weakly supervised method is treated as black-box to generate class-specific bounding boxes on train dataset. A de-noise method is then applied to the noisy bounding boxes. Then the de-noised pseudo-strong labels are used to train a strongly object detection network. The whole framework is still weakly supervised because the entire process only uses the image-level labels. The experiment results on PASCAL VOC 2007 prove the validity of our framework, and we get result 43. 4% on mean average precision compared to 39. 5% of the previous best result and 34. 5% of the initial method, respectively. And this frame work is simple and distinct, and is promising to be applied to other method easily.

object-detection Weakly Supervised Object Detection

Cannot find the paper you are looking for? You can Submit a new open access paper.