Search Results for author: dianhai yu

Found 28 papers, 17 papers with code

SE-MoE: A Scalable and Efficient Mixture-of-Experts Distributed Training and Inference System

1 code implementation20 May 2022 Liang Shen, Zhihua Wu, Weibao Gong, Hongxiang Hao, Yangfan Bai, HuaChao Wu, Xinxuan Wu, Haoyi Xiong, dianhai yu, Yanjun Ma

With the increasing diversity of ML infrastructures nowadays, distributed training over heterogeneous computing systems is desired to facilitate the production of big models.

Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters

no code implementations19 May 2022 Yang Xiang, Zhihua Wu, Weibao Gong, Siyu Ding, Xianjie Mo, Yuang Liu, Shuohuan Wang, Peng Liu, Yongshuai Hou, Long Li, Bin Wang, Shaohuai Shi, Yaqian Han, Yue Yu, Ge Li, Yu Sun, Yanjun Ma, dianhai yu

We took natural language processing (NLP) as an example to show how Nebula-I works in different training phases that include: a) pre-training a multilingual language model using two remote clusters; and b) fine-tuning a machine translation model using knowledge distilled from pre-trained models, which run through the most popular paradigm of recent deep learning.

Cross-Lingual Natural Language Inference Language Modelling +1

Simple and Effective Relation-based Embedding Propagation for Knowledge Representation Learning

1 code implementation13 May 2022 Huijuan Wang, Siming Dai, Weiyue Su, Hui Zhong, Zeyang Fang, Zhengjie Huang, Shikun Feng, Zeyu Chen, Yu Sun, dianhai yu

Notably, it averagely brings about 10% relative improvement to triplet-based embedding methods on OGBL-WikiKG2 and takes 5%-83% time to achieve comparable results as the state-of-the-art GC-OTE.

Knowledge Graphs Representation Learning

End-to-end Adaptive Distributed Training on PaddlePaddle

1 code implementation6 Dec 2021 Yulong Ao, Zhihua Wu, dianhai yu, Weibao Gong, Zhiqing Kui, Minxu Zhang, Zilingfeng Ye, Liang Shen, Yanjun Ma, Tian Wu, Haifeng Wang, Wei Zeng, Chao Yang

The experiments demonstrate that our framework can satisfy various requirements from the diversity of applications and the heterogeneity of resources with highly competitive performance.

Language Modelling Recommendation Systems

Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning

no code implementations22 Oct 2021 Yang Yang, Hongchen Wei, HengShu Zhu, dianhai yu, Hui Xiong, Jian Yang

In detail, considering that the heterogeneous gap between modalities always leads to the supervision difficulty of using the global embedding directly, CPRC turns to transform both the raw image and corresponding generated sentence into the shared semantic space, and measure the generated sentence from two aspects: 1) Prediction consistency.

Image Captioning Informativeness

PP-LCNet: A Lightweight CPU Convolutional Neural Network

7 code implementations17 Sep 2021 Cheng Cui, Tingquan Gao, Shengyu Wei, Yuning Du, Ruoyu Guo, Shuilong Dong, Bin Lu, Ying Zhou, Xueying Lv, Qiwen Liu, Xiaoguang Hu, dianhai yu, Yanjun Ma

We propose a lightweight CPU network based on the MKLDNN acceleration strategy, named PP-LCNet, which improves the performance of lightweight models on multiple tasks.

Image Classification Object Detection +1

PP-YOLOv2: A Practical Object Detector

1 code implementation21 Apr 2021 Xin Huang, Xinxin Wang, Wenyu Lv, Xiaying Bai, Xiang Long, Kaipeng Deng, Qingqing Dang, Shumin Han, Qiwen Liu, Xiaoguang Hu, dianhai yu, Yanjun Ma, Osamu Yoshie

To meet these two concerns, we comprehensively evaluate a collection of existing refinements to improve the performance of PP-YOLO while almost keep the infer time unchanged.

Distilling Knowledge from Pre-trained Language Models via Text Smoothing

no code implementations8 May 2020 Xing Wu, Yibing Liu, Xiangyang Zhou, dianhai yu

As an alternative, we propose a new method for BERT distillation, i. e., asking the teacher to generate smoothed word ids, rather than labels, for teaching the student model in knowledge distillation.

Knowledge Distillation Language Modelling

RLTM: An Efficient Neural IR Framework for Long Documents

no code implementations22 Jun 2019 Chen Zheng, Yu Sun, Shengxian Wan, dianhai yu

This paper proposes a novel End-to-End neural ranking framework called Reinforced Long Text Matching (RLTM) which matches a query with long documents efficiently and effectively.

Information Retrieval Text Matching

A New Method of Region Embedding for Text Classification

1 code implementation ICLR 2018 chao qiao, Bo Huang, guocheng niu, daren li, daxiang dong, wei he, dianhai yu, Hua Wu

In this paper, we propose a new method of learning and utilizing task-specific distributed representations of n-grams, referred to as “region embeddings”.

Classification General Classification +1

Cannot find the paper you are looking for? You can Submit a new open access paper.