Search Results for author:
Found 19 papers, 14 papers with code
Code Comparison Tuning for Code Large Language Models
We present Code Comparison Tuning (CCT), a simple and effective tuning method for code large language models (Code LLMs) to better handle subtle code errors.
DeepAdaIn-Net: Deep Adaptive Device-Edge Collaborative Inference for Augmented Reality
Specifically, DeepAdaIn-Net encompasses a partition point selection (PPS) module, a high feature compression learning (HFCL) module, a bandwidth-aware feature configuration (BaFC) module, and a feature consistency compensation (FCC) module.
RecycleGPT: An Autoregressive Language Model with Recyclable Module
Existing large language models have to run K times to generate a sequence of K tokens.
TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training
In this paper, we propose TA-MoE, a topology-aware routing strategy for large-scale MoE trainging, from a model-system co-design perspective, which can dynamically adjust the MoE dispatch pattern according to the network topology.
HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddle
Due to the complex model architecture and large memory consumption, it requires lots of computational resources and time to implement the training and inference of AlphaFold2 from scratch.
SE-MoE: A Scalable and Efficient Mixture-of-Experts Distributed Training and Inference System
With the increasing diversity of ML infrastructures nowadays, distributed training over heterogeneous computing systems is desired to facilitate the production of big models.
Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters
We took natural language processing (NLP) as an example to show how Nebula-I works in different training phases that include: a) pre-training a multilingual language model using two remote clusters; and b) fine-tuning a machine translation model using knowledge distilled from pre-trained models, which run through the most popular paradigm of recent deep learning.
Cross-Lingual Natural Language Inference
Distributed Computing
+2
ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation
To explore the landscape of large-scale pre-training for bidirectional text-image generation, we train a 10-billion parameter ERNIE-ViLG model on a large-scale dataset of 145 million (Chinese) image-text pairs which achieves state-of-the-art performance for both text-to-image and image-to-text tasks, obtaining an FID of 7. 9 on MS-COCO for text-to-image synthesis and best results on COCO-CN and AIC-ICC for image captioning.
Ranked #42 on
Text-to-Image Generation
on MS COCO
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
A unified framework named ERNIE 3. 0 was recently proposed for pre-training large-scale knowledge enhanced models and trained a model with 10 billion parameters.
End-to-end Adaptive Distributed Training on PaddlePaddle
The experiments demonstrate that our framework can satisfy various requirements from the diversity of applications and the heterogeneity of resources with highly competitive performance.
HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments
The training process generally exploits distributed computing resources to reduce training time.
PaddleRec
A quick start tool of search & recommendation algorithm based on PaddlePaddle A complete solution of recommendation system for beginners, developers and researchers.
PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
To explore the limit of dialogue generation pre-training, we present the models of PLATO-XL with up to 11 billion parameters, trained on both Chinese and English social media conversations.
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph.
ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
Specifically, we propose two tasks to learn the appearance and speed consistency, respectively.
MNN: A Universal and Efficient Inference Engine
Deploying deep learning models on mobile devices draws more and more attention recently.
Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availability
In this work, we consider a practical and ubiquitous issue when deploying federated learning in mobile environments: intermittent client availability, where the set of eligible clients may change during the training process.
Secure Federated Submodel Learning
Nevertheless, the "position" of a client's truly required submodel corresponds to her private data, and its disclosure to the cloud server during interactions inevitably breaks the tenet of federated learning.
From Server-Based to Client-Based Machine Learning: A Comprehensive Survey
Another benefit is the bandwidth reduction because various kinds of local data can be involved in the training process without being uploaded.
