Search Results for author:
Found 16 papers, 6 papers with code
External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation
Second, large-volume data arrive in a streaming mode with data distributions dynamically shifting, as new users/ads join and existing users/ads leave the system.
The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exit
The deployment of Large Language Models (LLMs) in recommender systems for predicting Click-Through Rates (CTR) necessitates a delicate balance between computational efficiency and predictive accuracy.
Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1. 4B dense baseline across key image generation metrics.
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
Under a 1-trillion-token training budget, the MoMa 1. 4B model, featuring 4 text experts and 4 image experts, achieves impressive FLOPs savings: 3. 7x overall, with 2. 6x for text and 5. 2x for image processing compared to a compute-equivalent dense baseline, measured by pre-training loss.
Wukong: Towards a Scaling Law for Large-Scale Recommendation
Scaling laws play an instrumental role in the sustainable improvement in model quality.
Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation
We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology.
Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models
In this work, we explore a "pre-train, and search" paradigm for efficient sharding.
Self-discipline on multiple channels
Comparative experimental results on both datasets show that SMC-2 outperforms Label Smoothing Regularizaion and Self-distillation From The Last Mini-batch on all models, and outperforms the state-of-the-art Sharpness-Aware Minimization method on 83% of the models. Compatibility of SMC-2 and data augmentation experimental results show that using both SMC-2 and data augmentation improves the generalization ability of the model between 0. 28% and 1. 80% compared to using only data augmentation.
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains.
DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction
To overcome the challenge brought by DHEN's deeper and multi-layer structure in training, we propose a novel co-designed training system that can further improve the training efficiency of DHEN.
Characterizing and Taming Resolution in Convolutional Neural Networks
Image resolution has a significant effect on the accuracy and computational, storage, and bandwidth costs of computer vision model inference.
Cloud Collectives: Towards Cloud-aware Collectives forML Workloads with Rank Reordering
ML workloads are becoming increasingly popular in the cloud.
Accelerating SpMM Kernel with Cache-First Edge Sampling for Graph Neural Networks
To evaluate ES-SpMM's performance, we integrated it with a popular GNN framework, DGL, and tested it using representative GNN models and datasets.
Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models
Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers.
Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
Distributed deep neural network (DDNN) training constitutes an increasingly important workload that frequently runs in the cloud.
