Search Results for author:
Found 9 papers, 5 papers with code
Effective Long-Context Scaling of Foundation Models
We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.
Federated Learning with Partial Model Personalization
We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices.
FedSynth: Gradient Compression via Synthetic Data in Federated Learning
Model compression is important in federated learning (FL) with large models to reduce communication cost.
Papaya: Practical, Private, and Scalable Federated Learning
Our work tackles the aforementioned issues, sketches of some of the system design challenges and their solutions, and touches upon principles that emerged from building a production FL system for millions of clients.
Federated Learning with Buffered Asynchronous Aggregation
On the other hand, asynchronous aggregation of client updates in FL (i. e., asynchronous FL) alleviates the scalability issue.
Federated User Representation Learning
Furthermore, we show that user embeddings learned in FL and the centralized setting have a very similar structure, indicating that FURL can learn collaboratively through the shared parameters while preserving user privacy.
Active Federated Learning
To exploit this we propose Active Federated Learning, where in each round clients are selected not uniformly at random, but with a probability conditioned on the current model and the data on the client to maximize efficiency.
