How to Build Low-cost Networks for Large Language Models (without Sacrificing Performance)?

no code implementations • 22 Jul 2023 • Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, Naader Hasani

We show that LLMs exhibit a unique communication pattern where only small groups of GPUs require high-bandwidth communication to achieve near-optimal training performance.

Blocking Language Modelling +1