Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis

1 code implementation • 13 Jul 2024 • Xiuying Wei, Skander Moalla, Razvan Pascanu, Caglar Gulcehre

State-of-the-art LLMs often rely on scale with high computational costs, which has sparked a research agenda to reduce parameter counts and costs without significantly impacting performance.

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

1 code implementation • 24 Jun 2024 • Xiuying Wei, Skander Moalla, Razvan Pascanu, Caglar Gulcehre

Additionally, we propose a novel training regime, called \textit{self-guided training}, aimed at improving the poor training dynamics that these approximations exhibit when used from initialization.

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO

1 code implementation • 1 May 2024 • Skander Moalla, Andrea Miele, Razvan Pascanu, Caglar Gulcehre

We find that there is a connection between representation collapse and the degradation of the trust region, one exacerbating the other, and present Proximal Feature Optimization (PFO), a novel auxiliary loss that, along with other interventions, shows that regularizing the representation dynamics improves the performance of PPO agents.

Reinforcement Learning (RL)

SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning

1 code implementation • NeurIPS 2023 • Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, Shimon Whiteson

In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex *closed-loop* policies.

reinforcement-learning SMAC+ +1