ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters

no code implementations • 31 Jan 2024 • Shiwei Liu, Guanchen Tao, Yifei Zou, Derek Chow, Zichen Fan, Kauna Lei, Bangfei Pan, Dennis Sylvester, Gregory Kielian, Mehdi Saligane

Compared to state-of-the-art Softmax hardware, ConSmax results in 14. 5x energy and 14. 0x area savings with a comparable accuracy on a GPT-2 model and the WikiText103 dataset.

Language Modelling Large Language Model