Search Results for author: Phillip Wallis

Found 3 papers, 2 papers with code

LoRA: Low-Rank Adaptation of Large Language Models

1 code implementation17 Jun 2021 Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Weizhu Chen

We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.

Language Modelling

Differential Equation Units: Learning Functional Forms of Activation Functions from Data

1 code implementation6 Sep 2019 MohamadAli Torkamani, Shiv Shankar, Amirmohammad Rooshenas, Phillip Wallis

Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure.

Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions

no code implementations19 May 2019 MohamadAli Torkamani, Phillip Wallis, Shiv Shankar, Amirmohammad Rooshenas

Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure.

Cannot find the paper you are looking for? You can Submit a new open access paper.