Prompt Cache: Modular Attention Reuse for Low-Latency Inference

no code implementations • 7 Nov 2023 • In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong

We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts.

Question Answering

SmartChoices: Augmenting Software with Learned Implementations

no code implementations • 12 Apr 2023 • Daniel Golovin, Gabor Bartok, Eric Chen, Emily Donahue, Tzu-Kuo Huang, Efi Kokiopoulou, Ruoyan Qin, Nikhil Sarda, Justin Sybrandt, Vincent Tjeng

We are living in a golden age of machine learning.

Philosophy

Predicted Variables in Programming

no code implementations • ICLR 2019 • Victor Carbune, Thierry Coppey, Alexander Daryin, Thomas Deselaers, Nikhil Sarda, Jay Yagnik

We leverage the existing concept of variables and create a new type, a predicted variable.

Reinforcement Learning (RL)

SmartChoices: Hybridizing Programming and Machine Learning

no code implementations • ICLR 2019 • Victor Carbune, Thierry Coppey, Alexander Daryin, Thomas Deselaers, Nikhil Sarda, Jay Yagnik

As opposed to previous work applying ML to algorithmic problems, our proposed approach does not require to drop existing implementations but seamlessly integrates into the standard software development workflow and gives full control to the software developer over how ML methods are applied.

BIG-bench Machine Learning Reinforcement Learning (RL)