PaLM 2 Technical Report

no code implementations • 17 May 2023 • Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, Yaguang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, ZiRui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, Yonghui Wu

Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.

Language Modelling Question Answering

HyperPrompt: Prompt-based Task-Conditioning of Transformers

no code implementations • 1 Mar 2022 • Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, Yaguang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, Ed H. Chi

Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way.

Multi-Task Learning Natural Language Understanding

LaMDA: Language Models for Dialog Applications

2 code implementations • 20 Jan 2022 • Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, Quoc Le

We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.

Code Generation Information Retrieval +2

The relationship between photometric and spectroscopic oscillation amplitudes from 3D stellar atmosphere simulations

no code implementations • 3 Feb 2021 • Yixiao Zhou, Thomas Nordlander, Luca Casagrande, Meridith Joyce, Yaguang Li, Anish M. Amarsi, Henrique Reggiani, Martin Asplund

For the Sun, we predict theoretical ratios of 21. 0 and 23. 7 ppm/[m/s] from BiSON and SONG respectively, in good agreement with observations 19. 1 and 21. 6 ppm/[m/s].

Solar and Stellar Astrophysics

Mode Angular Degree Identification in Subgiant Stars with Convolutional Neural Networks based on Power Spectrum

no code implementations • 24 Dec 2020 • Minghao Du, Shaolan Bi, Xianfei Zhang, Yaguang Li, Tanda Li, Ruijie Shi

Identifying the angular degrees $l$ of oscillation modes is essential for asteroseismology and depends on visual tagging before fitting power spectra in a so-called peakbagging analysis.

Solar and Stellar Astrophysics

Predicting Origin-Destination Flow via Multi-Perspective Graph Convolutional Network

1 code implementation • 2020 IEEE 36th International Conference on Data Engineering (ICDE) 2020 • Hongzhi Shi, Quanming Yao, Qi Guo, Yaguang Li, Lingyu Zhang, Jieping Ye, Yong Li, Yan Liu

Predicting Origin-Destination (OD) flow is a crucial problem for intelligent transportation.

Philosophy

DETECT: Deep Trajectory Clustering for Mobility-Behavior Analysis

no code implementations • 3 Mar 2020 • Mingxuan Yue, Yaguang Li, Haoze Yang, Ritesh Ahuja, Yao-Yi Chiang, Cyrus Shahabi

Identifying mobility behaviors in rich trajectory data is of great economic and social interest to various applications including urban planning, marketing and intelligence.

Clustering Marketing +1

Spatiotemporal Multi-Graph Convolution Network for Ride-hailing Demand Forecasting

1 code implementation • AAAI 2019 • Xu Geng, Yaguang Li, Leye Wang, Lingyu Zhang, Qiang Yang, Jieping Ye, Yan Liu

This task is challenging due to the complicated spatiotemporal dependencies among regions.

Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting

14 code implementations • ICLR 2018 • Yaguang Li, Rose Yu, Cyrus Shahabi, Yan Liu

Spatiotemporal forecasting has various applications in neuroscience, climate and transportation domain.

Multivariate Time Series Forecasting Spatio-Temporal Forecasting +2