You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments

1 code implementation • 16 Nov 2023 • Bangzhao Shu, Lechen Zhang, MinJe Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, David Jurgens

The versatility of Large Language Models (LLMs) on natural language understanding tasks has made them popular for research in social sciences.

Natural Language Understanding Negation +1

Exploring Linguistic Style Matching in Online Communities: The Role of Social Context and Conversation Dynamics

no code implementations • 6 Jul 2023 • Aparna Ananthasubramaniam, Hong Chen, Jason Yan, Kenan Alkiek, Jiaxin Pei, Agrima Seth, Lavinia Dunagan, MinJe Choi, Benjamin Litterer, David Jurgens

Linguistic style matching (LSM) in conversations can be reflective of several aspects of social influence such as power or persuasion.

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand

2 code implementations • NAACL 2022 • Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R. Fabbri, Yejin Choi, Noah A. Smith

We therefore propose a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation.

Image Captioning Machine Translation +1

Transparent Human Evaluation for Image Captioning

2 code implementations • NAACL 2022 • Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, Noah A. Smith

We establish THumB, a rubric-based human evaluation protocol for image captioning models.

Image Captioning