Search Results for author: Jian Gang Ngui

Found 6 papers, 4 papers with code

Do pretrained transformers infer telicity like humans?

no code implementations CoNLL (EMNLP) 2021 Yiyun Zhao, Jian Gang Ngui, Lucy Hall Hartley, Steven Bethard

Pretrained transformer-based language models achieve state-of-the-art performance in many NLP tasks, but it is an open question whether the knowledge acquired by the models during pretraining resembles the linguistic knowledge of humans.

Open-Ended Question Answering

SEA-HELM: Southeast Asian Holistic Evaluation of Language Models

1 code implementation20 Feb 2025 Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xian Bin Yong, Weiqi Leong, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Yifan Mai, William Chandra Tjhi

Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and authentic evaluation suite for the SEA languages has not been developed thus far.

Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino

1 code implementation20 Sep 2024 Jann Railey Montalan, Jian Gang Ngui, Wei Qi Leong, Yosephine Susanto, Hamsawardhini Rengarajan, Alham Fikri Aji, William Chandra Tjhi

Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users.

BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models

3 code implementations12 Sep 2023 Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, William Chandra Tjhi

As GPT-4 is purportedly one of the best-performing multilingual LLMs at the moment, we use it as a yardstick to gauge the capabilities of LLMs in the context of SEA languages.

Diagnostic Natural Language Understanding

Cannot find the paper you are looking for? You can Submit a new open access paper.