Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions

1 code implementation • 2 Aug 2024 • Jin Gao, Lei Gan, Yuankai Li, Yixin Ye, Dequan Wang

Large multimodal models (LMMs) excel in adhering to human instructions.

Benchmarking

Lost in Translation: Latent Concept Misalignment in Text-to-Image Diffusion Models

1 code implementation • 1 Aug 2024 • Juntu Zhao, Junyu Deng, Yixin Ye, Chongxuan Li, Zhijie Deng, Dequan Wang

The root of such misalignment is attributed to the confusion in the latent semantic space of text-to-image diffusion models, and hence we refer to the "a tea cup of iced coke" phenomenon as Latent Concept Misalignment (LC-Mis).

OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI

1 code implementation • 18 Jun 2024 • Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang, Dahua Lin, Yu Qiao, PengFei Liu

We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions.

Benchmarking scientific discovery