TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation

19 Dec 2024  ·  Jiatong Li, Junxian Li, Yunqing Liu, Dongzhan Zhou, Qing Li ·

In this paper, we propose Text-based Open Molecule Generation Benchmark (TOMG-Bench), the first benchmark to evaluate the open-domain molecule generation capability of LLMs. TOMG-Bench encompasses a dataset of three major tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and customized molecule generation (MolCustom). Each task further contains three subtasks, with each subtask comprising 5,000 test samples. Given the inherent complexity of open molecule generation, we have also developed an automated evaluation system that helps measure both the quality and the accuracy of the generated molecules. Our comprehensive benchmarking of 25 LLMs reveals the current limitations and potential areas for improvement in text-guided molecule discovery. Furthermore, with the assistance of OpenMolIns, a specialized instruction tuning dataset proposed for solving challenges raised by TOMG-Bench, Llama3.1-8B could outperform all the open-source general LLMs, even surpassing GPT-3.5-turbo by 46.5\% on TOMG-Bench. Our codes and datasets are available through https://github.com/phenixace/TOMG-Bench.

PDF Abstract

Datasets


Introduced in the Paper:

TOMG-Bench

Used in the Paper:

ChEBI-20
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Description-guided molecule generation TOMG-Bench Claude-3.5 wAcc 35.92 # 1
Description-guided molecule generation TOMG-Bench Qwen2-7B-Instruct wAcc 0.15 # 25
Description-guided molecule generation TOMG-Bench MolT5-small wAcc 1.299 # 24
Description-guided molecule generation TOMG-Bench MolT5-base wAcc 1.300 # 23
Description-guided molecule generation TOMG-Bench Llama-3.1-1B-Instruct wAcc 1.99 # 22
Description-guided molecule generation TOMG-Bench MolT5-large wAcc 2.89 # 21
Description-guided molecule generation TOMG-Bench BioT5-base wAcc 4.21 # 20
Description-guided molecule generation TOMG-Bench Mistral-7B-Instruct-v0.2 wAcc 4.81 # 19
Description-guided molecule generation TOMG-Bench yi-1.5-9B wAcc 7.32 # 18
Description-guided molecule generation TOMG-Bench Llama3.2-1B (OpenMolIns-large) wAcc 8.10 # 17
Description-guided molecule generation TOMG-Bench Galactica-125M (OpenMolIns-light) wAcc 13.136 # 16
Description-guided molecule generation TOMG-Bench chatglm-9B wAcc 13.137 # 15
Description-guided molecule generation TOMG-Bench Llama3-8B-Instruct wAcc 13.75 # 14
Description-guided molecule generation TOMG-Bench Llama3.1-8B-Instruct wAcc 14.09 # 13
Description-guided molecule generation TOMG-Bench Galactica-125M (OpenMolIns-small) wAcc 15.18 # 12
Description-guided molecule generation TOMG-Bench GPT-3.5-turbo wAcc 18.58 # 11
Description-guided molecule generation TOMG-Bench Galactica-125M (OpenMolIns-medium) wAcc 19.89 # 10
Description-guided molecule generation TOMG-Bench Galactica-125M (OpenMolIns-large) wAcc 23.42 # 9
Description-guided molecule generation TOMG-Bench Llama3-70B-Instruct (INT4) wAcc 23.93 # 8
Description-guided molecule generation TOMG-Bench Galactica-125M (OpenMolIns-xlarge) wAcc 25.73 # 7
Description-guided molecule generation TOMG-Bench Llama-3.1-8B (OpenMolIns-large) wAcc 27.22 # 6
Description-guided molecule generation TOMG-Bench Claude-3 wAcc 30.47 # 5
Description-guided molecule generation TOMG-Bench GPT-4o wAcc 32.29 # 4
Description-guided molecule generation TOMG-Bench GPT-4-turbo wAcc 34.23 # 3
Description-guided molecule generation TOMG-Bench Gemini-1.5-pro wAcc 34.80 # 2

Methods