A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

12 Sep 2022  ·  Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, Ji-Rong Wen ·

Although artificial intelligence (AI) has made significant progress in understanding molecules in a wide range of fields, existing models generally acquire the single cognitive ability from the single molecular modality. Since the hierarchy of molecular knowledge is profound, even humans learn from different modalities including both intuitive diagrams and professional texts to assist their understanding. Inspired by this, we propose a molecular multimodal foundation model which is pretrained from molecular graphs and their semantically related textual data (crawled from published Scientific Citation Index papers) via contrastive learning. This AI model represents a critical attempt that directly bridges molecular graphs and natural language. Importantly, through capturing the specific and complementary information of the two modalities, our proposed model can better grasp molecular expertise. Experimental results show that our model not only exhibits promising performance in cross-modal tasks such as cross-modal retrieval and molecule caption, but also enhances molecular property prediction and possesses capability to generate meaningful molecular graphs from natural language descriptions. We believe that our model would have a broad impact on AI-empowered fields across disciplines such as biology, chemistry, materials, environment, and medicine, among others.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Molecule Captioning ChEBI-20 MoMu+MolT5-Large BLEU-2 59.9 # 7
BLEU-4 51.5 # 7
METEOR 59.7 # 12
Text2Mol 58.2 # 4
Molecule Captioning ChEBI-20 MoMu+MolT5-Small BLEU-2 53.2 # 18
BLEU-4 44.5 # 18
METEOR 55.7 # 18
Text2Mol 55.3 # 10
Molecule Captioning ChEBI-20 MoMu+MolT5-Base BLEU-2 54.9 # 15
BLEU-4 46.2 # 14
METEOR 57.6 # 15
Text2Mol 55.8 # 8


No methods listed for this paper. Add relevant methods here