Step by step: a hierarchical framework for multi-hop knowledge graph reasoning with reinforcement learning

Recently, knowledge graph reasoning has sparked great interest in research community, which aims at inferring missing information in triples and provides critical support to various tasks (e.g., question answering and recommendation). To date, multi-hop reasoning is a dominant approach which infers the target answer by walking along the path connecting entities and relations, ensuring both accuracy and interpretability. However, in most knowledge graphs, there are multiple relations related to an identical entity, and multiple tail entities for an identical pair of head entity and relation. Due to this one-to-many dilemma, enlarged action space and ignoring logical relationship between entity and relation increase the difficulty of learning. In order to deal with such an issue, this work presents a novel paradigm for knowledge graph reasoning by decomposing it to a two-level hierarchical decision process. We apply the hierarchical reinforcement learning framework which dismantles the task into a high-level process for relation detector and a low-level process for entity reasoning, respectively. In this way, the action space is effectively controlled where the policies can be optimized. The interactions between entity and relation decision enhance the rationality of reasoning. Moreover, we introduce a dynamic prospect mechanism for lowlevel policy where the information can guide us to a refined and improved action space, assisted by embedding based method. Our proposed model is evaluated on four benchmark datasets and the results validate its superiority over state-of-the-art baselines, showing the interpretability of reasoning process simultaneously.