Identifying Sensible Lexical Relations in Generated Stories

As with many text generation tasks, the focus of recent progress on story generation has been in producing texts that are perceived to {``}make sense{''} as a whole. There are few automated metrics that address this dimension of story quality even on a shallow lexical level. To initiate investigation into such metrics, we apply a simple approach to identifying word relations that contribute to the {`}narrative sense{'} of a story. We use this approach to comparatively analyze the output of a few notable story generation systems in terms of these relations. We characterize differences in the distributions of relations according to their strength within each story.