Temporal Word Analogies: Identifying Lexical Replacement with Diachronic Word Embeddings

This paper introduces the concept of temporal word analogies: pairs of words which occupy the same semantic space at different points in time. One well-known property of word embeddings is that they are able to effectively model traditional word analogies ({``}word $w_1$ is to word $w_2$ as word $w_3$ is to word $w_4${''}) through vector addition. Here, I show that temporal word analogies ({``}word $w_1$ at time $t_\alpha$ is like word $w_2$ at time $t_\beta${''}) can effectively be modeled with diachronic word embeddings, provided that the independent embedding spaces from each time period are appropriately transformed into a common vector space. When applied to a diachronic corpus of news articles, this method is able to identify temporal word analogies such as {``}Ronald Reagan in 1987 is like Bill Clinton in 1997{''}, or {``}Walkman in 1987 is like iPod in 2007{''}.