Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events

Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model~(TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a $3.1\%$ increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.

PDF Abstract NAACL 2021 PDF NAACL 2021 Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Procedural Text Understanding ProPara TSLM (Our Model) Sentence-level Cat 1 (Accuracy) 78.81 # 1
Sentence-level Cat 2 (Accuracy) 56.798 # 1
Sentence-level Cat 3 (Accuracy) 40.9 # 2
Document level (P) 68.4 # 2
Document level (R) 68.9 # 1
Document level (F1) 68.6 # 1


No methods listed for this paper. Add relevant methods here