Improving Chinese Word Segmentation with Wordhood Memory Networks

Contextual features always play an important role in Chinese word segmentation (CWS). Wordhood information, being one of the contextual features, is proved to be useful in many conventional character-based segmenters. However, this feature receives less attention in recent neural models and it is also challenging to design a framework that can properly integrate wordhood information from different wordhood measures to existing neural frameworks. In this paper, we therefore propose a neural framework, WMSeg, which uses memory networks to incorporate wordhood information with several popular encoder-decoder combinations for CWS. Experimental results on five benchmark datasets indicate the memory mechanism successfully models wordhood information for neural segmenters and helps WMSeg achieve state-of-the-art performance on all those datasets. Further experiments and analyses also demonstrate the robustness of our proposed framework with respect to different wordhood measures and the efficiency of wordhood information in cross-domain experiments.

PDF Abstract


  Add Datasets introduced or used in this paper

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Chinese Word Segmentation AS WMSeg + ZEN F1 96.62 # 2
Chinese Word Segmentation CITYU WMSeg + ZEN F1 97.93 # 1
Chinese Word Segmentation CTB6 WMSeg + ZEN F1 97.25 # 3
Chinese Word Segmentation MSR WMSeg + ZEN F1 98.40 # 3
Chinese Word Segmentation PKU WMSeg + ZEN F1 96.53 # 4


No methods listed for this paper. Add relevant methods here