Chinese Named Entity Recognition with Graph-based Semi-supervised Learning Model
1 like•129 views
This document summarizes an experiment on using graph-based semi-supervised learning to improve a conditional random field model for Chinese named entity recognition. The experiment used unlabeled data from previous NER tasks to extend the labeled training data via label propagation. This enhanced CRF model was evaluated on a standard test corpus and showed a slight improvement over a closed CRF baseline, particularly for person and organization entities. However, the unlabeled data was not large enough to cover all entity types. Future work could explore using more unlabeled data and optimizing features for the graph construction.
Chinese Named Entity Recognition with Graph-based Semi-supervised Learning Model
- 2. Chinese Named Entity Recognition with Graph-based Semi-supervised Learning Model Aaron Li-Feng Han* Xiaodong Zeng+ Derek F. Wong+ Lidia S. Chao+ * ILLC, University of Amsterdam, The Netherlands + NLP2CT Laboratory, University of Macau, Macau S.A.R., China SIGHAN 2015 @ Beijing, July 30-31
- 3. • NER Tasks • Traditional methods • Motivation • Designed model • Experiments • Conclusion
- 4. NER Tasks NER tasks: The annotations in MUC-7 Named Entity tasks (Marsh and Perzanowski, 1998) - Entities (organization, person, and location), times and quantities such as monetary values and percentages - Languages of English, Chinese and Japanese.
- 5. The entity categories in CONLL-02 (Tjong Kim Sang, 2002) and CONLL-03 (Tjong Kim Sang and De Meulder, 2003) NER shared tasks - Persons, locations, organizations and names of miscellaneous entities - Languages span from Spanish, Dutch, English, to German. The SIGHAN bakeoff-3 (Levow, 2006) and bakeoff-4 (Jin and Chen, 2008) tasks offer standard Chinese NER (CNER) corpora - Three commonly used entities, i.e., personal names, location names, and organization names.
- 6. Traditional methods Traditional methods used for the entity recognition tend to employ external annotated corpora to enhance the machine learning stage, and improve the testing scores using the enhanced models (Zhang et al., 2006; Mao et al., 2008; Yu et al., 2008). The conditional random filed (CRF) models have shown advantages and good performances in CNER tasks as compared with other machine learning algorithms (Zhou et al., 2006; Zhao and Kit, 2008) However, the annotated corpora are generally very expensive and time consuming.
- 7. On the other hand, there are a lot of freely available unlabeled data in the internet that can be used for our researches. Due to this reason, some researchers begin to explore the usage of the unlabeled data and the semi-supervised learning methods based on labeled training data and unlabeled external data have shown their ad- vantages (Blum and Chawla, 2001; Shin et al., 2006; Zha et al., 2008; Zhang et al., 2013).
- 8. Motivation Named entity recognition (NER) plays an important role in the NLP literature The traditional methods tend to employ large annotated corpus to achieve a high performance. - However the annotated corpus is usually expensive to gain. Many semi-supervised learning models for NER task are proposed to utilize the freely available unlabeled data. Graph-based semi-supervised learning (GBSSL) methods have been employed in many NLP tasks, e.g. - sentiment categorization (Goldberg and Zhu, 2006) - question- answering (Celikyilmaz et al., 2009) - class-Instance acquisition (Talukdar and Pereira, 2010) - structured tagging models (Subramanya et al., 2010) - Joint Chinese word segmentation and part of speech (POS) tagging (Zeng et al., 2013)
- 9. Why not: GBSSL for NER Extend the labeled data Enhance the learning model , e.g.conditional random field (CRF)
- 10. Designed model • Enhanced learning with unlabelled data • GBSSL to extend labeled data • CRF learning
- 11. • Graph based semi-supervised learning • - Graph Construction & Label Propagation
- 12. • Graph Construction • - Follow the research of Subramanya et al. (2010), represent the vertices using character trigrams in labeled and unlabeled sentences for graph construction • - A symmetric k-NN graph is utilized with the edge weights calculated by a symmetric similarity function designed by Zeng et al. (2013).
- 13. • where k(i) is the k nearest neighbours of xi. sim() is a similarity measure of two vertices. The similarity is computed based on the co-occurrence statistics (Zeng et al., 2013). wi, j = sim(xi ,xj ) if j ∈k(i) or i ∈k( j) 0 otherwise ⎧ ⎨ ⎪ ⎩ ⎪
- 14. The feature set we employed to measure the similarity of two vertices based on the co-occurrence statistics is the optimized one by Han et al. (2013) for CNER tasks, as denoted in Table 1.
- 15. • Label propagation After the graph construction on both labeled and unlabeled data - Use the sparsity inducing penalty (Das and Smith, 2012) label propagation algorithm to induce trigram level label distributions from the constructed graph - Based on the Junto toolkit (Talukdar and Pereira, 2010).
- 16. Enhance CRF learning - put the propagated labeled data into the training data - based on the CRF++ toolkit - Use the same feature set as in Table 1.
- 17. Experiments Dataset - We employ the SIGHAN bakeoff-3 (Levow, 2006) MSRA (Microsoft research of Asia) training and testing data as standard setting. - To testify the effectiveness of the GBSSL method for CRF model in CNER tasks, we utilize some plain (un- annotated) text from SIGHAN bakeoff-2 (Emerson, 2005) and bakeoff-4 (Jin and Chen, 2008) as external unlabeled data. - The data set is introduced in Table 2 from the aspect of sentence number.
- 19. We set two baseline scores for the evaluation. - One baseline is the simple left-to-right maximum matching model (MaxMatch) based on the training data - Another baseline is the closed CRF model (Closed-CRF) without using unlabeled data. - The employment of GBSSL model into semi-supervised CRF learning is denoted as GBSSL-CRF.
- 20. • Training costs - The comparison shows that the extracted features grow from 8,729,098 to 11,336,486 (29.87%) due to the external dataset, and the corresponding iterations and training hours also grow by 12.86% and 77.04% respectively. - The training costs of the CRF learning stage are detailed in Table 3.
- 22. • Evaluation results - The evaluation results are shown in Table 4, from the aspects of recall, precision and the harmonic mean of recall and precision (F1- score).
- 23. - The evaluation shows that both the Closed-CRF and GBSSL-CRF models have largely outperformed baseline-1 model (MaxMatch). - As compared with the Closed-CRF model, the GBSSL- CRF model yielded a higher performance in precision score, a lower performance in recall score, and finally resulted in a faint improvement in F1 score. - Both the GBSSL-CRF and Closed-CRF show higher performance in precision and lower performance in recall value.
- 24. To look inside the GBSSL performance on each kind of entity - denote the detailed evaluation results from the aspect of F1-score in Table 5.
- 25. - The detailed evaluation from three kinds of entities shows that both the GBSSL-CRF and Closed-CRF show higher performance in LOC entity type, and lower performance in PER and ORG entities. - Fortunately, the GBSSL model can enhance the CRF learning on the two kinds of difficult entities PER and ORG with the better perfor- mances of 0.28% and 0.58% respectively. However, the GBSSL model decreases the F1 score on LOC entity by 0.19%.
- 26. The lower performance of GBSSL model on LOC entity may be due to that the unlabelled data is only as much as 62.75% of the training corpus, which is not large enough to cover the Out-of-Vocabulary (OOV) testing words of LOC entity; on the other hand, the unlabeled data also bring some noise into the model.
- 27. Conclusion and future works This paper makes an effort to see the effectiveness of the GBSSL model for the traditional CNER task. Extend data - The experiments verify that the GBSSL can enhance the state-of-the- art CRF learning models. The improvement score is a little weak because the unlabeled data is not large enough. In the future work, we decide to use larger unlabeled dataset to enhance the CRF learning model.
- 28. Feature selection: The feature set optimized for CRF learning may be not the best one for the similarity calculation in graph construction stage. So we will make efforts to select the best feature set for the measuring of vertices similarity in graph construction on CNER documents.
- 29. Corpus type: In this paper, we utilized the Microsoft research of Asia corpus for experiments. We will use more kinds of Chinese corpora for testing, such as CITYU and LDC corpus, etc.
- 30. Performance on OOV words: The GBSSL model generally improves the tagging accuracy of the Out- of-Vocabulary OOV) words in the test data, which are unseen in the training corpora. In the future work, we plan to give a detailed analysis of the GBSSL model performance on the OOV words for CNER tasks.
- 31. • Thanks for your attention! View publication statsView publication stats