5a use of annotated corpus
- 1. USE OF ANNOTATED CORPUS Thennarasu Sakkan
- 2. Annotated Text Corpora is an important resource for advances in NLP research and for developing different language technologies. The annotation of corpora is done using a set of tags, which mark the linguistic properties of a word, sentence or discourse. The corpora annotated with various linguistic information not only forms a precious resource for language technologies but also involves large amount of effort and time.
- 3. Therefore, it is important to create corpora which once created can be used for various purposes. Layered approach It was proposed to follow a layered approach. Some of the layers are: Layer 1: Morphology Layer 2: POS <morphosyntactic> Layer 3: LWG Layer 4: Chunks Layer 5: Syntactic Analysis Layer 6: Thematic roles/Predicate Argument structure Layer 7: Semantic properties of the lexical items Layers 8,9,10,11: Word sense, Pronoun referents (Anaphora), etc, etc
- 4. Example, ((My younger sister Suguna))_NP ((will be coming))_VP ((from Tamil Nadu))_PP ((early this month))_NP.
- 5. ((செவ்ாில்_NNP))_NP ((சற்நிகாக_RB))_RBP ((ரார்_NNP ிண்கனம்_NN))_NP ((ரிநங்கிது_VF))_VP ! (ாொ_NNP ிஞ்ஞாணிகள்_NN))_NP ((ொரண_NN))_NP !!_RD_SYM (See here exclamation marker.) ((ியூார்க்_NNP))_NP :_RD_PUNC ((செவ்ாய்_NNP கிகத்ர_NN ஆய்வு_NN))_NP ((செய்ற்காக_RB))_RBP ((அசரிக்கா_NNP))_NP ((அனுப்தி_VNF))_VGNF (ரார்_NNP ிண்கனம்_NN))_NP ((கிட்டத்ட்ட_RB))_RBP ((8_TC? ா_NN தத்ிற்கு_NN))_NP ((திநகு_NST))_? இன்று_NST))_? (06.08.12) ((சற்நிகாக_RB))_RBP ((ரிநங்கிது_VF))_VP ((._PUNC))_? ((ிண்சபி_NN ஆய்வு_NN ரத்ில்_NN))_NP ((இது_PRP))_?? ((ஒய_TC ிகப்_INTF சதரி_JJ ரல்கல்னாக_RB??))_NP?? / RBP?? ((கயப்தடுகிநது_VF))_VP ((._PUNC))_??
- 6. ((பூிில்_NN))_NP ((இயந்து_N_NST))_NP?/N_ST? ((சுார்_RB)) ((570_TC ில்னின்_NN கி.ீ.,_NN சாரனில்_NN))_NP ((உள்பது_VF))_VGF ((செவ்ாய்_NNP கிகம்_NNP))_NP ._PUNC ((இந்_DMD கிகத்ில்_NN ஊிரிணங்கள்_NN))_NP ((ாழ்ற்காண_VNF))_VGNF ((ற்ந_JJ சூல்_NN))_NP ((இயக்கிநா_VF))_VGF ((ன்தது_CCS))_?? ((குநித்து_PSP))_?? ((ஆய்வு_NN))_NP ((செய்_VINF))_VGINF ((அசரிக்காின்_NNP ாொ_NNP ிண்சபி_NNP ஆாய்ச்ெி_NNP ரம்_NNP))_NP ((தல்ரறு_JJ))_JJP ((ஆய்வுகரப_NN))_NP ((ரற்சகாண்டு_VNF))_VGNF ((யகிநது_VF))_VGF. ((செவ்ாய்_NNP கிகம்_NN))_NP ((சாடர்தாண_JJ))_JJP ((தடங்கரபயும்_NN))_NP ((அவ்ப்ரதாது_RB))_RBP ((சபிிட்டு_VNF யகிநது_VM))_VGF ._SYM
- 7. How are corpora annotated? • Automatic annotation • Computer-assisted annotation • Manual annotation Sinclair (1992): the introduction of the human element in corpus annotation reduces consistency.
- 8. Corpus in NLP NLP is unthinkable without involving corpora. Corpora are essential ingredients of every aspects of natural language processing
- 9. a) Morph analysis – the morph features of a given word are marked. If the word has multiple morph feature sets, all are provided for it. • Morphological level –Prefixes –Suffixes –Stems - (morphological annotation) Example: pens <root=”pen” cat=”n” gender=”m” number=”pl” person=”3”>|<root=”pen” cat=”v” gender=”m” number=”sing” person=”3” tense=”present” aspect=”hab”>
- 10. Corpus Vs Morph • 10% 63 54 59 4 (Te, Ma, Ta, Hi,) • 20% 293 335 257 11 • 30% 934 1196 728 26 • 40% 2433 3439 1803 74 • 50% 5707 8810 4091 186 • 60% 13280 21663 8992 454 • 70% 31941 53718 20191 1092
- 11. b) POS a word is tagged for its POS category in a given sentence. Example: I need two <pos=”NN”>pens </pos=”NN”> to finish this article. He <pos=”VBS”> pens </pos=”VBS”> his views regularly. c) Word sense – the appropriate sense of a word in a given context is marked. Example: I need two <word_sense=”pen”> pens </word_sense=”pen”> to finish this article. He <word_sense=”write”> pens </word_sense=”write”> his views regularly.
- 12. POS Vs Corpus 11% of words in Brow corpus are ambiguous. What about our languages?
- 13. At the sentence level the information could be a) Identification of chunks/MWEs/LWGs/phrases Chunks are minimal constituent units. The chunk analysis of a sentence provides a shallow level of parsing. Thus, a corpora annotated with POS and chunks can be useful for building a shallow parser. Example, I saw a man with telescope.
- 14. • Syntactic level – parsing – treebanking – bracketing • Discourse level – Anaphoric relations (coreference annotation) – Speech acts (pragmatic annotation) – Stylistic features such as speech and thought in presentation (stylistic annotation).
- 15. Corpus Vs Machine translation parallel and comparable corpora, which include their use in lexicography, terminology extraction to build terminology databases and bilingual reference tools, pride of place must be given to machine translation (MT). parallel corpora have played a pivotal role in a (partial) paradigm shift from rule-based approaches to statistical and example-based approaches to MT.
- 16. Essentially, statistical MT (SMT) involves computing the probability that a TL string is the translation of an SL string, based on the frequency of the co- occurrence of these strings in the corpus, whereas example-based MT (EBMT) involves searching for similar phrases in previous translations and extracting the TL fragments corresponding to the SL fragments.
- 17. Show demo on KWIC