Sinmin final presentation
Download as PPTX, PDF4 likes866 views
Sinmin is a corpus for the Sinhala language that is continuously updating and scalable. It covers a wide range of structured and unstructured Sinhala language data from sources like news, academic writings, fiction, and more. The architecture includes crawlers to fetch web pages, data cleaning mechanisms to handle issues like erroneous characters and short forms, and Cassandra as the main storage system. The API and user interface allow users to perform queries on the corpus to obtain word and ngram frequencies, latest articles, and more. Performance testing was conducted on the storage systems and API.
Sinmin final presentation
- 1. Sinmin - Corpus for Sinhala Language 1 Upeksha W. D. Wijayarathna D. G. C. D. Siriwardena M. P. Lasandun K. H. L. Supervisors : Dr. Chinthana Wimalasuriya Prof. Gihan Dias Mr. N. H. N. D. de Silva
- 2. Outline ● Introduction ● Crawler Implementation and Design ● Data Cleaning and Tokenizing Mechanisms ● Selecting Data Storage Mechanism ● Data Storage Model of SinMin ● User Interface Design and Implementation ● API Design and Implementation ● Unit Testing ● Performance Testing of the API ● Implemented Sample Usages 2
- 3. What is a Corpus?? “A corpus is a principled collection of authentic texts stored electronically that can be used to discover information about language that may not have been noticed through intuition alone.” - Bennet (2010) 3
- 4. Usages of a Corpus ● Implementing translators, spell checkers and grammar checkers. ● Identifying lexical and grammatical features of a language. ● Identifying varieties of language of context of usage and time. ● Retrieving statistical details of a language. ● Providing backend support for tools like OCR, POS Tagger, etc. 4
- 5. Sinmin is a Corpus for Sinhala language which is ➢ Continuously updating ➢ Dynamic (Scalable) ➢ Covers wide range of language (Structured and unstructured) 5
- 7. Identified Sinhala Resources 7 News Academic Creative Writing Spoken Gazette News Paper Text books Fiction Subtitle Gazette News Items Religious Blogs Wikipedia Magazine mahawansa
- 10. Crawlers are responsible of finding web pages that contain sinhala content, fetching, parsing and storing them in a manageable format. 10
- 12. Sample Xml File With One Article Stored In It 12
- 13. Crawler Controller Crawler controller monitors and handles the status of the web crawlers. 13
- 14. 14
- 15. Data Cleaning and Tokenizing Mechanisms used 15
- 16. Identified Issues ● Erroneous characters of the texts ● Short forms ● Consecutive Sinhala vowel sign problem fixing 16
- 17. Erroneous Characters Of The Texts ● Invalid Unicode characters Eg: Characters in a private user area, Replacement character ● Symbols Eg: “,”, “.”, “{“, “(“, “?” 17
- 18. Erroneous Characters Of The Texts ● Unwanted non-Sinhala characters Eg: ‘u+200C’, Á, À, ®, ¡, ª, º ● Non-symbolic characters which were terminating words 18
- 19. Short Forms ● Short forms consists of full stops. ● But those full stop marks aren’t separating sentences nor words. E.g.: පෙ. ව. (pm), රු. (Rupees) 19
- 20. Identified Common Short Forms "ඒ.", "බී.", "සී.", "ඩී.", "ඊ.", "එෆ්." "පෙ.", "ව.", "ෙ.", "රු." "0.", "1.", "2.", "3.", "4.", "5.", "6.", "7.", "8.", "9." 20
- 21. Consecutive Sinhala Vowel Sign Problem 21
- 22. Consecutive Sinhala Vowel Sign Problem ● Solution: Mapping them into one format ● Convention: Only one vowel sign to a Sinhala letter 22
- 23. Consecutive Sinhala Vowel Sign Problem 23
- 24. Selecting Data Storage Mechanism for Sinmin 24
- 25. The performance of data insertion and retrieval mainly depend on the Data Storage Mechanism used for the Corpus. 25
- 26. We tested performance of several database systems to determine what should we use to store data. 26
- 27. We Considered Following Data Storage Systems 27
- 28. We considered performance for inserting data and for retrieving 12 different information needs. Data set and source code https://github.com/madurangasiriwardena/performance-test 28
- 29. Data Insertion Time Comparison 29
- 30. Information Retrieval Performance Comparison - Part 1 30
- 31. Information Retrieval Performance Comparison - Part 2 31
- 32. Cassandra performed better than others in most of the scenarios, and its insertion time increased linearly. So we chose it for implementing corpus. 32
- 33. Data Storage Model of Sinmin 33
- 34. ● We Used Cassandra as the Main Storage System of Sinmin ● Apache Cassandra version 2.1.2 used. ● cqlsh version 5.0.1 used 34 Cassandra
- 35. ● Most queries of API are retrieved from Cassandra Database. ● Cassandra Database consist of more than 50 Column Families where each of them provides a specific information need 35 Cassandra
- 36. ● Oracle used as a backup storage server. 36 Oracle
- 37. Oracle Schema 37
- 38. Wildcard Search Feature Wildcard search feature enables users to run wild- card queries on the corpus Eg: පෙ? ෙහ* 38
- 39. Wildcard Search Feature ● Implemented using Apache Solr ● More than 1.2 million distinct words ● Supports at most 10 asterisks and atmost 10 question marks 39
- 40. Sinhala Vowel Sign Problem At Wildcard Search In Sinhala Unicode, Sinhala vowel signs are separate Unicode characters 40
- 41. Sinhala Vowel Sign Problem At Wildcard Search Solution: Represent Sinhala letter and vowel sign as one entity 41
- 42. User Interface Design and Implementation 42
- 43. ● Web interface of Sinmin has been designed for users who would prefer a visualised and summarized view of statistical data of Sinmin. ● Visual design of the interface has been made in a way that any user without prior experience of the interface is able to fulfill his information requirements with little effort. 43
- 44. Sinmin user interface allows to, ● Find the probability of an n-gram ● Find the most probable word comes after an n-gram ● Compare the usage of n-grams ● Find statistics of words, bigrams and trigrams ● Wildcard search ● Find latest articles for an n-gram 44
- 45. 45
- 46. 46
- 47. API Design and Implementation 47
- 48. REST API ● REST API to expose Corpus services ● Much complex and customizable data retrieval and filtering ● Interface for third party applications to consume 48
- 49. REST API ● Depends on backend databases (Cassandra, Oracle, Solr) ● Cassandra acts as main storage system ● Oracle is used as a backup database ● Solr is used for wildcard search functions 49
- 50. Architecture 50
- 51. API Functions ● wordFrequency ● bigramFrequency ● trigramFrequency ● frequentWords ● frequentBigrams ● frequentTrigrams ● latestArticlesForWord ● latestArticlesForBigram ● latestArticlesForTrigram 51 ● frequentWordsAroundWord ● frequentWordsInPosition ● frequentWordsInPositionReverse ● frequentWordsAfterWordTimeRange ● frequentWordsAfterBigramTimeRange ● wordCount ● bigramCount ● trigramCount
- 52. Performance Testing of the API 52
- 53. Throughput Under Different Load Conditions 53
- 54. Time Taken To Process Requests Under Different Load Conditions 54
- 55. Full Stop Predictor For OCR ● One challenge in OCR development is identifying fullstops. ● This tool is a consumer application of Sinmin that predicts the full stop marks of Sinhala texts. 55
- 56. Publications ● Implementing a Corpus for Sinhala Language - Symposium on Language Technology for South Asia (Presented) ● Comparison between performance of various database systems for implementing a language corpus – 11th International Beyond Databases, Architectures and Structures conference (Accepted) 56
- 57. Future Works ● Annotate Words with POS Taggers and lemmas. ● Implement tools and applications that make use of the corpus 57
- 58. Q & A 58
- 59. Thank You! 59