Download as PDF, PPTX
More Related Content
Digital Types
- 1. CHAPTER 01: TYPESOF DIGITAL DATA
- 2. Data • Any datathat can be processed by digital computer and stored in the sequences of 0's and 1's (Binary language) is knowns as digital data. • Whenever you send an email, read a social media post, or take pictures with your digital camera, you are working with digital data. • In general, data can be any character, text, numbers, voice messages, SMS, WhatsApp messages, pictures, sound, or video.
- 3. Data • Byte isthe basic unit of information in computer storage and processing, and is composed of eight bits; a kilobyte is 1,000 bytes; one megabyte is 1,000 kilobytes . (GB, TB, PB, EB, ZB, YB) • Digitizing is the process of converting information into digital form and is necessary for a computer to be able to process and store the information.
- 4. Data • It isan invaluable asset of any enterprise (big or small). • Data is present internal to the enterprise and also exists outside the firewalls of the enterprise. • Data may be in homogeneous or heterogeneous. • Need of the hour is to – Understand, manage, process, – and take the data for analysis – to draw valuable insights.
- 5. Types of digitaldata 1. Structured Data : data stored in the form of rows and columns (databases, Excel) 2. Un-structured Data: No pre-defined schema (PPTs, images, Videos, pdfs) 3. Semi-structured Data: Hybrid schema (JSON, HTML, XML, Email, and so on),
- 7. Distribution of digitaldata (in %) (by Gartner) 80 10 10 Unstructured Semi-structured Structured
- 8. Structured Data • Datawhich is in an organized form (In rows & columns). • Computer programs can use this data easily. • Relationships exists between entities of data. • Example – Data stored in databases – ERP – CRM – DW – Data Cube
- 9. Structured Data • Thedata conforms to a pre-defined schema or structure is known as structured data. • The data can be processed, stored, and retrieved in a fixed format. This data can be processed easily by programs. • Conforms to a relational data model. • Structured data is organized in semantic chunks/entities with similar entities grouped together to form relations/tables.
- 10. structured Data • Descriptionsfor all entities in a group • Have the same defined format • Have a predefined length • Follow the same order.
- 11.
- 12. Sources of StructuredData Structured Data OLTP systems Excel Databases
- 13. Ease with structureddata Ease with structured data Security Insert/Update/ delete Scalability Transaction processing (ACID) Indexing/ Searching
- 14. Database (RDBMS) • OracleCorp. – Oracle • IBM – DB2, IBM-Informix • Microsoft – SQL • EMC – Greenplum • Teradata – Teradata • Open source- MySQL, PostgresSQL • Sqlite • Sequel Pro • Amazon Aurora • SAP SQL Anywhere, SAP IQ (Sybase)
- 15. Semi-structured Data • Datawhich does not conform to a data model but has some structure. • Computer programs can not use this data easily. • Example – emails – XML – HTML – JSON, and so on.
- 16. Semi-structured data (SSD) •It is referred to as self describing structure. • It is a form of structured data that does not conform with the formal structure of data models associated with relational databases or other forms of data tables. • It uses metadata and tags to provide semantic information.
- 17. Characteristics of semi-structureddata (SSD) • Does not conform to a data model • Cannot be stored in the form of rows and columns as in a database. • The tags and elements are used to describe data. • Attributes in a group may not be the same. • Similar entities are grouped. • Size of the same attributes in a group may differ • Type of same attributes in group may differ. • Evolving Schema • Schema and data are tightly coupled.
- 18. Example (Names &Emails) • One way is: Name: Raju Patil Email : [email protected], [email protected] • Another way is: First Name: Raju Last Name :Patil Email : [email protected]
- 20. Sources of SSD •Email • XML • TCP/IP • Zipped files • Mark-up languages • Integration of data from heterogeneous sources.
- 21. Example: Email format To:<Name> From: <Name> Subject: <Text> CC: <Name> Body: <Text, Graphics, Images, etc.><Name>
- 22. ABC Healthcare BloodTest Report Date <> ---- Department <> ----- Patient Name <> Attending Doctor <> Hemoglobin content <> Patient Age <> RBC count <> WBC count <> Platelet count <> Diagnosis <notes> Conclusion <notes>
- 23.
- 24. Integration of datafrom heterogeneous sources User Mediator : Uniform access to multiple data sources OODBMS RDBMS Legacy system Structured file
- 26. Getting to knowUnstructured data • Over the past few days, Dr. Ben and Dr. Stanley had been exchanging long emails about a particular case of gastro-intestinal problem. • Email contains procedure practiced by Dr. Stanley, about combination of drugs that has successfully cured gastro-intestinal disorders in patients. • Dr. Mark has a patient in the “GoodLife” emergency unit with quite similar case of gastro- intestinal disorder.
- 27. Unstructured Data • Unstructureddata refers to the data that lacks any specific form or structure. • This makes it very difficult and time-consuming to process and analyze unstructured data. • Data which does not conform to any data model is USD. • Computer programs can not use this data directly. • About 80-90% data of an organization is in this format. • An enormous amount of knowledge is hidden in this data. • Hence finding useful knowledge/insight from USD is very crucial.
- 28. Unstructured Data • Unstructureddata is a generic label for describing data that is not contained in a database or some other type of data structure. • Unstructured data can be textual or non-textual. • Textual unstructured data is generated in media like email messages, PowerPoint presentations, Word documents, comments in social media, etc. • Non-textual unstructured data is generated in media like images, CCTV footage, audio files and video files. • Anything in a non-database form is unstructured data.
- 29. Unstructured Data • Twotypes: 1. Bitmap objects : image, video, or audio files 2. Textual objects : word, emails, ppts and so on.
- 30. Unstructured Data • Example –Memos, QR code (Quick Response), Blogs – Chat rooms, Tweets, Comments, likes, tags – PPTs, emoji's, emoticons (emotion icons) – Images, log files, social media posts – Videos, sensor data (raw), weather data – Doc files, geospatial data, surveillance data – Body of email , GPS data, sensor data, etc. – WhatsApp messages, CCTV footage and so on.
- 31. Getting to knowUnstructured data
- 32. Characteristics of Unstructureddata • This data cannot be stored in the form of rows and columns as in a database and does not conform to any data model. • It is difficult to determine the meaning of the data. • It does not follow any rule or semantics, i.e. Not in any particular format or sequence. • Not easily usable by a program.
- 33. Sources of Unstructureddata • Web pages • Audio and Videos • Images • Body of an email • Word document • PPT and reports • Chats and text messages • Social media data • White papers • Surveys • SMS • Free form text • Server Log files • Product reviews
- 34. Web page isunstructured data Web Page Multimedia Image Database Text XML
- 35. Challenges • Storage space:A lot of space is required to store USD. • Scalability: As the data grows, scalability becomes an issue and the cost of storing USD increases. • Retrieve information: Difficult to retrieve required information from USD • Security: Ensuring security is difficult due to varied sources of data. E.g. emails, web pages, etc. • Indexing & searching: Very difficult and error-prone as the structure of the USD is not clear.
- 36. Challenges • Interpretation :USD is not easily interpreted by conventional search algorithms. • Classification : Different naming conventions followed across the organization make it difficult to classify data. • Deriving meaning : Computer programs cannot automatically derive meaning or structure from USD. • File formats : Increasing number of file formats makes it difficult to interpret data.
- 37. Portion of Unstructureddata USD SD
- 38. Dealing with USD 1.Data mining 2. Text mining /Text Analytics 3. NLP 4. Noisy text analytics 5. Manual tagging with meta data 6. Part of speech tagging 7. UIMA 8. Web Scraping Possible Solutions
- 39. Data Mining • Itis the computing process of discovering patterns in large data sets involving methods at the intersection of AI, machine learning & DL, statistics, and database systems. • Popular algorithms: – Association rule mining (MBA) – Regression Analysis (Y=mX+ c) – Collaborative filtering
- 40. Collaborative filtering • collaborativefiltering uses similarities between users and items simultaneously to provide recommendations. • It is a method of making automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating). • Collaborative filtering works on a fundamental principle: you are likely to like what someone similar to you likes.
- 41. Collaborative filtering • Collaborativefiltering (CF) is a technique commonly used • Collaborative filtering (CF) is a technique used by recommender systems to build personalized recommendations on the Web. • Companies that employ CF model include Amazon, Facebook, Twitter, LinkedIn, Spotify, Google News, Netflix, iTunes.
- 42.
- 43. Text analytics ortext mining • It is the process of converting unstructured text data into meaningful data for analysis, to measure customer opinions, product reviews, feedback and sentimental analysis to support fact based decision making. • Uses many linguistic, statistical, and machine learning techniques such as clustering, pattern recognition, tagging, association analysis, predictive analytics, etc.
- 44. Text analytics ortext mining • It helps organizations to find potentially valuable business insights in corporate documents, customer emails, call center logs, survey comments, social network posts, medical records and other sources of text-based data. • Text mining capabilities are also being incorporated into AI chatbots/virtual agents that companies deploy to provide automated responses to customers as part of their marketing, sales and customer service operations.
- 45. Natural Language Processing(NLP) • Natural language processing (NLP) is the ability of a computer program to understand human language as it is spoken. NLP is a component of artificial intelligence (AI). • It is a field of computer science, artificial intelligence and computational linguistics concerned with the interactions between computers and human (natural) languages (HCI domain). • NLP strives to build machines that understand and respond to text or voice data.
- 46.
- 47. Noisy text analytics •It is the process of extracting structured or semi- structured information from noisy unstructured text data such as online chat, text messages, emails, message boards, blogs, wikis, etc. • The noisy unstructured data comprises one or more of the followings: – Spelling mistakes, – Acronyms – Non-standard words (HBD, K, GN, GM, VGM, etc.) – Missing punctuations, – Missing letters and so on.
- 48. Manual tagging withmetadata • It is the process of tagging manually with adequate metadata to provide the semantics to understand unstructured data. . Road Accident
- 49. Part of SpeechTagging • It is also called as POS or POST or grammatical tagging. • It is the process of reading text and tagging each word in the sentence as belonging to a particular part of speech such as “noun”, “verb”, “adjective”, “pronoun”, etc. .
- 50. Unstructured Information Management Architecture(UIMA) •It is an open source platform from IBM, which integrates different kinds of analysis engines to provide a complete solution for knowledge discovery from USD. • It bridge the gap between structured and USD.
- 51. Uses of UIMA •Used to convert unstructured data such as repair logs and service notes into relational tables. • These tables can then be used by automated tools to detect maintenance or manufacturing problems.
- 52. Uses of UIMA •Used in medical contexts to analyze clinical notes, such as the Clinical Text Analysis and Knowledge Extraction System ( Apache CTAKES). • CTAKES is an open-source Natural Language Processing (NLP) system that extracts clinical information from electronic health/medical record free-text (Users are free to type whatever they want in any form).
- 53. UIMA block diagram Users Acquiredfrom various sources Subjected to semantic analysis Structured information access Query and presentation Structured information Analysis Delivery USD Transformed into
- 54.
- 56. Big Data • Bigdata is a term that describes large, hard- to-manage volumes of data – both structured and unstructured - none of traditional data management tools can store it or process it efficiently. • experts now predict that 74 zettabytes of data will be in existence by 2021.
- 57. Big Data • Everyday, we create 2.5 quintillion(1018) bytes of data —90% of the data in the world today has been created in the last two years alone. • This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, WhatsApp, IOT and so on.
- 58. Characteristics of Data •Composition: Deals with structure of data, i.e., sources of data, the granularity(Ex. Postal address), the types, nature of data (Static or real- time). • Condition: Deals with the state of data, that is, “Can one use data as it is for analysis?” or “Does it require cleansing for further enhancement and enrichment?”.
- 59. Characteristics of Data •Context: Deals with – Where, this data has been generated? – Why this data generated? – How sensitive is this data? – What are the events associated with this data? – And so on.
- 60. Gartner • Is aglobal research and advisory firm providing insights, advice, and tools for leaders in IT, Finance, HR, Customer Service and Support
- 61. Big data definition-Gartner • Big data is high-volume, high-velocity, and high- variety information assets that demand cost effective, innovative forms of information processing for enhanced insight and decision making. • Cost effective and innovative forms of information processing: Talks about embracing new techniques and technologies to capture, store, process, persevere, integrate and visualize the big data(3vs).
- 62. Definition of Bigdata by Gartner • Enhanced insight and decision making: Talks about deriving deeper, richer, and meaningful insights and then using these insights to make faster and better decisions to gain business value and thus a competitive edge.
- 63.
- 64. Challenges with BigData • Capture • Storage (Solution: Cloud Computing) • Curation ( Management of data + Data retention) • Search • Analysis • Transfer • Visualization • Privacy violations
- 65.
- 66. 3 V’s ofBig data • The data that is big in Volume, Velocity and Variety is known as big data.
- 67. Sources of bigdata • Archives: Archives of scanned documents, customer correspondence records, patient’s health records, student’s admission records, students’ assessment records and so on. • Sensor data: Car sensors, smart electric meters, office buildings, washing m/c, other electronic appliances and so on. • Machine log data: Event logs, application logs, audit logs, server logs, etc.
- 68. Sources of bigdata • Public web: Wikipedia, Weather, regulatory, census, etc. • Data storage: File systems, SQL database, NoSQL database (Mongo DB, Cassandra) and so on. • Media: Audio, Video, image, etc. • Docs: CSV, word docs, PDF, PPT, XLS, etc. • Business Apps: ERP, CRM, HR, Google Docs, etc. • Social media: Twitter blogs, Facebook, LinkedIn, YouTube, Instagram, etc. • IOT
- 69. Other characteristics ofbig data • Veracity and Validity: Refers to the accuracy (quality) and correctness of the data. • Volatility: Deals with how long the data is valid?, and how long should it be stored?. (OTP, Aadhar No., PW) • Variability: Data flows can be highly inconsistent with periodic peaks. (In total 7V’s of big data)
- 70. Why Big data Moreconfidence in decision making More Data More Accurate analysis Greater operational efficiency, cost reduction, time reduction, new product development, optimized offerings, etc.
- 71. Three reasons forleveraging big data 1. Competitive Advantage. 2. Decision making 3. To create new business value out of data.
- 72. Typical data warehouseEnvironment
- 73. Typical Hadoop Environment •It is different from DW environment. • Here data sources are web logs, images, audios, videos, social media, doc files, pdfs, etc.
- 74.
- 78. Big data &DW coexistence
- 79. Big data &DW coexistence