EDA_Revision_Session_1cdeba87-6912-4236-ba3b-079a5463bf00.pptx
- 2. EditEdit MasterMaster texttext stylesstyles Course : Exploratory Data Analysis Lecture On : Overview of EDA 2
- 3. Agendas 3 - We will learn the concepts of Exploratory Data Analysis or EDA. - Topics to be covered in this session: - Data Sourcing - Web Scraping - Data Cleaning - Univariate Analysis - Bivariate Analysis
- 4. What is Exploratory Data Analysis (EDA)? In simple terms, it means trying to understand the given data much better, so that we can make some sense out of it. EDA is arguably the most important and revelatory step in any kind of data analysis. In statistics, exploratory data analysis is an approach to analyzing data sets to summarize their main characteristics, often with visual methods. A statistical model can be used or not, but primarily EDA is for seeing what the data can tell us beyond the formal modeling or hypothesis testing task. 4
- 5. DATA SOURCING Data Sourcing is the process of finding and loading the data into our system. Depending upon its source, data can be classified into two types: DATA 1 1 PRIVATE DATA PUBLIC DATA
- 6. PRIVATE DATA 1 1 PUBLIC DATA Private data is given by private organizations. There are some security and privacy concerns attached to it. This type of data is used for mainly organizations internal analysis. This type of Data is available to everyone. We can find this in government websites and public organizations etc. Anyone can access this data, we do not need any special permissions or approval. Who uses data? Banking, Telecommunications, HR Industries, Advertisers, Media and Retail Industries
- 7. Categorize the data given below into public or private. 1. Commodity prices 2. Medical data 3. User input 4. Population of a place 5. Contact information 6. Website traffic 7. Stock market prices 8. Census data 1 1 QUESTION:
- 8. Categorize the data given below into public or private. 1. Commodity prices - public 2. Medical data - private 3. User input - private 4. Population of a place - public 5. Contact information - private 6. Website traffic - private 7. Stock market prices - public 8. Census data - public 1 1 SOLUTION:
- 9. •Need for and application of web scraping •The basics of an HTML page: •Inspect element •Attributes and tags •Class, ID, href •Heading, paragraph, hyperlink, div, span •Python libraries and code for web scraping •Request and BeautifulSoup •Reading, Parsing, Extraction, Transformation 1 1 WEB SCRAPING
- 10. DATA CLEANING 1 1 After completing the Data Sourcing, the next step in the process of EDA is Data Cleaning. It is very important to get rid of the irregularities and clean the data after sourcing it into our system. There could be different kinds of errors in the data like formatting errors, missing values, repeated rows, spelling inconsistencies, etc. These issues could make it difficult to analyse data and could lead to errors or irrelevant results. Thus, these issues need to be corrected before data is analysed.
- 11. There is no single structured process for Data Cleaning, but we will study in the following steps: 1 1
- 12. 1. Fixing rows and columns Click to add Title • Practice in teams of 4 students • Industry expert mentoring to learn better • Get personalised feedback for improvements 23/05/19 5 1 1 Checklist for fixing rows ● Delete summary rows: Total, subtotal rows ● Delete incorrect rows: Header rows, footer rows ● Delete extra rows: Column number, indicators, blank rows, page number Checklist for fixing columns ● Merge columns for creating unique identifiers if needed: e.g., merge state and city details into the full address. ● Split columns for more data: Eg: Split the address to get state and city details so that you can analyse each separately. ● Add missing column names. ● Rename columns consistently, with abbreviations and encoded columns. ● Delete unnecessary columns. ● Align misaligned columns: The data set might have shifted columns. 1. Fixing rows and columns Checklist for fixing rows ● Delete summary rows: Total, subtotal rows ● Delete incorrect rows: Header rows, footer rows ● Delete extra rows: Column number, indicators, blank rows, page number Checklist for fixing columns ● Merge columns for creating unique identifiers if needed: e.g., merge state and city details into the full address. ● Split columns for more data: Eg: Split the address to get state and city details so that you can analyse each separately. ● Add missing column names. ● Rename columns consistently, with abbreviations and encoded columns. ● Delete unnecessary columns. ● Align misaligned columns: The data set might have shifted columns.
- 13. 2. Missing Values 1 1 If there are missing values in the Dataset before doing any analysis, we need to handle those missing values. How to deal with missing values: ● Set values as missing values - Identify the values that indicate missing data (NA, 99) and replace such data with a blank cell ● Adding is good, exaggerating is bad - If you can not find information from a reliable external source, it is better to keep the missing values as such
- 14. ● Delete rows, columns - Delete rows if the number of missing values is insignificant and would not impact the analysis. ● Fill partial missing values using business judgement - Fill the easily identifiable values such as time zones, country codes etc. Columns can be removed if the missing values are significant in number. 1 1 NOTE: It is important to remember that it is always better to let missing values be and continue with the analysis rather than extrapolate the available information.
- 15. 3. Standardise Values 1 1 To perform data analysis on a set of values, we have to make sure the values in the same column should be on the same scale. For example, if the data contains the values of the top speed of different companies’ cars, then the whole column should be either in meters/sec scale or miles/sec scale. Things to keep in mind for standardising variables: ● Standardise units ● Scale values if required ● Standardise precision ● Remove outliers NOTE: An outlier may disproportionately affect the results of your analysis. This may lead to faulty interpretations. One must be careful not to let an outlier affect their analysis.
- 16. 4. Invalid Values 1 1 A data set can contain invalid values in various forms. For example: a string 'tr8ml' in a column containing mobile numbers would make no sense. Similarly, a height of 11 feet would be an invalid value in a set containing the heights of children. Hence these must be identified and removed from the dataset. How to deal with invalid values: ● Convert incorrect data types : Eg: string to number: '12,300' to '12300' ● Correct values that go beyond range : Correct values which are beyond logical range ● Correct values that are not in the list : Remove values that don’t belong to a list ● Correct wrong structure : Values that don’t follow a defined structure can be removed ● Validate internal rules : If there are internal rules, they should be correct and consistent.
- 17. 5. Filtering Data 1 1 You might not need the entire data set for your analysis. It is important to understand what you need to infer from the data and then choose the relevant parts of the data set for your analysis. This is where filtering of data comes into play. When filtering data: ● Deduplicate data: Remove identical rows or remove rows where some cells are identical. ● Filter rows: Filter by segment and date period to get only the rows relevant to the analysis. ● Filter columns: Pick the columns that are relevant to the analysis. ● Aggregate data: Group the data by the required keys and aggregate the rest.
- 18. QUESTION: 1 1 What is the use of data cleaning? A. to remove the noisy data B. correct the inconsistencies in data C. transformations to correct the wrong data D. All of the above
- 19. QUESTION:(Answer) 1 1 What is the use of data cleaning? A. to remove the noisy data B. correct the inconsistencies in data C. transformations to correct the wrong data D. All of the above
- 20. UNIVARIATE ANALYSIS 1 1 If we analyze data over a single variable/column from a dataset, it is known as univariate analysis. DATA DESCRIPTION Metadata, in simple terms, is the data that describes each variable in detail. Eg: size of the data set, how and when the data set was created, what the rows and variables represent.
- 21. Types of variables 1 1 1. Categorical Variables ● Unordered (nominal variable) An unordered variable is a categorical variable that has no defined order. Eg: Type of loan taken by a person = home, personal, auto, etc. ● Ordered (ordinal variable) Ordered variables are those variables that have a natural rank of order. Eg: Salary = High-Medium-low 2. Quantitative/ Numeric variables These are simply numeric variables, which can be added up, multiplied, divided, etc. Eg: salary, number of bank accounts, runs scored by a batsman, the mileage of a car, etc.
- 22. Unordered Categorical Variables ● Plots are immensely helpful in identifying hidden patterns in data. ● It is possible to extract meaningful insights from unordered categorical variables using rank-frequency plots. ● Rank-frequency plots of unordered categorical variables, when plotted on a log-log scale, typically result in a power law distribution. Eg: 1 1
- 23. Ordered Categorical Variables ● Derive insights from ordered categorical variables through univariate analysis Eg: 1 1
- 24. Quantitative Variables 1 1 Mean: This is the sum of all the data values, divided by the total number of sample values. Mode: In your sample data, the value that has the highest frequency is the mode. Median: If you arrange the sample data in ascending or descending order of frequency, the value in the middle (50th percentile) is called the median. Standard Deviation: Measures of the amount of variation or dispersion of a set of values. Interquartile Range: It is calculated as the difference between the 75th percentile and 25th percentile values. NOTE: The interquartile range is a much better metric than standard deviation if there are outliers in the data. This is because the standard deviation will be influenced by outliers, whereas the interquartile range will simply ignore them.
- 25. QUESTION: 1 1 Univariate data is a collection of information characterized by or depending on: 1. Only one random variable 2. Two independent variables 3. An independent and dependent variable 4. Two or more variables
- 26. SOLUTION: 1 1 Univariate data is a collection of information characterized by or depending on: 1. Only one random variable 2. Two independent variables 3. An independent and dependent variable 4. Two or more variables
- 27. BIVARIATE ANALYSIS 1 1 If we analyze data by taking two variables/columns into consideration from a dataset, it is known as bivariate analysis. It is understanding the relationship between two variables. We have: ● Bivariate analysis for continuous variables ● Bivariate analysis for categorical variables
- 28. Bivariate Analysis on Continuous Variables 1 1 Correlation is a number between -1 and 1, which quantifies the extent to which two variables ‘correlate’ with each other. ● If one variable increases as the other increases, the correlation is positive. ● If one variable decreases as the other increases, the correlation is negative. ● If one variable stays constant as the other varies, the correlation is zero. A perfect positive correlation means that the correlation coefficient is exactly 1. A perfect negative correlation means that two variables move in opposite directions with the fixed proportion. A zero correlation implies no relationship at all.
- 29. Click to add Title • Practice in teams of 4 students • Industry expert mentoring to learn better • Get personalised feedback for improvements 23/05/19 5 1 1
- 30. 1 1
- 31. Bivariate Analysis on Categorical Variables This type of analysis is used to quantify and understand the relationship between pairs of categorical variables. There are two fundamental aspects of analysing categorical variables: 1. To see the distribution of two categorical variables For example, if you want to compare the number of boys and girls who play games, you can make a ‘cross table’. 1 1
- 32. 2. To see the distribution of two categorical variables with one continuous variable. For example, you saw how a student’s percentage in science is distributed based on the father’s occupation (categorical variable 1) and the income level (categorical variable 2). 1 1
- 33. Bivariate Analysis in Python 1. Numerical-Numerical Analysis (correlation matrix, scatter plots) 2. Correlation vs Causation 3. Numerical-Categorical Analysis (Box plot) 4. Categorical-Categorical Analysis (Bar graph) 5. Multivariate Analysis (Heat map) 1 1