Lesson 2 data preprocessing

© Simplilearn. All rights reserved.
Lesson 2: Data Wrangling and Manipulation
Machine Learning

Concepts Covered
Data exploration techniques
Data wrangling techniques
Data acquisition
Data manipulation techniques
Typecasting

Learning Objectives
Demonstrate different data wrangling techniques and their significance
Perform data manipulation in python using coercion, merging, concatenation,
and joins
Demonstrate data import and exploration using Python
By the end of this lesson, you will be able to:

Data Preprocessing
Topic 1: Data Exploration

Loading .csv File in Python
Program Data Program
CSV File
Code
df = pandas.read_csv("/home/simpy/Datasets/BostonHousing.csv")
Path to file
Before starting with a dataset, the first step is to load the dataset. Below is the code for the
same:

Loading Data to .csv File
Program Data
Program CSV File
Code
df.to_csv("/home/simpy/Datasets/BostonHousing.csv")
Path to file
Below is the code for loading the data within an existing csv file:

Loading .xlsx File in Python
Program Data Program
XLS File
Code
df = pandas.read_excel("/home/simpy/Datasets/BostonHousing.xlsx")
Below is the code for loading an xlsx file within python:

Program Data
Program XLS File
Loading Data to .xlsx File
Code
df.to_excel("/home/simpy/Datasets/BostonHousing.xlsx")
Below is the code for loading program data into an existing xlsx file:

Assisted Practice
Data Exploration
Problem Statement: Extract data from the given SalaryGender CSV file and store the data from
each column in a separate NumPy array.
Objective: Import the dataset (csv) in/from your Python notebook to local system.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and
password that are generated. Click on the Launch Lab button. On the page that appears, enter the
username and password in the respective fields, and click Login.
Duration: 5 mins.

Data Exploration Techniques
The shape attribute returns a two-item tuple (number of rows and the
number of columns) for the data frame. For a Series, it returns a one-item
tuple.
Code
df.shape
Dimensionality Check
Type of Dataset
Slicing and Indexing
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

Data Exploration Techniques (Contd.)
You can use the type ( ) in python to return the type of object.
Code
type(df)
Checking the type of data frame:
Checking the type of a column (çhas) within a data frame:
Code
df['chas'].dtype
Type of Dataset
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

You can use the : operator with the start index on left and end index on
right of it to output the corresponding slice.
Slicing a list:
Slicing a Data frame (df) using iloc indexer:
Code
df.iloc[:,1:3]
list = [1,2,3,4,5]
Code
list[1:3]
Type of Dataset
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

Using unique ( ) on the column of interest will return a numpy array with
unique values of the column.
Extracting all unique values out of ‘’crim” column:
Code
df['crim'].unique()
Type of Dataset
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

Using value ( ) on the column of interest will return a numpy array with all
the values of the column.
Extracting values out of ‘’crim” column:
Code
df['crim’].values()
Type of Dataset
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

Using mean( ) on the data frame will return mean of the data frame across
all the columns.
Code
df.mean()
Type of Dataset
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

Using median( ) on the data frame will return median values of the data
frame across all the columns.
Code
df.median()
Type of Dataset
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

Using mode( ) on the data frame will return mode values of the data frame
across all the columns, rows with axis=0 and axis = 1, respectively.
Code
df.mode(axis=0)
Type of Dataset
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode

©Simplilearn. All rights reserved
Let’s now consider multiple features and understand the effect of one over other with
respect to correlation (using seaborn)

Seaborn is a library for making
attractive and informative statistical
graphics in Python. It is built on top of
matplotlib and integrated with the
PyData Stack, including support for
numpy and pandas data structures, and
statistical routines.

Plotting a Heatmap with Seaborn
Code
import matplotlib.pyplot as plt
import seaborn as sns
correlations = df.corr()
sns.heatmap(data = correlations,square = True, cmap = "bwr")
plt.yticks(rotation=0)
plt.xticks(rotation=90)
Rectangular dataset (2D
dataset that can be
coerced into an ndarray)
If True, set the Axes
aspect to “equal” so
each cell will be
square-shaped
Matplotlib
colormap name or
object, or list of
colors
Below is the code for plotting a heatmap within Python:

Plotting a Heatmap with Seaborn (Contd.)
Maximum
correlation
Minimum
correlation
Below is the heatmap obtained, where, approaching red colour means maximum correlation
and approaching blue means minimal correlation.

Assisted Practice
Data Exploration
Problem Statement: Suppose you are a public school administrator. Some schools in your state of Tennessee
are performing below average academically. Your superintendent under pressure from frustrated parents and
voters approached you with the task of understanding why these schools are under-performing. To improve
school performance, you need to learn more about these schools and their students, just as a business needs to
understand its own strengths and weaknesses and its customers. The data includes various demographic, school
faculty, and income variables.
Objective: Perform exploratory data analysis which includes: determining the type of the data, correlation
analysis over the same. You need to convert the data into useful information:
▪ Read the data in pandas data frame
▪ Describe the data to find more details
▪ Find the correlation between ‘reduced_lunch’ and ‘school_rating’
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that
are generated. Click on the Launch Lab button. On the page that appears, enter the username and password in
the respective fields, and click Login.
Duration: 15 mins.

Unassisted Practice
Data Exploration Duration: 15
mins.
Problem Statement: Mtcars, an automobile company in Chambersburg, United States has recorded the
production of its cars within a dataset. With respect to some of the feedback given by their customers they are
coming up with a new model. As a result of it they have to explore the current dataset to derive further insights out if
it.
Objective: Import the dataset, explore for dimensionality, type and average value of the horsepower across all the
cars. Also, identify few of mostly correlated features which would help in modification.
Note: This practice is not graded. It is only intended for you to apply the knowledge you have gained to solve real-
world problems.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login

Data Import
Code
df1 = pandas.read_csv(“mtcars.csv“)
The first step is to import the data as a part of exploration.

Data Exploration
Type of Dataset
Identifying mean value
Code
df1.shape
The shape property is usually used to get the current shape of an
array/df.

Type of Dataset
Code
type(df1)
Data Exploration
type(), returns type of the given object.

Data Exploration
Type of Dataset
Code
df1[‘hp’].mean()
mean( ) function can be used to calculate mean/average of a given list of numbers.

Identifying Correlation Using a Heatmap
Code
import matplotlib.pyplot as plt
correlations = df1.corr()
sns.heatmap(data = correlations,square = True, cmap = “viridis")
plt.yticks(rotation=0)
plt.xticks(rotation=90)
Heatmap function in seaborn is used to plot the correlation matrix.

Identifying Correlation Using a Heatmap
From the adjacent map, you
can clearly see that
cylinder (cyl) and
displacement (disp) are the
most correlated features.
Graphical representation of data where the individual values contained in a
matrix are represented in colors.

Data Preprocessing
Topic 2: Data Wrangling

Discovering Structuring
Cleaning
Validating
Enrichment
Different Tasks
in Data
Wrangling
The process of manually converting or mapping data from one raw format into another format is called data wrangling. This
includes munging and data visualization.
Data Wrangling

Need of Data Wrangling
Missing data, a very common problem
Presence of noisy data (erroneous data and outliers)
Inconsistent data
Develop a more accurate model
Prevent data leakage
Following are the problems that can be avoided with wrangled data:

Missing Values in a Dataset
Consider a random dataset given below, illustrating
missing values.

Missing Value Detection
Consider a dataset below, imported as df1 within Python, having some missing values.
Detecting
missing
values
Code
df1.isna().any()

Missing Value Treatment
Mean Imputation: Replace the missing value
with variable’s mean
Code
from sklearn.preprocessing import Imputer
mean_imputer =
Imputer(missing_values=np.nan,strategy='mean',axis=1)
mean_imputer = mean_imputer.fit(df1)
imputed_df = mean_imputer.transform(df1.values)
df1 = pd.DataFrame(data=imputed_df,columns=cols)
df1

Mean Imputation: Replace the missing value
with variable’s mean
Median Imputation: Replace the missing
value with variable’s median
Missing Value Treatment (Contd.)
Code
from sklearn.preprocessing import Imputer
median_imputer=Imputer(missing_values=np.nan,strategy
=‘median',axis=1)
median_imputer = median_imputer.fit(df1)
imputed_df = median_imputer.transform(df1.values)
df1 = pd.DataFrame(data=imputed_df,columns=cols)
df1
Note: Mean imputation/Median imputation is again model dependent and is valid only on numerical data.

Outlier Values in a Dataset
Note: Outliers skew the data when you are trying to do any type of average.
An outlier is a value that lies
outside the usual observation of
values.
0
1
2
3
4
5
7.0 7.5 8.0 8.5 9.0 9.5 10.0 10.5 11.0 11.5 12.0 12.5 13.0 13.5 14.0 14.5 15.0
x MIDPOINT
OUTLIER?
FREQUENCY
5.0 7.5 10.0 12.5 15.0
5
12
13
11
10
9
8
7
6
Y3
X1

Dealing with an Outlier
Outlier Detection
Outlier Treatment
Detect any outlier in the first column of df1
Code
sns.boxplot(x=df1['Assignment'])
Outliers:
Values < 60

Dealing with an Outlier
Outlier Detection
Outlier Treatment
Create a filter based on the boxplot obtained
and apply the filter to the data frame
Code
filter=df1['Assignment'].values>60
df1_outlier_rem=df1[filter]
df1_outlier_rem

Assisted Practice
Data Wrangling
Problem Statement: Load the load_diabetes datasets internally from sklearn and check for any missing value or
outlier data in the ‘data’ column. If any irregularities found treat them accordingly.
Objective: Perform missing value and outlier data treatment.
respective fields, and click Login.
Duration: 15 mins.

Unassisted Practice
Data Wrangling Duration: 5 mins.
Problem Statement: Mtcars, the automobile company in the United States have planned to rework on optimizing
the horsepower of their cars, as most of the customers feedbacks were centred around horsepower. However, while
developing a ML model with respect to horsepower, the efficiency of the model was compromised. Irregularity might
be one of the causes.
Objective: Check for missing values and outliers within the horsepower column and remove them.
world problems.

Check for Irregularities
Check for missing values
Code
df1['hp'].isna().any()
Check for Outliers
Outlier
Code
sns.boxplot(x=df1['hp'])

Outlier Treatment
Code
filter = df1['hp']<250
df1_out_rem = df1[filter]
sns.boxplot(x=df2_out_rem['hp'])
Outlier filtered
data
Data with hp>250 is the outlier data. Therefore, you can filter it accordingly.

Data Preprocessing
Topic 3: Data Manipulation

Functionalities of Data Object in Python
tail( )
values( )
groupby( )
Concatenation
Merging
head( )
A data object is a two-dimensional data structure, i.e., data is aligned in a tabular fashion in rows and
columns.

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
Functionalities of Data Object in Python (Contd.)
Head( ) returns the first n rows of the data structure
Code
import pandas as pd
import numpy as np
df=pd.Series(np.arange(1,51))
print(df.head(6))

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
Tail( ) returns the last n rows of the data structure
Code
import pandas as pd
import numpy as np
print(df.tail(6))

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
values( ) returns the actual data in the series of the array
Code
import pandas as pd
import numpy as np
print(df.values)

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
Functionalities of Data Object using Python (Contd.)
The Data Frame is grouped according to the ‘Team’ and ‘ICC_Rank’ columns
Code
import pandas as pd
world_cup={'Team':['West Indies','West
indies','India','Australia','Pakistan','Sri
Lanka','Australia','Australia','Australia','
Insia','Australia'],
'Rank':[7,7,2,1,6,4,1,1,1,2,1],
'Year':[1975,1979,1983,1987,1992,1996,1999,2003
,2007,2011,2015]}
df=pd.DataFrame(world_cup)
print(df.groupby(['Team','Rank’]).groups)

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
Concatenation combines two or more data structures.
Code
import pandas
world_champions={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka’],
'ICC_rank':[2,3,7,8,4],
'World_champions_Year':[2011,2015,1979,1992,1996],
'Points':[874,787,753,673,855]}
chokers={'Team':['South Africa','New
Zealand','Zimbabwe'],'ICC_rank':[1,5,9],
'Points':[895,764,656]}
df1=pandas.DataFrame(world_champions)
df2=pandas.DataFrame(chokers)
print(pandas.concat([df1,df2],axis=1))

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
The concatenated output:

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
Merging is the Pandas operation that performs database joins on objects
Code
import pandas
champion_stats={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka'],
'ICC_rank':[2,3,7,8,4],
'Points':[874,787,753,673,855]}
match_stats={'Team':['India','Australia','West
'World_cup_played':[11,10,11,9,8],
'ODIs_played':[733,988,712,679,662]}
df1=pandas.DataFrame(champion_stats)
df2=pandas.DataFrame(match_stats)
print(df1)
print(df2)
print(pandas.merge(df1,df2,on='Team'))

tail( )
values( )
groupby( )
Concatenation
Merging
head( )
The merged object
contains all the columns
of the data frames
merged

Different Types of Joins
Left Join Right Join Inner Join Full Outer Join
Joins are used to combine records from two or more tables in a database. Below
are the four most commonly used joins:

Left Join
Code
import pandas
'ICC_rank':[2,3,7,8,4],
'Points':[874,787,753,673,855]}
Zealand','Zimbabwe'],
'ICC_rank':[1,5,9],'Points':[895,764,656]}
print(pandas.merge(df1,df2,on='Team',how='left'))
Returns all rows from
the left table, even if
there are no matches in
the right table
Left Join

Right Join
Code
import pandas
'ICC_rank':[2,3,7,8,4],
'Points':[874,787,753,673,855]}
Zealand','Zimbabwe'],'ICC_rank':[1,5,9],'Points':[89
5,764,656]}
print(pandas.merge(df1,df2,on='Team',how=‘right'))
Preserves the unmatched
rows from the second
(right) table, joining them
with a NULL in the shape
of the first (left) table
Right Join

Inner Join
Code
import pandas
'ICC_rank':[2,3,7,8,4],
'Points':[874,787,753,673,855]}
5,764,656]}
print(pandas.merge(df1,df2,on='Team',how=‘inner'))
Selects all rows from
both participating tables
if there is a match
between the columns
Inner Join

Full Outer Join
Code
import pandas
'ICC_rank':[2,3,7,8,4],
'Points':[874,787,753,673,855]}
chokers={'Team':['South Africa',’New
5,764,656]}
print(pandas.merge(df1,df2,on='Team',how=‘outer'))
Returns all records when
there is a match in either
left (table1) or right
(table2) table records
Full Outer Join

Typecasting
It converts the data type of an object to the required data
type.
Int( )
Returns an integer object
from any number or string.
string( )
Returns string from any
numeric object or converts
any number to string
float( )
Returns a floating-point
number from a number or a
string

Typecasting Using Int, float and string( )
Code
int(12.32)
Code
int(‘43’)
Code
float(23)
Code
float('21.43
')
Code
int(12.32)
Few typecasted data types

Assisted Practice
Data Manipulation
Problem Statement: As a macroeconomic analyst at the Organization for Economic Cooperation and Development
(OECD), your job is to collect relevant data for analysis. It looks like you have three countries in the north_america data
frame and one country in the south_america data frame. As these are in two separate plots, it's hard to compare the
average labor hours between North America and South America. If all the countries were into the same data frame, it
would be much easier to do this comparison.
Objective: Demonstrate concatenation.
Duration: 10 mins.

Unassisted Practice
Data Manipulation Duration: 10 mins.
Problem Statement: SFO Public Department - referred to as SFO has captured all the salary data of its employees
from year 2011-2014. Now in 2018 the organization is facing some financial crisis. As a first step HR wants to
rationalize employee cost to save payroll budget. You have to do data manipulation and answer the below questions:
1. How much total salary cost has increased from year 2011 to 2014?
2. Who was the top earning employee across all the years?
Objective: Perform data manipulation and visualization techniques
world problems.

Answer 1
Code
salary = pd.read_csv('Salaries.csv')
mean_year =
salary.groupby('Year').mean()['TotalPayBenefits']
print ( mean_year)
Check the mean salary cost per year and see how it has increased per
year.

Answer 2
Code
top_sal =
salary.groupby('EmployeeName').sum()['TotalPayBenefi
ts']
print((top_sal.sort_values(axis=0)))
Group the total salary with respect to employee name:

Key Takeaways
Demonstrate data import and exploration using Python
Demonstrate different data wrangling techniques and their significance
Perform data manipulation in python using coercion, merging,
concatenation, and joins
Now, you are able to:

Knowledge
Check

Knowledge
Check
a.
b.
c.
d.
Which of the following plots can be used to detect an outlier?
1
Boxplot
Histogram
Scatter plot
All of the above

The correct answer is
a.
b.
c.
d.
Knowledge
Check
Which of the following plots can be used to detect an outlier?
All the above plots can be used to detect an outlier.
d . All of the above
1
Boxplot
Histogram
Scatter plot
All of the above

Knowledge
Check
a.
b.
c.
d.
What is the output of the below Python code?
import numpy as np percentiles = [98, 76.37, 55.55, 69, 88]
first_subject = np.array(percentiles) print first_subject.dtype
2
float32
float
int32
float64

The correct answer is
a.
b.
c.
d.
Knowledge
Check
What is the output of the below Python code?
import numpy as np
percentiles = [98, 76.37, 55.55, 69, 88]
first_subject = np.array(percentiles)
print first_subject.dtype
Float64’s can represent numbers much more accurately than other floats and has more storage capacity.
d. float64
2
float32
float
int32
float64

Lesson-End Project
Problem Statement: From the raw data below create a data frame:
'first_name': ['Jason', 'Molly', 'Tina', 'Jake', 'Amy'], 'last_name': ['Miller', 'Jacobson', ".", 'Milner', 'Cooze'],
'age': [42, 52, 36, 24, 73], 'preTestScore': [4, 24, 31, ".", "."],'postTestScore': ["25,000", "94,000", 57, 62, 70]
Objective: Perform data processing on raw data:
▪ Save the data frame into a csv file as project.csv
▪ Read the project.csv and print the data frame
▪ Read the project.csv without column heading
▪ Read the project.csv and make the index columns as 'First Name’ and 'Last Name'
▪ Print the data frame in a Boolean form as True or False. True for Null/ NaN values and false for
non-null values
▪ Read the data frame by skipping first 3 rows and print the data frame
Access: Click the Labs tab in the left side panel of the LMS. Copy or note the username and password that are
generated. Click the Launch Lab button. On the page that appears, enter the username and password in the
respective fields and click Login.
Duration: 20 mins.

Lesson 2 data preprocessing

More Related Content

What's hot (20)

Similar to Lesson 2 data preprocessing (20)

Recently uploaded (20)

Lesson 2 data preprocessing