Data analytics using Scalable Programming

2
• Data Definitions and Analysis Techniques –
• Elements Variables and Data categorization-
• Levels of Measurement- Data management and indexing-
• Statistical learning, Descriptive Statistics
• Basic Analysis Techniques
• Data analysis techniques (Regression analysis, Classification techniques, Clustering)
Contents

3
What is Data Science?
• Data science is the study of data to extract meaningful insights.
• It is a multidisciplinary approach that combines principles and practices from the fields of mathematics,
statistics, artificial intelligence, and computer engineering to analyze large amounts of data.
• This analysis helps data scientists to ask and answer questions like what happened, why it happened, what will
happen, and what can be done with the results.
• Data science is important because it combines tools, methods, and technology to generate meaning from data.
• Modern organizations are inundated with data; there is a proliferation of devices that can automatically collect
and store information.
• Online systems and payment portals capture more data in the fields of e-commerce, medicine, finance, and
every other aspect of human life. We have text, audio, video, and image data available in vast quantities.

4
What is Data Science?
• While the term data science is not new, the meanings and connotations have changed over time. The word first
appeared in the ’60s as an alternative name for statistics. In the late ’90s, computer science professionals
formalized the term.
• A proposed definition for data science saw it as a separate field with three aspects: data design, collection, and
analysis. It still took another decade for the term to be used outside of academia.
• Artificial intelligence and machine learning innovations have made data processing faster and more efficient.
Industry demand has created an ecosystem of courses, degrees, and job positions within the field of data
science.
• Because of the cross-functional skillset and expertise required, data science shows strong projected growth over
the coming decades.

6
What is data science used for?
Data science is used to study data in four main ways:
1. Descriptive analysis
Descriptive analysis examines data to gain insights into what happened or what is happening in the
data environment. It is characterized by data visualizations such as pie charts, bar charts, line graphs,
tables, or generated narratives. For example, a flight booking service may record data like the number
of tickets booked each day. Descriptive analysis will reveal booking spikes, booking slumps, and high-
performing months for this service.
2. Diagnostic analysis
Diagnostic analysis is a deep-dive or detailed data examination to understand why something
happened. It is characterized by techniques such as drill-down, data discovery, data mining, and
correlations. Multiple data operations and transformations may be performed on a given data set to
discover unique patterns in each of these techniques.

7
What is data science used for?
3. Predictive analysis
Predictive analysis uses historical data to make accurate forecasts about data patterns that may occur
in the future. It is characterized by techniques such as machine learning, forecasting, pattern
matching, and predictive modeling.For example, the flight service team might use data science to
predict flight booking patterns for the coming year at the start of each year. The computer program or
algorithm may look at past data and predict booking spikes for certain destinations in May. Having
anticipated their customer’s future travel requirements, the company could start targeted advertising
for those cities from February.
4. Prescriptive analysis
Prescriptive analytics takes predictive data to the next level. It not only predicts what is likely to
happen but also suggests an optimum response to that outcome. It can analyze the potential
implications of different choices and recommend the best course of action. It uses graph analysis,
simulation, complex event processing, neural networks, and recommendation engines from machine
learning.

8
What is the data science process?
A business problem typically initiates the data science process. A data scientist will work with business
stakeholders to understand what business needs. Once the problem has been defined, the data scientist may solve
it using the OSEMN data science process:
O – Obtain data
Data can be pre-existing, newly acquired, or a data repository downloadable from the internet. Data scientists can
extract data from internal or external databases, company CRM software, web server logs, social media or purchase
it from trusted third-party sources.
S – Scrub data
Data scrubbing, or data cleaning, is the process of standardizing the data according to a predetermined format. It
includes handling missing data, fixing data errors, and removing any data outliers. Some examples of data
scrubbing are:·
Changing all date values to a common standard format.·
Fixing spelling mistakes or additional spaces.·
Fixing mathematical inaccuracies or removing commas from large numbers.

9
What is the data science process?
E – Explore data
Data exploration is preliminary data analysis that is used for planning further data modeling strategies. Data
scientists gain an initial understanding of the data using descriptive statistics and data visualization tools. Then
they explore the data to identify interesting patterns that can be studied or actioned.
M – Model data
Software and machine learning algorithms are used to gain deeper insights, predict outcomes, and prescribe the
best course of action. Machine learning techniques like association, classification, and clustering are applied to the
training data set. The model might be tested against predetermined test data to assess result accuracy. The data
model can be fine-tuned many times to improve result outcomes.
N – Interpret results
Data scientists work together with analysts and businesses to convert data insights into action. They make
diagrams, graphs, and charts to represent trends and predictions. Data summarization helps stakeholders
understand and implement results effectively.

10
Task-1
1. Data science in health care
2. Transforming e-commerce with data science
3. Weather prediction

11
Datafication
• Datafication is the transformation of social action into online quantified data, thus allowing for
real-time tracking and predictive analysis.
• It is about taking previously invisible process/activity and turning it into data, that can be
monitored, tracked, analysed and optimised. Latest technologies enabled lots of new ways of
‘datify’ our daily and basic activities.
• Datafication is a technological trend turning many aspects of our lives into computerized data using
processes to transform organizations into data-driven enterprises by converting this information
into new forms of value.
• Datafication refers to the fact that daily interactions of living things can be rendered into a data
format and put to social use.

12
Datafication: Examples
Let’s say social platforms, Facebook or Instagram, for example, collect and monitor data information
of our friendships to market products and services to us and surveillance services to agencies which in
turn changes our behaviour; promotions that we daily see on the socials are also the result of the
monitored data. In this model, data is used to redefine how content is created by datafication being
used to inform content rather than recommendation systems.
However, there are other industries where datafication process is actively used:
• Insurance: Data used to update risk profile development and business models.
• Banking: Data used to establish trustworthiness and likelihood of a person paying back a loan.
• Human resources: Data used to identify e.g. employees risk-taking profiles.
• Hiring and recruitment: Data used to replace personality tests.
• Social science research: Datafication replaces sampling techniques and restructures the
manner in which social science research is performed.

13
Datafication vs. Digitization
“Datafication is not the same as digitization, which takes analog content—books, films, photographs—
and converts it into digital information, a sequence of ones and zeros that computers can read.
Datafication is a far broader activity: taking all aspects of life and turning them into data format.Once
we datafy things, we can transform their purpose and turn the information into new forms of value.”
Datafication is more about the process of collecting, storing, and managing customer data
from real-world actions, while digitization is the process of converting chosen media into computer-
ready format.

14
Current landscape of perspectives
• We have massive amounts of data about many aspects of our lives, and, simultaneously, an
abundance of inexpensive computing power. Shopping, communicating, reading news, listening to
music, searching for information, expressing our opinions—all this is being tracked online.
• What people might not know is that the “datafication” of our offline behavior has started as well,
mirroring the online data collection revolution. Put the two together, and there’s a lot to learn
about our behavior and, by extension, who we are as a species.
• It’s not just Internet data, though—it’s finance, the medical industry, pharmaceuticals,
bioinformatics, social welfare, government, education, retail, and the list goes on. There is a
growing influence of data in most sectors and most industries.
• In some cases, the amount of data collected might be enough to be considered “big” (more on this
in the next chapter); in other cases, it’s not.

15
• It’s not only the massiveness that makes all this new data interesting but the data itself, often in real
time, becomes the building blocks of data products.
• On the Internet, Amazon recommendation systems, friend recommendations on Facebook, film and
music recommendations, and so on. In finance, credit ratings, trading algorithms, and models. In
education, this is starting to mean dynamic personalized learning and assessments coming out of places
like Coursera and Khan Academy. In government, this means policies based on data.
• We’re witnessing the beginning of a massive, culturally saturated feedback loop where our behavior
changes the product and the product changes our behavior. Technology makes this possible:
infrastructure for large-scale data processing, increased memory, and bandwidth, as well as a cultural
acceptance of technology in the fabric of our lives. This wasn’t true a decade ago.
• Considering the impact of this feedback loop, we should start thinking seriously about how it’s being
conducted, along with the ethical and technical responsibilities for the people responsible for the process.

16
“Data science, as it’s practiced, is a blend of Red-Bull-fueled hacking and espresso-inspired statistics.”
Metamarket CEO Mike Driscoll’s [2010]
Statisticians are the ones who make sense of the data deluge occurring in science, engineering, and
medicine; that statistics provides methods for data analysis in all fields, from art history to zoology; that it is
exciting to be a Statistician in the 21st century because of the many challenges brought about by the data
explosion in all of these fields.
DJ Patil and Jeff Hammerbacher—then at LinkedIn and Facebook, respectively—coined the term “data
scientist” in 2008. So that is when “data scientist” emerged as a job title. (Wikipedia finally gained an entry
on data science in 2012.)

17

18
Task
Skill set needed to become data scientist/analyst/Engineer?

What is Big Data?
• No standard definition! here is from Wikipedia:
• Big data is a term for data sets that are so large or complex that
traditional data processing applications are inadequate.
• Challenges include analysis, capture, data curation, search, sharing,
storage, transfer, visualization, querying, updating and information
privacy.
• Analysis of data sets can find new correlations to "spot business trends,
prevent diseases, combat crime and so on."
19

Who is generating Big Data?
Homeland Security
Real Time Search
Social
eCommerce
User Tracking &
Engagement
Financial Services
20

• The total amount of data created, captured, copied, and consumed globally increases
rapidly, reaching 64.2 zettabytes in 2020.
• It’s not easy to measure the total volume of data stored electronically, but an estimate is
that over the next five years up to 2025, global data creation is projected to grow to more
than 180 zettabytes.
Consider the following:
• The New York Stock Exchange generates about 4-5 terabytes of data per day.
• Facebook hosts more than 240 billion photos, growing at 7 petabytes per month.
• Ancestry.com, the genealogy site, stores around 10 petabytes of data.
• The Internet Archive stores around 18.5 petabytes of data.
21
What is Big Data?

Data Storage and Analysis
• Although the storage capacities of hard drives have increased massively over the years,
access speeds—the rate at which data can be read from drives have not kept up.
• The size, speed, and complexity of big data necessitates the use specialist of software
which in turn relies on significant processing power and storage capabilities. While costly,
embracing big data analytics enables organizations to derive powerful insights and gain a
competitive edge.
• By 2029, the value of the big data analytics market is expected to reach over 655 billion
U.S. dollars, up from around
• 15 billion U.S. dollars in 2019.
• 68 billion U.S. dollars by 2025
• 655 billion U.S. dollars by 2029
22

Big Data Characteristics: 3V
23

Volume (Scale)
• Data Volume
• Growth 40% per year
• From 8 zettabytes (2016) to 44zb (2020)
• Data volume is increasing exponentially
Exponential increase in
collected/generated data
24

How much data?
Hadoop: 10K nodes, 150K
cores, 150 PB (4/2014)
Processes 20 PB a day (2008)
Crawls 20B web pages a day (2012)
Search index is 100+ PB (5/2014)
Bigtable serves 2+ EB, 600M QPS (5/2014)
300 PB data in Hive +
600 TB/day (4/2014)
400B pages,
10+ PB (2/2014)
LHC: ~15 PB a year
LSST: 6-10 PB a year
(~2020)
640K ought to be
enough for
anybody.
150 PB on 50k+ servers
running 15k apps (6/2011)
S3: 2T objects, 1.1M
request/second (4/2013)
SKA: 0.3 – 1.5 EB
per year (~2020)
Hadoop: 365 PB, 330K
nodes (6/2014)
25

Variety (Complexity)
• Different Types:
• Relational Data (Tables/Transaction/Legacy Data)
• Text Data (Web)
• Semi-structured Data (XML)
• Graph Data
• Social Network, Semantic Web (RDF), …
• Streaming Data
• You can only scan the data once
• A single application can be generating/collecting many types of data
• Different Sources：
• Movie reviews from IMDB and Rotten Tomatoes
• Product reviews from different provider websites
To extract knowledge all these types of
data need to linked together
26

A Single View to the Customer
Customer
Social
Media
Gaming
Entertain
Banking
Finance
Our
Known
History
Purchase
27

A Global View of Linked Big Data
patient
doctors
gene
protein
drug
“Ebola”
tissue
Heterogeneous information network
Diversified social network
28

Velocity (Speed)
• Data is begin generated fast and need to be processed fast
• Online Data Analytics
• Late decisions  missing opportunities
• Examples
• E-Promotions: Based on your current location, your purchase history, what you like
 send promotions right now for store next to you
• Healthcare monitoring: sensors monitoring your activities and body  any
abnormal measurements require immediate reaction
• Disaster management and response
29

Real-Time Analytics/Decision Requirement
Customer
Product
Recommendations
that are Relevant
& Compelling
Friend Invitations
to join a
Game or Activity
that expands
business
Preventing Fraud
as it is Occurring
& preventing more
proactively
Learning why Customers
Switch to competitors
and their offers; in
time to Counter
Improving the
Marketing
Effectiveness of a
Promotion while it
is still in Play
30
Influence
Behavior

Extended Big Data Characteristics: 6V
• Volume: In a big data environment, the amounts of data collected and processed are much larger
than those stored in typical relational databases.
• Variety: Big data consists of a rich variety of data types.
• Velocity: Big data arrives to the organization at high speeds and from multiple sources
simultaneously.
• Veracity: Data quality issues are particularly challenging in a big data context.
• Visibility/Visualization: After big data being processed, we need a way of presenting the data in a
manner that’s readable and accessible.
• Value: Ultimately, big data is meaningless if it does not provide value toward some meaningful goal.
31

Veracity (Quality & Trust)
• Data = quantity + quality
• When we talk about big data, we typically mean its quantity:
• What capacity of a system provides to cope with the sheer size of the data?
• Is a query feasible on big data within our available resources?
• How can we make our queries tractable on big data?
• . . .
• Can we trust the answers to our queries?
• Dirty data routinely lead to misleading financial reports, strategic business planning
decision  loss of revenue, credibility and customers, disastrous consequences
• The study of data quality is as important as data quantity.
32

Data in real-life is often dirty
500,000 dead people retain
active Medicare cards
81 million National Insurance
numbers but only 60 million
eligible citizens
98000 deaths each year,
caused by errors in
medical data
33

Visibility/Visualization
• Visible to the process of big data management
• Big Data – visibility = Black Hole?
• Big data visualization tools:
A visualization of Divvy bike rides across Chicago
34

Value
• Big data is meaningless if it does not provide value toward some
meaningful goal
35

Big Data: 6V in Summary
Transforming Energy and Utilities through Big Data & Analytics. By Anders Quitzau@IBM
36

Other V’s
• Variability
Variability refers to data whose meaning is constantly changing. This is particularly
the case when gathering data relies on language processing.
• Viscosity
This term is sometimes used to describe the latency or lag time in the data relative to
the event being described. We found that this is just as easily understood as an element
of Velocity.
• Virality
Defined by some users as the rate at which the data spreads; how often it is picked
up and repeated by other users or events.
• Volatility
Big data volatility refers to how long is data valid and how long should it be stored.
You need to determine at what point is data no longer relevant to the current analysis.
• More V’s in the future …
37

Big Data Overview
Several industries have led the way in developing their ability to gather and exploit data:
• Credit card companies monitor every purchase their customers make and can identify fraudulent
purchases with a high degree of accuracy using rules derived by processing billions of transactions.
• Mobile phone companies analyze subscribers’ calling patterns to determine, for example, whether a
caller’s frequent contacts are on a rival network. If that rival network is offering an attractive
promotion that might cause the subscriber to defect, the mobile phone company can proactively offer
the subscriber an incentive to remain in her contract.
• For companies such as LinkedIn and Facebook, data itself is their primary product. The valuations of
these companies are heavily derived from the data they gather and host, which contains more and
more intrinsic value as the data grows.
38

McKinsey’s definition of Big Data implies that organizations will need new data architectures and
analytic sandboxes, new tools, new analytical methods, and an integration of multiple skills into the
new role of the data scientist
Big Data Overview
39

40
• Social media and genetic sequencing are among the fastest-growing sources of Big Data and
examples of untraditional sources of data being used for analysis.
• For example, in 2012 Facebook users posted 700 status updates per second worldwide, which can
be leveraged to deduce latent interests or political views of users and show relevant ads. For
instance, an update in which a woman changes her relationship status from “single” to “engaged”
would trigger ads on bridal dresses, wedding planning, or name-changing services.
• Facebook can also construct social graphs to analyze which users are connected to each other as an
interconnected network. In March 2013, Facebook released a new feature called “Graph Search,”
enabling users and developers to search social graphs for people with similar interests, hobbies,
and shared locations.
Big Data Overview

41
• Another example comes from genomics. Genetic sequencing and human genome mapping provide a
detailed understanding of genetic makeup and lineage. The health care industry is looking toward
these advances to help predict which illnesses a person is likely to get in his lifetime and take steps to
avoid these maladies or reduce their impact through the use of personalized medicine and treatment.
• Such tests also highlight typical responses to different medications and pharmaceutical drugs,
heightening risk awareness of specific drug treatments.
Big Data Overview

42
Mathematics for Data Science
Mathematics for Machine Learning and Data Science Specialization (Coursera)
https://www.coursera.org/specializations/mathematics-for-machine-learning-and-data-science#courses

43
Statistics
• Statistics is a method of interpreting, analyzing and summarizing the data.
• Statistical analysis is meant to collect and study the information available in large quantities
• For example, the collection and interpretation of data about a nation like its economy and
population, military, literacy, etc.
• Statistics have majorly categorized into two types:
o Descriptive statistics
o Inferential statistics

44
• In descriptive statistics, the data is summarized through the given observations.
• The summarization is done from a sample of population using parameters such as the mean or standard
deviation.
• Descriptive statistics is a way to organize, represent and describe a collection of data using tables, graphs,
and summary measures. For example, the collection of people in a city using the internet or using
Television.
• Descriptive statistics are also categorized into four different categories:
o Measure of frequency - frequency measurement displays the number of times a particular data
occurs
o Measure of dispersion - Range, Variance, Standard Deviation are measures of dispersion. It
identifies the spread of data
o Measure of central tendency - Central tendencies are the mean, median and mode of the data
o Measure of position - the measure of position describes the percentile and quartile ranks.
Descriptive Statistics

46
The marks scored by 4 students in Maths and Physics are given below
Students Maths Physics
A 85 80
B 70 40
C 95 75
D 50 70
Calculate Covariance Matrix from the above data.

47
X (Midterm exam) Y (Final exam)
72 84
50 63
81 77
74 78
94 90
86 75
59 49
83 79
65 77
33 52
88 74
81 90
𝑚 =
𝑁 𝑥𝑦 − 𝑥 𝑦
𝑁 𝑥2 − ( 𝑥)2
𝑏 =
𝑦 − 𝑚 𝑥
𝑁
Predict the final exam grade of a student who received an 86 on the midterm exam.

49
Inferential statistics
• Inferential statistics is a branch of statistics that involves using data from a sample to make
inferences about a larger population. It is concerned with making predictions, generalizations, and
conclusions about a population based on the analysis of a sample of data.
• Inferential statistics help to draw conclusions about the population while descriptive statistics
summarizes the features of the data set.
• Inferential statistics encompasses two primary categories –
o hypothesis testing and
o regression analysis.
• It is crucial for samples used in inferential statistics to be an accurate representation of the entire
population.

50
Statistical methods for evaluation:
• Hypothesis Testing
• Difference of Means
• Wilcoxon Rank-Sum Test
• Type I and Type II
• Errors
• power and sample size
• ANOVA

51
Hypothesis Testing
• Statistical hypothesis is an assumption made about the data of the population collected for any
experiment. Hypothesis testing is also known as “T Testing”.
• It is not mandatory for this assumption to be true every time.
• In order to validate a hypothesis, it will consider the entire population into account. However, this
is not possible practically. Thus, to validate a hypothesis, it will use random samples from a
population.
• On the basis of the result from testing over the sample data, it either selects or rejects the
hypothesis.
• As an example, you may make the assumption that the longer it takes to develop a product, the
more successful it will be, resulting in higher sales than ever before. Before implementing longer
work hours to develop a product, hypothesis testing ensures there’s an actual connection between
the two.

52
Hypothesis Testing
• Statistical Hypothesis Testing can be categorized into two types as below:
o Null Hypothesis – Hypothesis testing is carried out in order to test the validity of a claim or
assumption that is made about the larger population. This claim that involves attributes to the
trial is known as the Null Hypothesis. The null hypothesis testing is denoted by H0.
o Alternative Hypothesis – An alternative hypothesis would be considered valid if the null
hypothesis is fallacious. The evidence that is present in the trial is basically the data and the
statistical computations that accompany it. The alternative hypothesis testing is denoted by
H1or Ha.

53
Hypothesis Testing
• Hypothesis testing is conducted in the following manner:
1. State the Hypotheses – Stating the null and alternative hypotheses.
2. Formulate an Analysis Plan – The formulation of an analysis plan is a crucial step in this stage.
3. Analyze Sample Data – Calculation and interpretation of the test statistic, as described in the
analysis plan.
4. Interpret Results – Application of the decision rule described in the analysis plan.
• Hypothesis testing ultimately uses a p-value to weigh the strength of the evidence or in other words what
the data are about the population. The p-value ranges between 0 and 1. It can be interpreted in the
following way:
o A small p-value (typically ≤ 0.05) indicates strong evidence against the null hypothesis, so you reject
it.
o A large p-value (> 0.05) indicates weak evidence against the null hypothesis, so you fail to reject it.
• A p-value very close to the cutoff (0.05) is considered to be marginal and could go either way.

54
Hypothesis Testing
The two types of error that can occur from the hypothesis testing:
o Type I Error – Type I error occurs we rejects a null hypothesis when it is true. The term
significance level is used to express the probability of Type I error while testing the hypothesis.
The significance level is represented by the symbol α (alpha).
o Type II Error – Accepting a false null hypothesis H0 is referred to as the Type II error. The
term power of the test is used to express the probability of Type II error while testing
hypothesis. The power of the test is represented by the symbol β (beta).

55
“Does drinking a cup of coffee before an exam improve students' test performance?”
Draw null and alternative hypothesis for given question. Also, If the p-value obtained after hypothesis testing
is 0.03 (at Significance level = 0.05) then what will be the conclusion?

56
One Sample T-Testing
• One sample T-Testing approach collects a huge amount of data and tests it on random samples. To
perform T-Test normally distributed data is required.
• This test is used to test the mean of the sample with the population. For example, the height of
persons living in an area is different or identical to other persons living in other areas.
help("t.test")
# Defining sample vector
x <- rnorm(100)
# One Sample T-Test
t.test(x, mu = 5)

57
Two Sample T-Testing
• In two sample T-Testing, the sample vectors are compared
# Defining sample vector
x <- rnorm(100)
y <- rnorm(100)
# Two Sample T-Test
t.test(x, y)

58
Difference of Means
Reference:
https://stats.libretexts.org/Courses/Luther_College/Psyc_350%3ABehavioral_Statistics_(Toussaint)/08%3A_Te
sts_of_Means/8.03%3A_Difference_between_Two_Means

59
Wilcoxon Test
• The Student’s t-test requires that the distributions follow a normal distribution or if the sample size is
large enough (usually n≥30, thanks to the central limit theorem)
• Wilcoxon test compare two groups when the normality assumption is violated
• The Wilcoxon test is a non-parametric test, meaning that it does not rely on data belonging to any
particular parametric family of probability distributions.
• There are actually two versions of the Wilcoxon test:
o Wilcoxon rank sum test (also referred as The Mann-Withney-Wilcoxon test or Mann-Whitney U
test) is performed when the samples are independent (this test is the non-parametric equivalent to
the Student’s t-test for independent samples).
o The Wilcoxon signed-rank test (also sometimes referred as Wilcoxon test for paired samples) is
performed when the samples are paired/dependent (this test is the non-parametric equivalent to the
Student’s t-test for paired samples).

60
Wilcoxon rank sum test
Problem: Apply Wilcoxon rank sum test on the given data of following 24 students (12 boys and 12 girls)
Girls 19 18 9 17 8 7 16 19 20 9 11 18
Boys 16 5 15 2 14 15 4 7 15 6 7 14
The null and alternative hypothesis of the Wilcoxon test are as follows:
o H0 : the 2 groups are equal in terms of the variable of interest
o H1: the 2 groups are different in terms of the variable of interest
Applied to our research question, we have:
o H0 : grades of girls and boys are equal
o H1 : grades of girls and boys are different

61
data <- data.frame(Gender = as.factor(c(rep("Girl", 12), rep("Boy", 12))),
Grade = c(19, 18, 9, 17, 8, 7, 16, 19, 20, 9, 11, 18,16, 5, 15, 2,
14, 15, 4, 7, 15, 6, 7, 14))
library(ggplot2)
ggplot(data) + aes(x = Gender, y = Grade) +
geom_boxplot(fill = "#0c4c8a") + theme_minimal()
hist(subset(data, Gender == "Girl")$Grade,
main = "Grades for girls",
xlab = "Grades" )
hist(subset(data, Gender == "Boy")$Grade,
main = "Grades for boys",
xlab = "Grades" )
test <- wilcox.test(data$Grade ~ data$Gender)
test

62
Wilcoxon rank sum test with continuity correction
data: data$Grade by data$Gender
W = 31.5, p-value = 0.02056
alternative hypothesis: true location shift is not equal to 0
We obtain the following test statistic, the p-value and a reminder of the hypothesis tested.
The p-value is 0.02056. Therefore, at the 5% significance level, we reject the null hypothesis and we
conclude that grades are significantly different between girls and boys.

63
• Relation between variables where changes in some variables may “explain” or possibly
“cause” changes in other variables.
• Explanatory variables are termed the independent variables and the variables to be
explained are termed the dependent variables.
• Regression model estimates the nature of the relationship between the independent and
dependent variables.
– Change in dependent variables that results from changes in independent variables,
ie. size of the relationship.
– Strength of the relationship.
– Statistical significance of the relationship.
Correlation and Regression

64
Examples
• Dependent variable is retail price of gasoline – independent variable is the price of crude oil.
• Dependent variable is employment income – independent variables might be hours of work,
education, occupation, sex, age, region, years of experience, unionization status, etc.
• Price of a product and quantity produced or sold:
– Quantity sold affected by price. Dependent variable is quantity of product sold –
independent variable is price.
– Price affected by quantity offered for sale. Dependent variable is price – independent
variable is quantity sold.

65
0
100
200
300
400
500
600
1981M01
1982M01
1983M01
1984M01
1985M01
1986M01
1987M01
1988M01
1989M01
1990M01
1991M01
1992M01
1993M01
1994M01
1995M01
1996M01
1997M01
1998M01
1999M01
2000M01
2001M01
2002M01
2003M01
2004M01
2005M01
2006M01
2007M01
2008M01
0
20
40
60
80
100
120
140
160
Crude Oil price index, 1997=100, left axis Regular gasoline prices, regina, cents per litre, right axis

66
Bivariate and multivariate models
(Education) x y (Income)
(Education) x1
(Sex) x2
(Experience) x3
(Age) x4
y (Income)
Bivariate or simple regression model
Multivariate or multiple regression model
Price of wheat Quantity of wheat produced
Model with simultaneous relationship
100% Y = 0.2*x1+0.15* x2+0.5*x3+0.15*x4

67
Bivariate or simple linear regression
• x is the independent variable
• y is the dependent variable
• The regression model is
• The model has two variables, the independent or explanatory variable, x, and the dependent
variable y, the variable whose variation is to be explained.
• The relationship between x and y is a linear or straight line relationship.
• Two parameters to estimate – the slope of the line β1 and the y-intercept β0 (where the line crosses
the vertical axis).
• ε is the unexplained, random, or error component. Much more on this later.
𝑦 = 𝛽0 + 𝛽1𝑥 + 𝜀

68
Regression line
• The regression model is
• Data about x and y are obtained from a sample.
• From the sample of values of x and y, estimates b0 of β0 and b1 of β1 are obtained using the least
squares or another method.
• The resulting estimate of the model is
• The symbol is termed “y hat” and refers to the predicted values of the dependent variable y that
are associated with values of x, given the linear model.
𝑦 = 𝛽0 + 𝛽1𝑥 + 𝜀
𝑦 = 𝑏0 + 𝑏1𝑥
𝑦

69
Uses of regression
• Amount of change in a dependent variable that results from changes in the independent
variable(s) – can be used to estimate elasticities, returns on investment in human
capital, etc.
• Attempt to determine causes of phenomena.
• Prediction and forecasting of sales, economic growth, etc.
• Support or negate theoretical model.
• Modify and improve theoretical models and explanations of phenomena.

70
Summer Income as a Function of Hours Worked
0
5000
10000
15000
20000
25000
30000
0 10 20 30 40 50 60
Hours per Week
Income

71
R2 = 0.311
Significance = 0.0031

74
Outliers
• Rare, extreme values may distort the outcome.
• Could be an error.
• Could be a very important observation.
• Outlier: more than 3 standard deviations from the mean.

75
GPA vs. Time Online
0
2
4
6
8
10
12
50 55 60 65 70 75 80 85 90 95 100
GPA
Time
Online

76
Classification
• Labelled Data
• Logistic Regression
• ANN
• SVM etc

77
Clustering
• Cluster: A collection of data objects
o similar (or related) to one another within the same group
o dissimilar (or unrelated) to the objects in other groups
• Cluster analysis (or clustering, data segmentation, …)
Finding similarities between data according to the characteristics found in the data and
grouping similar data objects into clusters
• Unsupervised learning: no predefined classes (i.e., learning by observations vs. learning by
examples: supervised)
• Typical applications
o As a stand-alone tool to get insight into data distribution
o As a preprocessing step for other algorithms

78
Clustering: Applications
• Biology: taxonomy of living things: kingdom, phylum, class, order, family, genus and species
• Information retrieval: document clustering
• Land use: Identification of areas of similar land use in an earth observation database
• Marketing: Help marketers discover distinct groups in their customer bases, and then use this
knowledge to develop targeted marketing programs
• City-planning: Identifying groups of houses according to their house type, value, and
geographical location
• Earth-quake studies: Observed earth quake epicenters should be clustered along continent
faults
• Climate: understanding earth climate, find patterns of atmospheric and ocean
• Economic Science: market research

79
Clustering as a Preprocessing Tool (Utility)
• Summarization:
o Preprocessing for regression, PCA, classification, and association analysis
• Compression:
o Image processing: vector quantization
• Finding K-nearest Neighbors
o Localizing search to one or a small number of clusters
• Outlier detection
o Outliers are often viewed as those “far away” from any cluster

80
Quality: What Is Good Clustering?
• A good clustering method will produce high quality clusters
o high intra-class similarity: cohesive within clusters
o low inter-class similarity: distinctive between clusters
• The quality of a clustering method depends on
o the similarity measure used by the method
o its implementation, and
o Its ability to discover some or all of the hidden patterns

81
Measure the Quality of Clustering
• Dissimilarity/Similarity metric
o Similarity is expressed in terms of a distance function, typically metric: d(i, j)
o The definitions of distance functions are usually rather different for interval-scaled, boolean,
categorical, ordinal ratio, and vector variables
o Weights should be associated with different variables based on applications and data
semantics
• Quality of clustering:
o There is usually a separate “quality” function that measures the “goodness” of a cluster.
o It is hard to define “similar enough” or “good enough”
o The answer is typically highly subjective

82
Considerations for Cluster Analysis
• Partitioning criteria
o Single level vs. hierarchical partitioning (often, multi-level hierarchical partitioning is
desirable)
• Separation of clusters
o Exclusive (e.g., one customer belongs to only one region) vs. non-exclusive (e.g., one
document may belong to more than one class)
• Similarity measure
o Distance-based (e.g., Euclidian, road network, vector) vs. connectivity-based (e.g., density or
contiguity)
• Clustering space
o Full space (often when low dimensional) vs. subspaces (often in high-dimensional clustering)

83
Requirements and Challenges
• Scalability
o Clustering all the data instead of only on samples
• Ability to deal with different types of attributes
o Numerical, binary, categorical, ordinal, linked, and mixture of these
• Constraint-based clustering
o User may give inputs on constraints
o Use domain knowledge to determine input parameters
• Interpretability and usability
• Others
o Discovery of clusters with arbitrary shape
o Ability to deal with noisy data
o Incremental clustering and insensitivity to input order
o High dimensionality

84
Major Clustering Approaches
• Partitioning approach:
o Construct various partitions and then evaluate them by some criterion, e.g., minimizing the sum of square
errors
o Typical methods: k-means, k-medoids, CLARANS
• Hierarchical approach:
o Create a hierarchical decomposition of the set of data (or objects) using some criterion
o Typical methods: Diana, Agnes, BIRCH, CAMELEON
• Density-based approach:
o Based on connectivity and density functions
o Typical methods: DBSACN, OPTICS, DenClue
• Grid-based approach:
o based on a multiple-level granularity structure
o Typical methods: STING, WaveCluster, CLIQUE

Data analytics using Scalable Programming

More Related Content

Similar to Data analytics using Scalable Programming (20)

Recently uploaded (20)

Data analytics using Scalable Programming

Editor's Notes