Cse443 Project Report - LPU (Modern Big Data Analysis with SQL Specialization)

1
CSE443 Seminar on Summer Training
Modern Big Data Analysis with SQL Specialization
A Training Report
Submitted in partial fulfillment of the requirements for the award of
degree of
Bachelor of Technology (Computer Science and Engineering)
Submitted to
LOVELY PROFESSIONAL UNIVERSITY
PHAGWARA, PUNJAB
From 05/13/2021 to 07/12/2021
Submitted By
Qazi Maaz Arshad (11906424)
B. Tech
Computer Science and Engineering
Lovely Professional University, Phagwara
Maaz

2
Student Declaration
To whom so ever it may concern
I, Qazi Maaz Arshad, 11906424, hereby declare that the work done
by me on training “Modern Big Data Analysis with SQL
Specialization” from 13 May 2021 to 12 July 2021, is a record of
original work for the partial fulfillment of the requirements for the
award of the degree, Bachelor of Technology (Computer Science
and Engineering).
B. Tech
Maaz
09/02/2021

3
MOOC Certificate
Authenticate Certificate: Link
Verify Verify
Verify

4
Acknowledgement
The opportunity of attaining a course based on Modern Big Data
Analysis with SQL Specialization through Coursera, provided by
CLOUDERA was worth learning. It was a prestige for me to be part
of it. During the period of my course, I received tremendous
knowledge related to SQL and Database Management.
Pre-eminently, I would like to express my deep gratitude and special
thanks to my course instructors Glynn Durham & Ian Cook of
CLOUDERA for their theoretical knowledge and encouragement on
this project and for their valuable guidance and affection for the
successful completion of this work.
Secondly, I would like to thank Lovely Professional University for
giving me an opportunity to learn this course by sponsoring my
Coursera subscription.
Lastly, I would like to thank the almighty and my parents for their
constant encouragement, moral support, personal attention, and care.
B. Tech
Maaz

5
Contents
Sr. No. Topic Page No.
1 Cover Page 1
2 Student Declaration 2
3 MOOC Certificate 3
4 Acknowledgement 4
5 Contents 5
6 List of Tables/Figures 6
7 List of Abbreviations 6
8 Abstract 7
9 Chapter 1 – Introduction 8 - 10
10  Overview of Training 8 - 9
11  Overview of Project 9 - 10
12 Chapter 2 – Reason for Choosing this
Technology
10 - 11
13 Chapter 3 – Project: Movies Data Analysis 11 - 43
14  Profile of the Project 11
15  System Requirements and Specifications 12
16  Technologies and Tools Used 13
17  Design and Charts 14 - 15
18  Project Source Code 16 - 43
19 Chapter 4 – Timeline of Training and Project
Work
44
20 Chapter 5 – Skills Gained from Training and
Project
45
20 Final Chapter – Conclusion and Future
Outlook of Big Data, SQL Databases, and
Data Analysis
46 - 47
21 Summary of Report 48
22 References 49

6
List of Tables and Figures
Table No. Table Name Page No.
3.1 ER Diagram – Created using Creately 14
3.2 ER Diagram – Created using MySQL
Workbench
15
List of Abbreviations
 SQL – Structured Query Language
 DBMS – Database Management System
 RDBMS – Relational Database Management System
 CPU – central processing unit
 GB – Gigabyte
 RAM – Random-access memory

7
Abstract
The Structured Query Language (SQL) is the main programming
language designed to manage data stored in database systems. While
SQL was initially used only with relational database management
systems (RDBMS), its use has been significantly extended with the
advent of new types of database systems. Specifically, SQL has been
found to be a powerful query language in highly distributed and
scalable systems that process Big Data, i.e., datasets with high
volume, velocity, and variety. While traditional relational databases
represent now only a small fraction of the database systems
landscape, most database courses that cover SQL consider only the
use of SQL in the context of traditional relational systems. In this
report, I have shared a project where I have done data analysis of a
movie's data set. This report also presents my learning and
contributions during my summer training.

8
Chapter 1 – Introduction
“Modern Big Data Analysis with SQL Specialization” is an online
specialization course offered by CLOUDERA consisting of 3 courses
“Foundations for Big Data Analysis with SQL”, “Analyzing Big
Data with SQL”, and “Managing Big Data in Clusters and Cloud
Storage”. This specialization teaches the essential skills for working
with large-scale data using SQL.
Overview of Training Course
I completed this MOOC through Coursera, it was a specialization
course comprised of 3 other courses, the total duration of this
specialization is approximately 4 months (if invested 3 hours per
week), I completed this specialization within two months. This
specialization course was provided by CLOUDERA. All the courses
comprised multiple video lectures, reading materials, quizzes, and
multiple assignments. Once the course has started all the tasks like
completing video lectures, reading materials, including quizzes and
assignments were to be completed before the deadline. Also, to pass
the course one must have to score a minimum of 75% in all the
quizzes. Also, the submitted assignments must have to be checked
and awarded a minimum of 75% marks by fellow peers. Also, I must
evaluate at least 3 assignments of my peers to pass the courses.
Although I had done few assignments in the specialization, I decided
to work on some other projects to evaluate my skills.
Course 1: Foundations for Big Data Analysis with SQL –
In this course, I learned about database systems and the distinction
between operational and analytic databases. Learned the benefits of
organizing data. In short, it was an overview of data, database
systems, and the common querying language (SQL). This course gave

9
insights into the characteristics of Big Data and SQL tools for
working on big data platforms. This course helped me to understand
how database and table designs provide structures for working with
data.
Course 2: Analyzing Big Data with SQL – This course
provided in-depth knowledge of the SQL SELECT statement and its
main clauses. I understood how and why to filter results, explored
grouping and aggregation to answer analytic questions, working with
sorting and limiting results, and combining tables in different ways.
Course 3: Managing Big Data in Clusters and Cloud
Storage – This course's main focus was to manage big datasets, how
to apply structures to the data in order to run queries.
Overview of the Project
Although I had done a few assignments in this specialization, I
decided to work on some other projects to evaluate my skills. I have
performed data analysis on a movie's data set, using SQL. I have
created the database and performed all the operations and ran all
queries on MYSQL Workbench.
The movies database consists of 4 tables, films, people, reviews, and
roles. All the data sets combined consist of more than 20000 entries.
The movie's database contains information regarding the name of the
movies, the year when they were released, country of origin, duration
of films, the language of films, the budget of the movie, and the profit
earned. It also contains the name of people who were involved in the
films, what was their role like actors or directors, when they were
born, and when they died. It also gives details about the number of
reviews a movie received, what was the response it received from the
audience, how was the response of the movie over social media, etc.
In this project, I have calculated various results from this large

10
database, like how many French movies are there, which movie
earned the highest profit in the ’90s, which actor acted in most of the
movies, what was the average duration of movies, which was the
longest English movie, which Indian movie received highest up-votes
on social media, etc.
In this project, I have implemented all the knowledge I gained in the
course, like how to use Structured Query Language (SQL) to extract
and analyze data stored in databases. I have extracted data, joined
tables together, and performed aggregations. I have also done more
complex analysis and manipulations using sub-queries.
Chapter 2 – Reason for Choosing this
Technology
My main reason behind choosing this technology was the exponential
increase in the use of databases nowadays. In today’s world, it won’t
be wrong to say that we are surrounded by data, data is everywhere
from the marks in our report card, our bank account statements, what
movies we have downloaded to binge-watch this weekend, etc.
But what actually is data? Data is unstructured facts and figures. Facts
and figures relay something specific, but which are not organized in
any way and which provide no further information regarding patterns,
context, and other details.
But what is the use of unstructured data, we can get information from
the data only if our data is contextualized, categorized, calculated, and
condensed. This is why it is so important to arrange, calculate,
condense, filter, and optimize the data. And this is where databases
come into the picture.
To say that the databases are everywhere would be an understatement.
They virtually permeate our lives: Online stores, health care
providers, clubs, libraries, video stores, beauty salons, travel agencies,
phone companies, government agencies like FBI, INS, IRS, and

11
NASA, giant businesses like Netflix, Microsoft, Google they all use
database.
So, I thought its the best time to get hands-on experience on the most
demanded skill i.e. data analysis. Data analysis is the future, and the
future will demand skills for jobs as functional analysts, data
engineers, data scientists, and advanced analysts. I believed that after
this training I will have the ability to analyze data and make informed
recommendations to drive effective decision-making and this will turn
me into an indispensable member of any team.
Chapter 3 – Project: Movies Data Analysis
Profile of the Project
A movie database finds its real application in online movie streaming
sites, but the movies data set involved in this picture consists of
records of more than 20000 movies of different languages, made in
different countries, directed by different directors over a range of 100
years. This data set is very useful in calculating results from past
movies to analyze trends in the movie industries across the globe and
use the past experiences and trends as samples for working on newer
projects.
The movies database consists of 4 tables, films, people, reviews, and
roles. All the data sets combined consist of more than 20000 entries.
The movie's database contains information regarding the name of the
movies, the year when they were released, country of origin, duration
of films, the language of films, the budget of the movie, and the profit
earned. It also contains the name of people who were involved in the
films, what was their roles like actors or directors, when they were
born, and when they died. It also gives details about the number of
reviews a movie received, what was the response it received from the
audience, how was the response of the movie over social media, etc.

12
System Requirements Specification
The environmental specification specifies the hardware and software
requirements for carrying out this project. The following are the
hardware and software requirements.
Hardware –
Minimum Recommended
CPU 64bit x86 CPU Multi Core 64bit x86 CPU, 8 GB RAM
RAM 4 GB 8 GB or higher
Display 1024×768 1920×1200 or higher
Software -
Operating System Architecture
Oracle Linux 8 / Red Hat Enterprise Linux 8 x86_64
Ubuntu 21.04 x86_64
Ubuntu 20.04 LTS x86_64
Windows Server 2019 x86_64
Windows 10 x86_64
macOS 11 x86_64
• Microsoft .NET Framework 4.5.2
• Microsoft Visual C++ Redistributable for Visual Studio 2019
• MySQL server
• MySQL Workbench

13
Technologies and Tools Used
MySQL Workbench - MySQL Workbench is a visual database
design tool that integrates SQL development, administration, database
design, creation and maintenance into a single integrated development
environment for the MySQL database system.
SQL - SQL is a domain-specific language used in programming and
designed for managing data held in a relational database management
system, or for stream processing in a relational data stream
management system.
CSV File - A comma-separated values file is a delimited text file
that uses a comma to separate values. Each line of the file is a data
record. Each record consists of one or more fields, separated by
commas. The use of the comma as a field separator is the source of
the name for this file format.
Creately - Creately is a SaaS visual collaboration tool with
diagramming and design capabilities designed by Cinergix. Creately
has two versions: an online cloud edition and a downloadable offline
edition for desktop which is compatible with Windows, Mac and
Linux.

14
Design and Charts
Figure 3.1 - ER Diagram – Created using Creately

15
Figure 3.2 - ER Diagram – Created using MySQL Workbench

16
Project Source Code
CREATE DATABASE movies;
USE movies;
CREATE TABLE films (
id INTEGER,
title VARCHAR(50),
release_year INTEGER,
country VARCHAR(30),
duration INTEGER,
language VARCHAR(30),
certification VARCHAR(25),
gross BIGINT,
budget BIGINT,
CONSTRAINT films_pk PRIMARY KEY(id)
);
CREATE TABLE people (
id INTEGER PRIMARY KEY,
name VARCHAR(30),
birthdate DATE,
deathdate DATE
);
CREATE TABLE reviews (
film_id INTEGER,
num_user INTEGER,
num_critic INTEGER,
imdb_score REAL,
num_votes INTEGER,
facebook_likes INTEGER
);
CREATE TABLE roles (
film_id INTEGER,
person_id INTEGER,
role VARCHAR(30)
);

17
CREATE TABLE roles (
film_id INTEGER,
person_id INTEGER,
role VARCHAR(30)
);
-- Show All tables
SHOW TABLES;
-- Show content of films table
SELECT * FROM films;
-- Show content of people table
SELECT * FROM people;
-- Show content of reviews table
SELECT * FROM reviews;

18
-- Show content of roles table
SELECT * FROM roles;
-- Selecting Columns: SELECT, SELECT DISTINCT
-- Get the title of every film
SELECT title
FROM films;
-- Get all details for every film
SELECT *
FROM films;
-- Get the names of everyone involved in working
on the films
SELECT name
FROM people;

19
-- Get the title and release year of every film
SELECT title, release_year
FROM films;
-- Get the title, release year and country for
every film
SELECT title, release_year, country
FROM films;
-- Get every person's name and their date of birth
where possible
SELECT name, birthdate
FROM people;
-- Get every person name and their date of death
where possible
SELECT name, deathdate
FROM people;

20
-- Get everyone's name, date of birth, and date of
death (where possible)
SELECT name, birthdate, deathdate
FROM people;
- Get all the different countries
SELECT DISTINCT country
FROM films;
-- Get all the different film languages
SELECT DISTINCT language
FROM films;
-- Get the different types of film roles
SELECT DISTINCT role
FROM roles;

21
-- Get all the different certification categories
SELECT DISTINCT certification
FROM films;
-- Get all the different IMDB scores
SELECT DISTINCT imdb_score
FROM reviews;
-- Aggregate Functions: COUNT, SUM, AVG, MIN, MAX
-- Count the number of rows in the people table
SELECT COUNT(*)
FROM people;
-- Result 781
-- Count the number of birthdate entries in the
people table
SELECT COUNT(birthdate)
FROM people;
-- Result 781
-- Count the number of unique birthdate entries in
the people table
SELECT COUNT(DISTINCT birthdate)
FROM people;
-- Result 757

22
-- Count the number of unique languages
SELECT COUNT(DISTINCT language)
FROM films;
-- Result 39
-- Count the number of unique countries
SELECT COUNT(DISTINCT country)
FROM films;
-- Result 46
-- Count the number of people who have died
SELECT COUNT(deathdate)
FROM people;
-- Result 781
-- Count the number of years the dataset covers
SELECT COUNT(DISTINCT release_year)
FROM films;
-- Result 75
-- Get the total duration of all films
SELECT SUM(duration)
FROM films;
-- Result 426426
-- Get the average duration of all films
SELECT AVG(duration)
FROM films;
-- Result 109.9319
-- Get the duration of the shortest film
SELECT MIN(duration)
FROM films;
-- Result 37
-- Get the amount made by the highest grossing film
SELECT MAX(gross)
FROM films;
-- Result 936627416
-- Get the amount made by the lowest grossing film
SELECT MIN(gross)
From films;
-- Result 162

23
-- Get both the lowest and highest grossing films,
for comparision
SELECT MIN(gross), MAX(gross)
FROM films;
-- Get the highest number of Facebook likes for any
film
SELECT MAX(facebook_likes)
FROM reviews;
-- Result 150000
-- Aliasing and Basic Arithmetic
-- Get the profit (or loss) for each movie, where
possible
SELECT title, gross - budget
AS profit_or_loss
FROM films;
-- Get the duration in hours for each film
SELECT title, duration / 60.0 AS duration_hours
FROM films;

24
-- Get the average film duration in hours
SELECT AVG(duration) / 60.0
AS duration_hours
FROM films;
-- Result 1.83219902
-- Get the percentage of people who have died
SELECT COUNT(deathdate) * 100 / COUNT(*)
AS percentage_dead
FROM people;
-- Result 100.0000
-- Check if there's an even number of unique
languages
SELECT COUNT(DISTINCT language) % 2
AS result
FROM films;
-- Result 1 (0 = yes, 1 = no)
-- Get the of years between the oldest film and
newest film
SELECT MAX(release_year) - MIN(release_year)
AS difference
FROM films;
-- Result 96
-- Get the number of decades this dataset covers
SELECT (MAX(release_year) - MIN(release_year)) / 10
AS number_of_decades
FROM films;
-- Result 9.6000
-- Rounding Functions: ROUND, FLOOR, CEILING
-- Get the average duration of all films, rounded to
the nearest minute
SELECT ROUND(AVG(duration))
AS rounded_avg_run_time
FROM films;
-- Result 110

25
-- Get the average duration of all films, rounded
down to nearest minute
SELECT FLOOR(AVG(duration))
AS floored_avg_run_time
FROM films;
-- Result 109
-- Get the average duration of all films, rounded up
to the nearest minute
SELECT CEILING(AVG(duration))
FROM films;
-- Result 110
-- Filtering: WHERE, =, <>, <, <=, >, >=, AND, OR
-- Get all French language films
SELECT *
FROM films
WHERE language = 'French';
SELECT COUNT(*)
FROM films
WHERE language = 'Hindi';
-- Result 10
-- Get all movies with an R certification
SELECT *
FROM films
WHERE certification = 'R';

26
-- Get all films released in 2016
SELECT *
FROM films
WHERE release_year = 2016;
-- Count of actors
SELECT COUNT(*)
FROM roles
WHERE role = 'actor';
-- Result 14862
-- Count of directors
SELECT COUNT(*)
FROM roles
WHERE role = 'director';
-- Result 4929
-- Count of movies not rated
SELECT COUNT(*)
FROM films
WHERE certification = 'Not Rated' OR certification
IS NULL;
-- Result 42
-- Count of movies not in English
SELECT COUNT(*)
FROM films
WHERE language <> 'English';
-- Result 184
-- Get the number of films released before 2000
SELECT COUNT(*)
FROM films
WHERE release_year < 2000;
-- Result 1050

27
-- Get the title and release year of films released
since 2000
FROM films
WHERE release_year > 2000;
-- Get all Spanish films released before 2000
FROM films
WHERE release_year < 2000
AND language = 'Spanish';
-- Get the all Spanish films released since 2000
SELECT *
FROM films
WHERE release_year > 2000
AND language = 'Spanish';
-- Get average duration for films released in France
in 1993
FROM films
WHERE release_year = 1993
AND country = 'France';
-- Result 103.0000

28
-- Get films released in 1990 or released in 2000 in
French or Spanish
FROM films
WHERE release_year = 1990 OR release_year = 2000
AND language = 'French' OR language = 'Spanish';
-- Get films released since 2000 that are in French
or Spanish, and made more than $20m
SELECT *
FROM films
AND language = 'French' OR language = 'Spanish'
AND gross > 20000000;
-- Get films released in the 90s
FROM films
WHERE release_year >= 1990 AND release_year <= 2000;

29
-- Get average duration for films released in the UK
or which were released in 2012
FROM films
AS average_duration
WHERE release_year = 2012
OR COUNTRY = 'UK';
-- Result 111.3194
-- Advanced Filtering: BETWEEN, IN, IS NULL, IS NOT
NULL, LIKE, NOT LIKE
-- Get films released in the 90s
FROM films
WHERE release_year BETWEEN 1990 AND 2000;
-- Get the number of films released in the 90s
SELECT COUNT(*)
FROM films
WHERE release_year BETWEEN 1990 AND 2000;
-- Result 111.3194
-- Get the number of films released between 2000 and
2015 that were longer than two hours
SELECT COUNT(*)
FROM films
WHERE release_year BETWEEN 2000 AND 2015
AND duration > 120;
-- Result 587

30
-- Get the number of films made between 2000 and 2015
with budgets over $100 million
SELECT title, budget
FROM films
WHERE release_year
BETWEEN 2000 AND 2015
AND budget > 100000000;
-- Get films released in in 1990 or released in 2000
that were longer than two hours
FROM films
WHERE release_year IN (1990, 2000)
AND duration > 120;
-- Get the names of people who are still alive
SELECT name
FROM people
WHERE deathdate IS NULL OR deathdate = '';

31
-- Get people whose names begin with 'B'
SELECT name
FROM people
WHERE name LIKE 'B%';
-- Get people whose names begin with 'Br'
SELECT name
FROM people
WHERE name LIKE 'Br%';
-- Get people whose names have 'r' as the second
letter
SELECT name
FROM people
WHERE name LIKE '_r%';

32
-- Get people whose names don't start with A
SELECT name
FROM people
WHERE name NOT LIKE 'A%';
-- Sorting and Grouping
-- Get people, sort by name
SELECT name
FROM people ORDER BY name;
-- Get people, in order of when they were born
SELECT birthdate, name
FROM people
ORDER BY birthdate;

33
-- Get films released in 2000 or 2015, in the order
they were released
FROM films
WHERE release_year in (2000, 2015)
ORDER BY release_year;
-- Get all films except those released in 2015, and
order them
SELECT *
FROM films
WHERE release_year <> 2015
-- Get the score and film id for every film, from
highest to lowest
SELECT imdb_score, film_id
FROM reviews
ORDER BY imdb_score DESC;

34
-- Get the titles of films in reverse order
SELECT *
FROM films
ORDER BY title DESC;
-- Get people, in order of when they were born, and
alphabetical order
SELECT birthdate, name
FROM people
ORDER BY birthdate, name;
-- Get films from in 2000 or 2015, sorted in the
order they were released, and how long they were
SELECT release_year, duration, title
FROM films
ORDER BY release_year, duration;

35
-- Get films between 2000 and 2015, sorted by
certification and the year they were released
SELECT certification, release_year, title
FROM films
ORDER BY certification, release_year;
-- Get people whose names start with A, B or C,
(redundantly) ordered
SELECT name, birthdate
FROM people
WHERE name LIKE 'A%' OR name LIKE 'B%' OR name LIKE
'C%'
ORDER BY birthdate;
-- Get count of films made in each year
SELECT release_year, COUNT(release_year)
FROM films
GROUP BY release_year;

36
-- Get count of films, group by release year then
order by release year
SELECT release_year, COUNT(title) as films_released
FROM films
GROUP BY release_year
-- Get count of films released in each year, ordered
by count, lowest to highest
SELECT release_year, COUNT(title) AS films_released
FROM films
ORDER BY films_released;
-- Get count of films released in each year, ordered
by count highest to lowest
SELECT release_year, COUNT(title) AS films_released
FROM films
ORDER BY films_released DESC;

37
-- Get lowest box office earnings per year
SELECT release_year, MIN(gross)
FROM films
-- Get the total amount made in each language
SELECT language, SUM(gross)
FROM films
GROUP BY language;
-- Get the total amount spent by each country
SELECT country, SUM(gross)
FROM films
GROUP BY country;

38
-- Get the max budget spent in making film for each
year, for each country
SELECT release_year, country, MAX(budget)
FROM films
GROUP BY release_year, country
ORDER BY release_year, country;
-- Get the lowest box office made by each country in
each year
SELECT release_year, country, MIN(gross)
FROM films
GROUP BY release_year, country
ORDER BY release_year, country;
-- Get the rounded average budget and average box
office earnings for movies since 1990, but only if the
average budget was greater than $60m in that year
SELECT release_year, ROUND(AVG(budget)) AS avg_budget,
ROUND(AVG(gross)) AS avg_box_office
FROM films
HAVING AVG(budget) > 20000000
ORDER BY release_year DESC;

39
-- Get the name, average budget, average box office
take of countries who have made more than 10 films.
Order by name, and get the top five
SELECT country, ROUND(AVG(budget)) AS avg_budget,
ROUND(AVG(gross)) AS avg_box_office
FROM films
GROUP BY country
HAVING COUNT(title) > 10
ORDER BY country
LIMIT 5;
-- Get the average amount made by each country
SELECT country, AVG(gross)
FROM films
GROUP BY country;
-- Get rounded average box office earnings per year
SELECT release_year, ROUND(AVG(gross))
FROM films

40
-- Get lowest and highest box office earnings per year
SELECT release_year, MIN(gross), MAX(gross)
FROM films
ORDER BY release_year DESC;
-- Get the highest box office take per country
SELECT country, MAX(gross)
FROM films
GROUP BY country;
-- Longest duration per year
SELECT release_year, MAX(duration) AS max_duration
FROM films
ORDER BY max_duration DESC;

41
-- Subqueries
-- Get the title, duration and release year of the
shortest film(s)
SELECT title, duration, release_year
FROM films
WHERE duration = (
SELECT MIN(duration) FROM films
);
-- Get the title, duration and release year of the
longest film(s)
SELECT title, duration, release_year
FROM films
WHERE duration = (
SELECT MAX(duration)
FROM films
);
-- Get the title, release_year and box office take for
the highest grossing film
SELECT title, release_year, gross
FROM films
WHERE gross = (
SELECT MAX(gross)
FROM films
);

42
-- Get the title, release_year and box office take for
the lowest grossing film
SELECT title, release_year, gross
FROM films
WHERE gross = (
SELECT MIN(gross)
FROM films
);
-- Get the duration of the longest movie made in the USA
SELECT title, duration
FROM films
WHERE duration = (
SELECT MAX(duration)
FROM films
WHERE country = 'USA'
);
-- Get details for the film with the lowest box office
earnings per year
SELECT release_year, title, gross
FROM films
WHERE release_year IN (
SELECT release_year
FROM films
WHERE gross IN (
SELECT MIN(gross)
FROM films
)
);

43
-- Get details for the film with the highest box
office earnings per year
SELECT release_year, title, gross
FROM films
WHERE release_year IN (
SELECT release_year
FROM films
WHERE gross IN (
SELECT MAX(gross)
FROM films
)
);
-- Joins
-- Get film id of films whose review exist
SELECT films.id
FROM films
INNER JOIN reviews
ON films.id = reviews.film_id;

44
Chapter 4 – Timeline of Training and Project

45
Chapter 5 – Skills Gained from Training and
Project

46
Final Chapter – Conclusion and Future
Outlook of Big Data, SQL Databases, and Data
Analysis
The future of SQL Server will depend on the future of the use of SQL
as a query language. Relational Database Management Systems as we
know them have not really changed much over the last two decades
while just about every other subject relating to computing has. The
success of SQL is in its simplicity and at lower levels of abstraction,
we will always need a technology like this.
However, there are a handful of needs that SQL and RDBMS' simply
don't provide in their current form.
Complex Interface – SQL has a difficult interface that makes few
users uncomfortable while dealing with the database.
Cost – Some versions are costly and hence, programmers cannot
access them.
Partial Control – Due to hidden business rules, complete control is not
given to the database.
Almost every business problem or need involves the use and
maintenance of data. Data is virtually the lifeblood of a business so it
will always be important. After all, data – and big data – are just
point-in-time recordings of business or operational events (something
a person, machine, or business did.
Up until perhaps five years ago, most business data was still at a very
coarse level - representing discrete transactions (e.g. purchases,
trades, orders, line items, travel segments, etc.). With big data, we
now have the ability to capture and analyze transactions that are
happening at a finer, more granular level, so we're moving from
transactional to behavioral understanding. A good example of this
centers around e-commerce, and the contrast between tracking and
analyzing purchases on the one hand, and the measurement and
analysis of clickstream data, to understand customer behavior, on the

47
other. Big data analytics have gained popularity over the past decade,
and many experts see the same to continue for the next decade.
Big data analytics is going to be mainstream with increased adoption
among every industry and forms a virtuous cycle with more people
wanting access to even bigger data.
However, often the requirements for big data analysis are really not
well understood by the developers and business owners, thus creating
an undesirable product.
For organizations to not waste precious time and money and
manpower over these issues, there is a need to develop expertise and
process of creating small-scale prototypes quickly and test them to
demonstrate their correctness, matching with business goals.
A survey by Gartner found that 48% of the companies invested in big
data in 2016, and nearly three-quarters of those surveyed had already
invested, or were planning to invest in data analytics. Big data is
helping companies in different sectors, from marketing to
pharmaceutical companies to third sector organizations. By 2023, it is
predicted that the amount of data that is worthy of being analyzed,
will surprisingly triple.
Seeing and analyzing the applications of big data analytics, and the
huge support that it provides to companies, it is clear that it is here to
stay. It is efficient and predicts most of the data right and saves time
and cost. Therefore, for every area touched by big data analytics the
word "better" can be added in front of it, that is, better security, better
training, better education, better business, etc. That is the potential of
this technology.

48
Summary of Report
In this report, I have shared my journey and learning throughout my
summer training. I have discussed what I learnt from all the three
courses of the specialization, what skills I have gained, and timeline
of the summer training.
I have discussed in detail my entire data analysis using SQL project
where I have calculated results using different SQL functions. In this
project, I have calculated various results from this large database, like
how many French movies are there, which movie earned the highest
profit in the ’90s, which actor acted in most of the movies, what was
the average duration of movies, which was the longest English movie,
etc.
I have also discussed in brief the future of relational databases and
Big Data Analytics.

49
References
 https://www.wikipedia.org/ . (Accessed on 6th
Sept 2021).
 https://www.quora.com/ . (Accessed on 10th
Sept 2021).
 https://www.datacamp.com/courses/introduction-to-sql .
Downloaded data sets used in this project. (Accessed on 11th
Sept 2021).
 Kerry Koitzsch, Future of Big Data Analysis, published in book
Pro Hadoop Data Analytics, December 2017, pp. 263-273.
 Ali Salehnia – South Dakota State University, Comparisons of
Relational Databases with Big Data, 2015, pp. 01-17.
 Yasin N. Silva, Isadora Almeida, Michelle Queiroz, SQL from
Traditional Databases to Big Data, published in the 47th
ACM
Technical Symposium, February 2016, pp. 01-03.
 Modern Big Data Analysis with SQL Specialization offered by
CLOUDERA through Coursera.

Cse443 Project Report - LPU (Modern Big Data Analysis with SQL Specialization)

More Related Content

What's hot (20)

Similar to Cse443 Project Report - LPU (Modern Big Data Analysis with SQL Specialization) (20)

More from Qazi Maaz Arshad (12)

Recently uploaded (20)

Cse443 Project Report - LPU (Modern Big Data Analysis with SQL Specialization)