Introduction to Data Engineering
7 likes7,382 views
The document is an introduction to data engineering, discussing its definition, key skills, and differences between data engineers, data scientists, and data analysts. It covers various aspects such as data management, distributed computing, designing data pipelines, and useful resources for aspiring data engineers. Additionally, it highlights online courses and platforms for further learning in the field.
Introduction to Data Engineering
- 1. Introduction to Data Engineering Hadi Fadlallah
- 2. About Me … … … … Just a Data Engineer! 4/6/2022 2
- 3. Plan • What is Data Engineering? • Data Engineer vs. Data Scientist vs. Data Analyst • Understanding Data Management (Data Layers, DQS, MDS, Provenance) • Distributed Computing • Designing Data Pipelines (Choosing Paradigm / Technologies) • Data Engineer Jobs / Required Skills • Helpful Tips • Online Courses 4/6/2022 3
- 5. What is Data Engineering? 4/6/2022 5 • Data Engineering is the act of: • Collecting data • Transforming (…) data • Validating data “Making data consumable”
- 6. What is a Data Engineer? AI/ML Engineer BI Developer Data Analyst Database Administrator Report Developer Data Developer Data Architect Data Integration Specialist ETL Developer Data Scientist 4/6/2022 6
- 7. Data Engineer vs. Data Scientist 4/6/2022 7 Source: https://elu.nl/careers-in-data-science-data-analyst-vs-data-engineer-vs-data-scientist/
- 8. 4/6/2022 8
- 9. 4/6/2022 9 Data Sources Processed Data Storage Data Ingestion Raw Data Storage Data Processing Data Visualization Understanding Data Layers
- 12. Data Provenance 4/6/2022 12 Source: https://www.canto.com/blog/data-lineage/
- 13. Data Wrangling vs. Data Pre-processing 4/6/2022 13 Source: https://medium.com/swlh/data-pre-processing-data-wrangling-4a6a8624e747 Data Pre-processing Data Wrangling
- 14. What are Distributed Systems? 4/6/2022 14
- 15. 15/20 • 1 Cluster = many machines (nodes) • Parallel computing • Scalability • Fault-tolerance (High availability)
- 16. 16/20 Data ingestion Data storage Data Processing
- 18. Understanding Data & Requirements 4/6/2022 18 Source: https://firebrand.training/uk/lea rn/pmp/course-material/project- scope-management/collect- requirements
- 19. Choosing a Paradigm 4/6/2022 19
- 24. Relational databases 4/6/2022 24 Source: https://www.stambia.com/en/solutions/by-technology/integration-to-database/relational-database-rdbms
- 25. Data Integration 4/6/2022 25 Source: https://deevita.com/top-data-integration-tools/
- 26. NoSQL databases 4/6/2022 26 Source: https://dock.ie/data.html
- 27. 4/6/2022 27 Big Data ecosystem Source: https://sranka.wordpress.com/2014/01/29/big-data-technologies/
- 28. 4/6/2022 28 Data Engineer Jobs
- 29. 4/6/2022 29 Stack Overflow: https://stackoverflow.com/jobs?q=data+engineer LinkedIn: https://www.linkedin.com/jobs/search/?geoId=92000001&keywords=data%20engineer&location=Remote
- 30. Required Skills 2. SQL, Python 3. Pandas library 4. AWS, Azure 5. Relational databases (SQL Server and MySQL) 6. Hadoop, Spark, Hive 7. NOSQL databases (MongoDB, Neo4j, and Cassandra) … 4/6/2022 30
- 31. 4/6/2022 31 1st Required Skill
- 33. 4/6/2022 33
- 34. 4/6/2022 34 First, make it run correctly (in an acceptable way). Then start improving performance
- 35. 4/6/2022 35 Again, make sure that you understand well the data and the requirements!
- 36. 4/6/2022 36
- 37. DON’T use Big Data technologies just because you have a huge volume of data. Follow best practices 4/6/2022 37
- 39. 4/6/2022 39 • Coursera: • Google Cloud - Data Engineering, Big Data, and Machine Learning on GCP Specialization • San Diego - Big Data Specialization • Udacity: • Data Engineering nanodegree • DataCamp: • Data Engineer with Python Track • IBM – CognitiveClass.ai • Free data science and data engineering courses • Udemy: • Data Science A-Z™: Real-Life Data Science Exercises Included