Prashant Kumar Pandey

Apache Spark - Scenario based question Lets say you have a 20 node spark cluster Each node is of size - 16 cpu cores / 64 gb RAM Let's say each node has 3 executors, with each executor of size - 5 cpu cores / 21 GB RAM => 1. What's the total capacity of cluster? We will have 20 * 3 = 60 executors Total CPU capacity: 60 * 5 = 300 cpu Cores Total Memory capacity: 60 * 21 = 1260 GB RAM => 2. How many parallel tasks can run on this cluster? We have 300 CPU cores, we can run 300 parallel tasks on this cluster. => 3. Let's say you requested for 4 executors then how many parallel tasks can run? so the capacity we got is 20 cpu cores 84 GB RAM so a total of 20 parallel tasks can run. => 4. Let's say we read a csv file of 10.1 GB stored in datalake and have to do some filtering of data, how many tasks will run? if we create a dataframe out of 10.1 GB file we will get 81 partitions in our dataframe. (will cover in my next post on how many partitions are created) so we have 81 partitions each of size 128 mb, the last partition will be a bit smaller. so our job will have 81 total tasks. but we have 20 cpu cores lets say each task takes around 10 second to process 128 mb data. so first 20 tasks run in parallel, once these 20 tasks are done the other 20 tasks are executed and so on... so totally 5 cycles, if we think the most ideal scenario. 10 sec + 10 sec + 10 sec + 10 sec + 8 sec first 4 cycles is to process 80 tasks all of 128 mb, last 8 sec is to process just one task of around 100 mb, so it takes little lesser but 19 cpu cores were free during this time. => 5. is there a possibility of, out of memory error in the above scenario? Each executor has 5 cpu cores and 21 gb ram. This 21 gb RAM is divided in various parts - 300 mb reserved memory, 40% user memory to store user defined variables/data. example hashmap 60% spark memory - this is divided 50:50 between storage memory and execution memory. so basically we are looking at execution memory and it will be around 28% roughly of the total memory allotted. so consider around 6 GB of 21 GB memory is meant for execution memory. per cpu core we have (6 GB / 5 cores) = 1.2 GB execution memory. That means our task can roughly handle around 1.2 GB of data. however, we are handling 128 mb so we are well under this range. I hope you liked the explanation :) Do mention in comment what you want me to bring in my next post! PS~ My new Data Engineering batch is starting on coming Saturday. DM to know more. #bigdata #dataengineering #apachespark

298

5 Comments

𝟓𝟎 𝐌𝐮𝐬𝐭-𝐊𝐧𝐨𝐰 𝐒𝐩𝐚𝐫𝐤 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 𝐭𝐨 𝐂𝐫𝐚𝐜𝐤 𝐘𝐨𝐮𝐫 𝐍𝐞𝐱𝐭 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 . . ➤ What is Lazy Evaluation in Spark? ➤ How do you add a new column in Spark? ➤ What is a Broadcast Join? ➤ What is the role of Catalyst Optimizer? ➤ What are Accumulators and Broadcast Variables? ➤ What makes Spark better than MapReduce? ➤ What is Checkpointing in Spark? ➤ How do you convert an RDD to a DataFrame? ➤ How would you handle a 10GB file in Spark and optimize it? ➤ What is the difference between Persist and Cache? ➤ What is Data Skewness? How do you handle it? ➤ What is an Out of Memory (OOM) issue, and how do you deal with it? ➤ What are Shared Variables in Spark? ➤ How can you read a CSV file without using an external schema? ➤ What is Cache in Spark? ➤ Difference between map and flatMap in Spark. ➤ How can you add two new columns to a DataFrame with calculated values? ➤ What is the advantage of a Parquet file? ➤ What is a Broadcast Variable? ➤ What are Transformations in Spark? What are their types? ➤ What are the different parameters that can be passed while using spark-submit? ➤ How do you drop a column in Spark? ➤ What happens in Cluster Deployment mode and Client Deployment mode? ➤ What is the difference between Order By, Sort By, and Cluster By? ➤ What is the difference between Coalesce and Repartition? ➤ What are the parameters you have used in spark-submit? ➤ What are the roles and responsibilities of the Driver in Spark architecture? ➤ What are the different types of joins in Spark? ➤ What are the challenges you face in Spark? ➤ What is the difference between Narrow Transformations and Wide Transformations? ➤ Difference between DataFrame & Dataset in Spark. ➤ What is a Skew Partition? ➤ What are the optimization techniques used in Spark? ➤ How do you convert a DataFrame to a Dataset? ➤ Can you explain Anti Join and Semi Join? ➤ Where can you find Spark application logs? ➤ Why Spark instead of Hive? ➤ What is DAG and Lineage in Spark? ➤ What is Spark Optimization? ➤ What are the Join Strategies in Spark? ➤ Avro vs ORC – which one do you prefer? ➤ What are some Actions in Spark that you have used? ➤ What are the parameters you have used in spark-submit? ➤ What is the difference between RDD & DataFrame? ➤ What is a Broadcast Variable? ➤ What is the difference between Cache and Persist? ➤ What is an Out of Memory (OOM) issue, and how do you deal with it? ➤ What are the optimization techniques used in Spark? ➤ What is a Broadcast Join? #PySpark #ApacheSpark #DataEngineering #BigData #SparkSQL #ETL #DataPipelines #DistributedComputing #CloudData #DataScience #Python #TechCareers #DataAnalytics #MachineLearning

73

3 Comments

Don’t get confused while learning PySpark. Understand these core concepts step by step to become truly proficient in PySpark. 𝗕𝗮𝘀𝗶𝗰𝘀 𝗼𝗳 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - PySpark Architecture - SparkContext and SparkSession - RDDs (Resilient Distributed Datasets) - DataFrames - Transformations and Actions - Lazy Evaluation 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲𝘀: - Creating DataFrames - Reading Data from CSV, JSON, Parquet - DataFrame Operations - Filtering, Selecting, and Aggregating Data - Joins and Merging DataFrames - Working with Null Values 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗖𝗼𝗹𝘂𝗺𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Defining and Using UDFs (User Defined Functions) - Column Operations (Select, Rename, Drop) - Handling Complex Data Types (Array, Map) - Working with Dates and Timestamps 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Understanding Partitions - Repartitioning and Coalescing - Managing Shuffle Operations - Optimizing Partition Sizes for Performance 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗮𝗻𝗱 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: - When to Cache or Persist - Memory vs Disk Caching - Checking Storage Levels 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗪𝗶𝘁𝗵 𝗦𝗤𝗟: - Spark SQL Introduction - Creating Temp Views - Running SQL Queries - Optimizing SQL Queries with Catalyst Optimizer - Working with Hive Tables in PySpark 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Data Cleaning and Preparation - Handling Missing Values - Data Normalization and Transformation - Working with Categorical Data 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗧𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Broadcasting Variables - Accumulators - PySpark Window Functions - PySpark with Machine Learning (MLlib) - Working with Streaming Data (Spark Streaming) 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Understanding Job, Stage, and Task - Tungsten Execution Engine - Memory Management and Garbage Collection - Tuning Parallelism - Using Spark UI for Performance Monitoring Once you master these PySpark fundamentals, you won’t just write code — you’ll design efficient, scalable, and production-ready data pipelines that harness the full potential of Spark.

270

33 Comments

Powering the Future of Data Engineering! From scalable storage to real-time analytics, cloud platforms offer powerful data solutions. ✅ Storage & Management – Azure Data Lake, AWS S3, GCP Cloud Storage ✅ Processing & Pipelines – Azure Data Factory, AWS Glue, GCP Dataflow ✅ Big Data & Analytics – Azure Synapse, AWS Redshift, Google BigQuery ✅ Real-Time & Streaming – Azure Stream Analytics, AWS Kinesis, Pub/Sub ✅ Visualization & BI – Power BI, AWS QuickSight, Looker Below is a breakdown of key Data Engineering functions. Microsoft Azure Azure SQL Database + SELECT = Data Retrieval  Azure Data Factory + Pipelines = Data Movement  Azure Synapse Analytics + Distributed Queries = Big Data Processing  Azure Data Lake + Hierarchical Storage = Scalable Data Management  Azure Stream Analytics + Real-time Processing = Instant Data Insights  Azure Cosmos DB + NoSQL = Global-Scale Applications  Azure Databricks + ML Models = AI & Advanced Analytics  Azure Functions + Event-driven Processing = Server less Data Workflows  Azure Monitor + Log Analytics = System Performance Tracking  Azure Purview + Metadata Management = Data Governance & Lineage ️ Power BI + Dashboards = Data Visualization & Business Intelligence Amazon Web Services (AWS) AWS S3 + Data = Scalable Storage AWS Glue + ETL = Data Transformation AWS Athena + SQL = Serverless Querying AWS Redshift + Analytics = Data Warehousing AWS Kinesis + Streaming = Real-time Data Processing AWS Lambda + Data Pipelines = Serverless Orchestration AWS DynamoDB + NoSQL = Fast & Scalable Datastore AWS Lake Formation + Governance = Secure Data Lakes AWS EMR + Spark = Big Data Processing AWS Step Functions + Workflow = Managed Data Orchestration AWS DMS + Migration = Seamless Database Replication AWS QuickSight + Visualization = Interactive Dashboards Google Cloud BigQuery + SELECT = Data Retrieval BigQuery + WHERE = Data Filtering  BigQuery + MATERIALIZED VIEWS = Precomputed Insights  Cloud Storage + GCS Buckets = Scalable Data Storage  Cloud Storage + Lifecycle Policies = Cost Optimization  Dataflow + Apache Beam = Scalable Data Processing  Dataflow + Streaming Pipelines = Real-Time Data Processing  Dataflow + Batch Pipelines = Scheduled Data Transformations  Pub/Sub + Topics & Subscriptions = Event-Driven Architecture  Pub/Sub + Message Retention = Reliable Messaging  Dataproc + Spark = Big Data Analytics  Dataproc + Hadoop = Distributed Data Processing  Cloud Composer + Apache Airflow = Orchestrating Data Pipelines  Data Fusion + GUI Pipelines = No-Code Data Integration  Looker + Dashboards = Data Visualization  Looker + Explorations = Self-Service BI 📙 Attaching a guide that compares Azure , AWS , and GCP Data Engineering cloud by aligning their core services with key engineering operations: 🚀 Join our thriving Data Engineering Community ✨  https://lnkd.in/gy4R55Tj Follow Abhisek Sahu for more . Please Like, repost ✅, if you find them useful. #cloud #engineering #dataengineering #bigdata

3,244

168 Comments

𝑰 𝒉𝒂𝒗𝒆 been there, 𝒅𝒐𝒏𝒆 𝒕𝒉𝒂𝒕 𝒂𝒏𝒅 𝒔𝒐 𝒄𝒂𝒏 𝒚𝒐𝒖!! ETL, PL/SQL developers, here's why you should learn #python NOW! ETL workloads are getting migrated to Cloud 𝐘𝐨𝐮 𝐰𝐢𝐥𝐥 𝐛𝐞 𝐚𝐬𝐤𝐞𝐝 - How do you move on prem etl workloads to #aws? Let's say you have your on prem etl workloads in ETL tooks like #informatica, #datastage, #abinitio, etc. which are primarily drag and drop tools. What do you do? You 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝 𝐭𝐡𝐞𝐦 𝐚𝐧𝐝 𝐫𝐞𝐛𝐮𝐢𝐥𝐝 𝐭𝐡𝐞 𝐞𝐧𝐭𝐢𝐫𝐞 𝐜𝐨𝐝𝐞 𝐢𝐧 𝐩𝐲𝐭𝐡𝐨𝐧 𝐨𝐫 𝐬𝐩𝐚𝐫𝐤 and then use services such as  Lambda 🖥️ ECS Fargate ⛴️ Glue Python Shell 🐍 Glue Spark 🔥 EMR 📊 etc. 𝐑𝐞𝐦𝐞𝐦𝐛𝐞𝐫, 𝐭𝐡𝐞 𝐛𝐚𝐬𝐢𝐜 𝐟𝐮𝐧𝐝𝐚 𝐫𝐞𝐦𝐚𝐢𝐧𝐬 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 - 𝐲𝐨𝐮 𝐞𝐱𝐭𝐫𝐚𝐜𝐭, 𝐲𝐨𝐮 𝐭𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦 𝐚𝐧𝐝 𝐲𝐨𝐮 𝐥𝐨𝐚𝐝.  Well, now you can also extract, load and then transform. Don't fret about ELT for now. 𝘛𝘳𝘶𝘴𝘵 𝘮𝘦, 𝘵𝘳𝘢𝘯𝘴𝘪𝘵𝘪𝘰𝘯𝘪𝘯𝘨 𝘧𝘳𝘰𝘮 𝘌𝘛𝘓 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘵𝘰 𝘢𝘯 𝘈𝘞𝘚 𝘋𝘢𝘵𝘢 𝘌𝘯𝘨𝘪𝘯𝘦𝘦𝘳 𝘪𝘴𝘯'𝘵 𝘷𝘦𝘳𝘺 𝘥𝘪𝘧𝘧𝘪𝘤𝘶𝘭𝘵. 𝑰 𝒉𝒂𝒗𝒆 been there, 𝒅𝒐𝒏𝒆 𝒕𝒉𝒂𝒕 𝒂𝒏𝒅 𝒔𝒐 𝒄𝒂𝒏 𝒚𝒐𝒖!! There's one thing you need to learn new thoug- Python! If you've read so far, do LIKE the post 👍 𝐏.𝐒: Learn no-nonsense AWS Data Engineering with 357+ others: https://aws.sachin.cloud & 𝐖𝐡𝐚𝐭𝐬𝐚𝐩𝐩 group for notification : https://w.sachin.cloud (phone privacy ensured, you won't see others )

167

24 Comments

Unity Catalog Update: What Changed & How It Impacts Us! Databricks just rolled out an important update related to governed tags in Unity Catalog. — What Changed : We can no longer drop a column that has governed tags assigned. The drop operation will fail until we remove those tags. This helps prevent accidental data leaks and keeps governance intact. — How to Fix It: Before dropping a column, remove the governed tags first: -- Step 1: Remove governed tag UNSET TAG ON COLUMN <catalog>.<schema>.<table>.<column> <tag_key>; -- Step 2: Drop the column ALTER TABLE <catalog>.<schema>.<table> DROP COLUMN <column>; -- Step 3: Vacuum to clear old data VACUUM <catalog>.<schema>.<table> RETAIN 0 HOURS; Real-Life Example: Let’s say we tagged a column email_id with a governed tag like PII=True. Now, if we try to drop this column directly, it’ll fail. We must first unset the tag, drop the column, and vacuum the table. Explanation This change enforces stricter governance on data with sensitive tags. It ensures we can’t remove tagged data without explicitly confirming it’s safe to do so… a small but crucial improvement for compliance and data protection. #Databricks #UnityCatalog #DataGovernance #DataEngineering

55

2 Comments

🔁 ETL Is Evolving — And So Must We. After 10+ years in ETL Development/Data Engineering, I’ve seen ETL move from nightly batch jobs on legacy systems… → to real-time pipelines on distributed cloud architectures. → to event-driven microservices with metadata-aware orchestration. But one thing hasn’t changed: 🚨 Bad data = bad decisions. Here’s what I believe separates a great ETL developer from a script-writer: ✅ Builds for data trust — not just delivery ✅ Designs for change — not just current state ✅ Understands the business impact — not just the pipeline flow In my journey, I’ve worked with AWS Glue, Apache Hudi, EMR, BigQuery, Databricks, and Airflow to automate pipelines for billions of rows of data — and the biggest lessons always came from production failures and late-night incident calls. If you’re starting your ETL career or leading teams — invest in: 🔹 Metadata-first thinking 🔹 Observability & lineage 🔹 Communicating data value, not just structure 💬 I'd love to hear from fellow engineers: What's one lesson you learned the hard way in ETL that still guides you today? #DataEngineering #ETL #CloudData #BigData #AWS #GCP #ApacheSpark #DataPipelines #Leadership #CareerInTech

63

3 Comments