Big Telco - Yousun Jeong
17 likes•4,303 views
This document provides an overview of SK Telecom's use of big data analytics and Spark. Some key points: - SKT collects around 250 TB of data per day which is stored and analyzed using a Hadoop cluster of over 1400 nodes. - Spark is used for both batch and real-time processing due to its performance benefits over other frameworks. Two main use cases are described: real-time network analytics and a network enterprise data warehouse (DW) built on Spark SQL. - The network DW consolidates data from over 130 legacy databases to enable thorough analysis of the entire network. Spark SQL, dynamic resource allocation in YARN, and integration with BI tools help meet requirements for timely processing and quick
![11
Use case 2: Network Enterprise DW
NMS#1
DBMS
…
NMS#1
DBMS
NMS#N-1
DBMS
[ Current ]
Siloed Data & IT Management
Access NW Core NW Transport
Expected advantages
• Unification of 130+ legacy DMBSs, each of which was managing separate network
monitoring system, enabling thorough analysis over the entire network
• Quick and accurate identification of root causes of network failure
Data scientists need unified platform to collect data from all network equipment
for management and analysis purpose
NMS
#1 …
NMS
#2
NMS
#N-1
Legacy
NMS
#N
Hadoop DW
DW
Legacy
NEWN
MS#1
… NEW
NMS#
N
BI &
Analytic
…
[ Goal (4Q, 2015) ]"
Network Enterprise DW](https://image.slidesharecdn.com/04yousunjeong-151030084956-lva1-app6891/85/Big-Telco-Yousun-Jeong-11-320.jpg)
Big Telco - Yousun Jeong
- 1. ‹#› Big Telco Real-Time Network Analytics Yousun Jeong
- 2. Who am I • Senior Software Engineer of SK Telecom, South Korea’s largest wireless communications provider • Work on commercial products (~ ’15) - She worked with Hadoop DW - She worked with IaaS(OpenStack) - She worked with PaaS(CloudFoundry) • Mail to : [email protected] 2
- 3. 3 Table of Contents 1. Big Data in SK Telecom 2. Benefit of Spark 3. Spark Real Workload Real-Time Network Analytics 4. Ongoing R&D
- 4. Big Data in SKT in a Nutshell ✓ Data Size - Currently collecting 250 TB/day ! ✓ Big Data Management Infrastructure - Hadoop cluster (1400+ nodes); migrated from MPP RDBMS ✓ Use cases - Real-Time Analytics of Base Stations - Network Enterprise DW ! ✓ Ongoing R&D - SKT Hadoop DW Appliance with H/W acceleration 4
- 5. Operating over 1400 nodes (30 PB+) of Hadoop cluster SKT Hadoop Infrastructure • Optimized configuration • Fault tolerant and effective resource management system 5 Data Collector Data Collect " & pre-processing Main Cluster Analysis R&D Cluster ~250 TB/day (500+ node) Service! Logic Repository (400+ Node) (100+ node) Service Cluster (400+ node) Marketing NW Analytics VoC SKT Hadoop Infra Data Feeding Data Feeding Commercialize Develop.
- 6. Batch LayerInterface Layer Flume Kafka" HDFS (Data Mart) oozie (workflow) Hive (ETL) Spark (ETL) Analytics Layer 1 2 Spark SQL Spark MlLib Spark GraphX Spark R YARN (Unified Resource Manager) Real-Time Layer NoSQL Elastic Search HDFS Data Service Layer BI Legacy App 3 Analytics Layer Batch Processing Layer - Hadoop EDW Real-Time Processing Layer – Real Time Analysis 3 1 2 【 Components 】 Spark Streaming" ! H/W Accelerator (SSD, FGPA) Cluster Manger Ambari SKT Big Data Reference Architecture Designed to handle both real-time & batch data processing and high level analysis using Spark as a core technology 6
- 7. Benefit of Spark Spark help us to have the gains in processing speed and implement various big data applications easily and speedily ▪ Support for Event Stream Processing ▪ Fast Data Queries in Real Time ▪ Improved Programmer Productivity ▪ Fast Batch Processing of Large Data Set Why SKT use spark … 7
- 8. Use cases: Summary Network Enterprise DW APOLLO • End-to-end network quality assurance and fault analysis in a timely manner • Real-time analysis of radio access network to improve operation efficiency Network analytics 8
- 9. 9 DC Parser Kafka" Broker Kafka" Producer Kafka" Topic Spark Streaming Kafka Direct" Stream" 1 minute widow 10 s HDFS ES 10 s Real-Time Dashboard Spark SQL BI Analysis JDBC" ODBC 1 2 4 5 Data Collector" (Flume) 3 Spark MLlib 6 Timely Processing" Quick Response Requirements Parallelism • Executors • Partitions • Using Akka Use case 1: Requirements & Challenges
- 10. “Hadoop S/W and Commodity H/W Based Cost-effective IT Infrastructure System” 【 SKT DW Infrastructure】 “High-price, High-performance Proprietary IT Infrastructure System” 【 Legacy IT Infrastructure 】 ※ MPP Massively Parallel Processing, SAN Storage Area Network, NAS Network Attached Storage, RDBMS Relational DB Management System Structured/Un-structured Data Scale-out Structure (Petabyte, Exabyte)Data Structured Data Scale-up Structure (Terabyte) Commodity H/W (x86 Server)H/W High Performance H/W (MPP, Fabric Switch, etc.) Hadoop Architecture SQL on Hadoop S/W Proprietary S/W (RDBMS, etc.) Transaction/Batch Processing" (SQL) Hadoop File System Hadoop DW can handle telco big data with scalability & cost efficiency Use case 2: Hadoop based Enterprise DW 10 ※ MPP Massively Parallel Processing
- 11. 11 Use case 2: Network Enterprise DW NMS#1 DBMS … NMS#1 DBMS NMS#N-1 DBMS [ Current ] Siloed Data & IT Management Access NW Core NW Transport Expected advantages • Unification of 130+ legacy DMBSs, each of which was managing separate network monitoring system, enabling thorough analysis over the entire network • Quick and accurate identification of root causes of network failure Data scientists need unified platform to collect data from all network equipment for management and analysis purpose NMS #1 … NMS #2 NMS #N-1 Legacy NMS #N Hadoop DW DW Legacy NEWN MS#1 … NEW NMS# N BI & Analytic … [ Goal (4Q, 2015) ]" Network Enterprise DW
- 12. Network EDW is a Hadoop-based data warehouse built on Spark for various network statistics or raw data User Benefits • End-to-End quality assurance, Fault analysis • Reduces analysis lead time (days → minutes) • Saves TCO (1/5 less than legacy DW) ! Hadoop DW • Spark-SQL functions and query optimizer • Bulk-loading and timely processing of large data • SSD caching applied for performance enhancement Acess Core Transport EMS EMS T-Pani EMS Hadoop DW DW Data Data Mart SQL on Hadoop (Spark SQL) IP EMS AnalyticsSQL ETL ETL O! D! S MQE* (Meta Query Engine) H/W Accelerator ! SSD Caching H/W Accelerator SSD Caching BI * MQE (Meta Query Engine) : Heterogeneous database integration query, including the Hadoop. Use case 2: Network Enterprise DW 12
- 13. 13 https://github.com/bitnine-oss/octopus Use case 2: Meta Query Engine Features" 1. Subset of ANSI-SQL" 2. Queries on multiple databases including Spark-SQL, Oracle." 3. SQL-based authorization" 4. User authentication" 5. Unified schema view
- 14. Use case 2: Requirements & Challenges Timely Processing -ETL" Integrated BI Tools" Quick Response Requirements 14 MDS #1 MQE #1 HA Proxy Thrift Server #1 Thrift Server #2 Spark SQL HDFS YARN WEB MDS BI MQE Meta Store Octopus NW EDW # 96 ETL Spark 3 2 1 4
- 15. Use case 2: YARN(Dynamic Resource Allocation) 15 spark.dynamicAllocation.enabled true! spark.shuffle.service.enabled true! spark.dynamicAllocation.minExecutors 50! spark.dynamicAllocation.maxExecutors 150! spark.dynamicAllocation.initialExecutors 50! spark.dynamicAllocation.cacheExecutorIdleTimeout 600! spark.dynamicAllocation.executorIdleTimeout! 5! spark.dynamicAllocation.schedulerBacklogTimeout! ! 5! spark.dynamicAllocation.sustainedSchedulerBacklogTimeout! 5 <property>! <name>yarn.nodemanager.aux-services</name>! <value>mapreduce_shuffle,spark_shuffle</value>! </property>! <property>! <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>! <value>org.apache.spark.network.yarn.YarnShuffleService</value>! </property> Configuration
- 16. Use case 2: BI Integration 16 spark.sql.thriftServer.incrementalCollect true! spark.driver.maxResultSize 10g Configuration
- 17. Use case 2: Patches 17 SPARK-7792! - HiveContext registerTempTable not thread safe! SPARK-7936! - Add configuration for initial size and limit of hash for aggregation! SPARK-8153! - Add configuration for disabling partial aggregation in runtime! SPARK-8285! - CombineSum should be calculated as unlimited decimal first! SPARK-8312! - Populate statistics info of hive tables if it's needed to be! SPARK-8333! - Spark failed to delete temp directory created by HiveContext! SPARK-8334 ! - Binary logical plan should provide more realistic statistics! SPARK-8357! - Memory leakage on unsafe aggregation path with empty input! SPARK-8420! - Inconsistent behavior with Dataframe Timestamp between 1.3.1 and 1.4.0! SPARK-8552! - Using incorrect database in multiple sessions! SPARK-8707! - RDD#toDebugString fails if any cached RDD has invalid partitions! SPARK-8826! - Fix ClassCastException in GeneratedAggregate! SPARK-9685! - Unspported dataType: char(X) in Hive! SPARK-10151! - Support invocation of hive macro! SPARK-10152! - Support Init script for hive-thriftserver! SPARK-10679! - javax.jdo.JDOFatalUserException in executor! SPARK-10684! - StructType.interpretedOrdering need not to be serialised! SPARK-10216 - Avoid creating empty files during overwrite into Hive table with group by query Open Issues
- 18. Use case 2: Performance 18 TPC-H
- 19. Use case 2: Performance 19 Job Server
- 20. Hadoop DW Appliance (ongoing) 【 SKT Hadoop DW Appliance 】 Management & Automation Core Software Solution Hardware Acceleration 3 1 2 ▪ Develop Interactive Spark SQL ▪ Develop Meta Query Engine ▪ Develop Flash Storage-based I/O Acceleration ▪ Develop FPGA-based CPU Acceleration ▪ Develop Data & System Security ▪ Workload Optimization & Automation Industry Oriented Solution4 ▪ Fault Detection & Classification in Manufacturing ▪ Mobile Network Data Analytic Solution ▪ Unstructured Data Collection/Processing Solution Develop a Hadoop DW appliance combining optimized S/W layer and H/W acceleration 20 H/W Acceleration Layer Data Processing Layer * Meta Query Engine DW Management Layer Industry" Oriented Solution ! ! ! ! ! ! ! Monitoring DB Migration Security OptimizationPackaging SQL Engine/Storage " ! ! ! * SPARK HIVE Legacy RDBMS FDC Telco others Hadoop Storage DB Storage * Flash based I/O Accelerator * FPGA Accelerator 2 1 3 4
- 21. 21 Thank You!