HBaseConAsia2018 Track3-2: HBase at China Telecom
- 1. hosted by HBase at China Telecom Chen Ze
- 2. hosted by Content 01 02 03 Infrastructure & Application HBase usage at China Telecom How do we monitor HBase and optimization Q&A Monitor & Optimization
- 3. hosted by Infrastructure & Application HBase usage at China Telecom 01
- 4. hosted by Infrastructure Offline computing platform HDFS Hive Impala Storm Elasticsearch Real-time computation platform Telecom Open platform Ad-hoc job scheduling Cluster managerment Monitor & Alert Spark Streaming Spark Sqoop DataX Flume FTP ETL development MapReduce Pig YARN Tachyon Tez Zookeeper HBase SQL MLlibMahout Data exchange Data collecting Kerberos ……
- 5. hosted by China Telecom HBase Platform • 322 hosts in a independent HDFS cluster ,32 cores，256GB memory，3.6T*12 disk • 6 HBase clusters with different kinds of application Persistence for Streaming Jobs Online writing/reading Kylin support • 520 TB data ，1 TB/day • HBase1.2.0--CDH 5.12.1
- 6. hosted by Persistence for Streaming Jobs • core system • data collecting system
- 7. hosted by Data collecting system Collect different kinds of data with different kinds of collecting method Collecting system accounts data DPI/signal/CDR … OSS MSS CRM/VSOP Quasi-real-time data real-time data batch data CRM /VSOP DPI/signal/CDR MSS accounts data OSS Core system
- 8. hosted by Data collecting system OSS data collecting use HBase replication to receive data from 31 Provincial branches HBase Provincial branches HBase HBase HBase Shanghai data center replication HDFS
- 9. hosted by Core System • core system use HBase to store middle layer data and Aggregation layer data through Spark Streaming
- 10. hosted by Online writing/reading application • DPI data service • Location-based data service
- 11. hosted by • DPI data service • What is DPI data?
- 12. hosted by • DPI data service • How to use DPI data ?
- 13. hosted by • Location-based data service Kafka HDFS/Hive Spark cluster Hbase cluster WEB/RestAPI DPI/signal/CDR data Data from Base station Mysql FTP
- 14. hosted by • Location-based data service
- 15. hosted by Monitor & Optimization How do we monitor HBase and optimization 02
- 16. hosted by • HBase Monitor • Tools ganglia-3.6.0 Zabbix-3.2.4 • Ganglia for HBase basic metrics • Zabbix for important items to alert
- 17. hosted by • HBase ganglia configuration Edit HBase config file hadoop-metrics2-hbase.properties Than we can see many metrics items
- 18. hosted by • Important metrics Type Items System status ping Available memory in percent Disk usage HDFS status NameNode port DataNode port JournalNode port HDFS avaliable space HBase status HBase Master Port HBase Region Server Port Zookeeper status ZooKeeper Port ZKFC Port HBase RPC regionserver.TotalCallTime regionserver.ProcessCallTime regionserver.QueueCallTime regionserver.numActiveHandler regionserver.ipc.numCallsInGeneralQueue regionserver.ipc.numOpenConnections regionserver.RegionServer.numCallsInWriteQueue regionserver.RegionServer.numCallsInReadQueue Hbase IO regionserver.Server.Mutate_99th_percentile regionserver.wal.SyncTime_99th_percentile regionserver.server.Get_99th_percentile regionserver.server.ScanTime_99th_percentile
- 19. hosted by • Important metrics Type Items HBase region Region num,size BlockCache hit radio JVM GC jvm.JvmMetrics.GcTimeMillis jvm.JvmMetrics.GcCount GC log
- 20. hosted by • HBase Debug case • One day, our core data system hanged when trying to connection Hbase cluster…
- 21. hosted by HBase Optimization • CMS vs G1 • Read/Write Splitting and some Optimization
- 22. hosted by CMS vs G1 CMS to G1 Why?
- 23. hosted by Read/Write Splitting Optimization • Use replication • Use Kerberos • Optimization hbase.ipc
- 24. hosted by Q&A
- 25. hosted by Thanks
Editor's Notes
- #5: Ad hoc 即席查询