hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
- 2. Phoenix Practice in China Life Insurance Co., Ltd. Leo Yuan [email protected]
- 3. 1 Scenarios 2 Designs 3 Optimizations 4 Problems 5 Future Work Agenda
- 4. Scenarios - Overview Main Processing Cluster A Application Procession Cluster C Readonly cluster B Sync basic data Sync result Real Time Query Cluster E
- 5. Scenarios - Overview 4 Cluste200+ Nodes（30+ Phoenix Nodes） Clusters Data 1300TB+ data 30TB+ biggest table Querys Ten million level perday Processing Hundreds of MR/Hive/Spark jobs per day 50TB+ Incremental data for update & insert
- 6. Scenarios - Overview step 1 get some money from company counter step 2 query operation detail from app This Is What We Do Real Cust Rights View System !
- 7. Scenarios - Overview 30,000,000 + Incremental data/day 700,000 + users / day 10,000+ records/sec 8,000+ sqls / sec
- 8. 1 Scenarios 2 Designs 3 Optimizations 4 Problems 5 Future Work Agenda
- 9. Designs - Data Architecture ü Initialize Data ü Build Phoenix Index ü Sync Real Time Data ü Provide Data Service China Life Insurance APP Business System SharePlex Kafka Spark Streaming Cust View System Phoenix HBase
- 10. Designs - Development Architecture Data Integration Data View Cross-province Integration Query Distribution Engine V_constract table (4+3) Data Service Ultimate Real Time Query Service Data Exchange Initial Data Sync Program Data Source Business System DataBase JobSchedule Monitor Resourceschedule monitor Schedule Monitor Real Time Data Link Incremental Data Sync Program V_pay table (4+3) V_cust service table (4+3) V_claim table (4+3) … dim Resourceaccess control UserAccess Control Privilege Control
- 11. Designs - Physical Architecture Phoenix Cluster Configuration DB Weblogic Server Weblogic Server Weblogic Server F5APP Server Phoenix Gateway
- 12. Kafka Designs - Real Time Data Sync cid c_no type amount branch syssource updtime incr_flag commit_time …… 001 001 M 100 000000 V6 2019-06-11 16:37.322 1 2019-06-11 16:37.322 …… 02 002 R 100 000001 V7 2019-06-11 16:38.689 2 2019-06-11 16:38.689 …… Contract（BeiJing） cid c_no type amount 01 001 M 100 1、Partition By Global Primary Key 2、Shield Upstream System Table Structure Adjustment 3、No Effect on Normal Stream Process when Data Supplement … … …… SparkStreaming Contract（ShangHai） cid c_no type amount 02 002 R 100 SparkStreaming_compt
- 13. Designs - Ultimate Real Time Query Service label name label type isHolder logic paidBonus sum holder sql1,2,3 paidMoenyList list holder sql4,5 paidExpire sum Insured sql6 … … … … label.properties Analyze Parameters Package Sqls Execue Sqls Collect Results Chinalife Insurance APP Ultimate Real Time Query Service
- 14. 1 Scenarios 2 Designs 3 Optimizations 4 Problems 5 Future Work Agenda
- 15. Optimizations - Sql Execution Process DriverManager.getConnection("phoenixUrl") con.prepareStatement(sql) pstat.executeQuery() rs.next() 1、SYSTEM.CATALOG 2、SYSTEM.STATS 3、SYSTEM.LOG 4、 SYSTEM.SEQUENCE 1、Query meta data of table/index from phoenix server(SYSTEM.CATALOG) 2、Determine the table/index sql need to scan 3、Query statistics information of table/index from phoenix server(SYSTEM.STATS) 4、Generate scans based on statistics information、meta data、sql 1、Parallelity decided by phoenix.query.threadPoolSize
- 16. Optimizations – Phoenix System Table SYSTEM.CATALOG SYSTEM.STATS Describe table/index meta information, such as l TABLE NEME l COLUMN NAME l SALT_BUCKETS l UPDATE_CACHE_FREQUENCY l GUIDE_POST_WIDTH Describe table/index accurate statistics information, such as l GUIDE_POST_KEY l GUIDE_POSTS_WIDTH PS:UPDATE STATISTICS TABLE_NEME
- 17. Optimizations - RS Group Hmaster SYSTEM:CATALOG SYSTEM:MUTEX SYSTEM:STATS CONTRACT INCOME PERSON RS Group 1 RS Group 2 rs1 SYSTEM:CATALOG SYSTEM:STATS Hmaster rs2 rs3 rs4 … rs1 rs3 rs4 …rs2 …… SYSTEM:STATSCONTRACTINCOME PERSON …… • Metadata Table Isolation to Decrease Impact on Business Table Query
- 18. Optimizations - UPDATE_CACHE_FREQUENCY • Adjust this parameter to decrease hotspot in SYSTEM.CATALOG p Decide the query frequency of SYSTEM.CATALOG p Default value is “Always” and will Cause read/write pressure in SYSTEM.CATALOG p Can be set per Cluster/Table 1. “phoenix.default.update.cache.frequency”: 86412345 2. create table test.test (a varchar not null primary key,b varchar ) SALT_BUCKETS = 10, UPDATE_CACHE_FREQUENCY=86400000;
- 19. Optimizations – Salt & Pre-Split • Data table use salt_buckets, Index table use pre-split CREATE TABLE …( )SALT_BUCKETS = 60 CREATE INDEX … ON …( … ) INCLUDE ( … ) ASYNC SALT_BUCKETS = 0 SPLIT ON ( … ) 1、Create index to ensure that all of query process stop at index table 2、use pre split to reduce the chunk number when execute a query 3、use async index to avoid OOM when building the index table 1、Use salted table to distribute data evenly Phoenix Client write Phoenix Client query
- 20. Optimizations - Open Offheap • Minimal read cost to improve query efficiency BucketCache Configuration Properties l hbase.bucketcache.combinedcache.enabled l hbase.bucketcache.ioengine l hfile.block.cache.size l hbase.bucketcache.size l hbase.bucketcache.bucket.sizes l -XX:MaxDirectMemorySize
- 21. Optimizations - Other Configurations • Region Balance By Table • G1GC • Manual MajorCompaction
- 22. 1 Scenarios 2 Designs 3 Optimizations 4 Problems 5 Future Work Agenda
- 23. Problems - Current time function cause Query performance degradation • This function will lead to client-cluster interaction frequently SELECT … FROM … WHERE date <= CURRENT_TIME() SELECT … FROM … WHERE date <= CURRENT_TIME() SELECT … FROM … WHERE date <= NOW_DATE replace CURRENT_TIME In java code to NOW_DATE ① ② ③
- 24. Problems - HBase cluster balance abnormal 1.Turn on RSGroup 2.Set two RSGroup default 、my_group 3.Move rs to default and my_group 4.Restart one rs in default process Problems Resolve org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer 2. balance_rsgroup 'default' abnormal 1. balancer abnormal
- 25. Problems - ACL abnormal when come together with RSGroup 1.Turn on RSGroup 2.Turn on ACL process Problems 1. Non-hbase user can’t creat table Resolve 1, Use hbase user to create table 2, grant this table 'RWX’ to Non-hbase user <property> <name> hbase.coprocessor.master.classes </name> <value> org.apache.hadoop.hbase.security. access.AccessController, org.apache.hadoop. hbase.rsgroup.RSGroupAdminEndpoint </value> </property>
- 26. 1 Scenarios 2 Designs 3 Optimizations 4 Problems 5 Future Work Agenda
- 27. Future Work - RPC Read/Write Isolation total queue = hbase.ipc.server.callqueue.handler.factor * handler read queue = total queue * hbase.ipc.server.callqueue.read.ratio write queue = total queue * (1- hbase.ipc.server.callqueue.read.ratio ) scan queue = = total queue * hbase.ipc.server.callqueue.read.ratio * hbase.ipc.server.callqueue.scan.ratio total queue write queue read queue scan queue
- 28. Future Work - Compaction Contral Set Offpeak Time Set Peak Time Throughput l hbase.offpeak.end.hour l hbase.offpeak.start.hour l hbase.hstore.compaction.throughput.offpeak l key:hbase.regionserver.throughput.controller value:org.apache.hadoop.hbase.regionserver.compactions. PressureAwareCompactionThroughputController Open Compaction Controller l hbase.hstore.compaction.throughput.higher.bound l hbase.hstore.compaction.throughput.lower.bound
- 29. Future Work - Join Optimization SELECT … FROM ( a1 JOIN (SELECT... FROM b1 WHERE …) a2 ON a1…. = a2…) WHERE … EXPLAIN