![MapReduce over
snapshots
HBASE-8369
Enis Soztutar
Enis [at] apache [dot] org
@enissoz
© Hortonworks Inc. 2011
Page 1](https://image.slidesharecdn.com/mapreduceoversnapshots-131025133030-phpapp02/85/Mapreduce-over-snapshots-1-320.jpg)
![Thanks
Questions?
Enis Söztutar
enis [ at ] apache [dot] org
@enissoz
Architecting the Future of Big Data
© Hortonworks Inc. 2011
Page 16](https://image.slidesharecdn.com/mapreduceoversnapshots-131025133030-phpapp02/85/Mapreduce-over-snapshots-16-320.jpg)
Mapreduce over snapshots
- 1. MapReduce over snapshots HBASE-8369 Enis Soztutar Enis [at] apache [dot] org @enissoz © Hortonworks Inc. 2011 Page 1
- 2. About Me • In the Hadoop space since 2007 • Committer and PMC Member in Apache HBase and Hadoop • Working at Hortonworks as member of Technical Staff • Twitter: @enissoz Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 2
- 3. Snapshots • Currently a snapshot is a bunch of reference files together with some metadata • A table’ snapshot can contain – Table descriptor – List of regions – References to files in the regions – References to WALs for regionservers • Current snapshot impl is flush based – Forces flush to all regions, so that in-memory data is written to disk Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 3
- 4. MR over Snapshots • Idea is do scan’s on the client side bypassing region servers • Use snapshots since they are immutable • Similar to short circuit hdfs reads • TableSnapshotInputFormat works similar to TableInputFormat • TableMapReduceUtil methods to configure the job Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 4
- 5. Deployment Options HBase online • Take snaphot while HBase is running • Run MR job over the snapshot HBase offline • Take snapshot while HBase is running • Export Snapshot using ExportSnapshot to a different hdfs • Run MR job over snapshot with or without HBase running Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 5
- 6. TableSnapshotInputFormat • Gets a Scan representing the query • Restore the snapshot to a temporary directory • For each region in the snapshot: – Determine whether the region should be scanned (falls between scan start row and stop row) – Create one split per region in the scan range ( # of map tasks) – Each RecordReader will open the region (Hregion) as in HRegionServer – An internal RegionScanner is used for running the scan Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 6
- 7. API Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 7
- 8. Timeline • Will (hopefully) be committed to trunk next week or so • Interest in bringing this to 0.94 and 0.96 bases as well • Will come in HDP-2.1, which will be based on 0.96 line Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 8
- 9. Security Aspects • HBase user owns the files in filesystem • Snapshot files are also owned by the HBase user • Mapreduce job should be able to read the files in the snapshot + actual data files • HDFS only has posix-like perms based on user/group/other – User running MR job has to be either the HBase user, or have group perms – HDFS does not have ACL’s, so there is no easy way to grant read access at filesystem layer • Idea: similar to current short circuit impl, we can implement a FD transfer – User will submit jobs under her own user credentials – Ask HBase daemons to open the files, and pass a handler / token Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 9
- 10. Performance ScanTest: • Scan : open a scanner, do full table scan • SnapshotScan : open a client-side scanner, do full table scan • ScanMR : parallel full table scan from MR • SnapshotScanMR : do full table scan • • • • 8 Region servers, 6 disks each HBase trunk Hadoop-2.2 (HDP-2.0.7.0-12) Load data with IntegrationTestBulkLoad – Evenly distributed rows, created as bulk loaded hfiles. 3 column families • # store files per region varies 3,6,9, and 12 (1,2,3,4 file per store) • Data sizes: 6.6G, 13.2G, 19.8G, 26.4G Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 10
- 11. Scan speed Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 11
- 12. API • We do not want to limit snapshot scanning only to MapReduce • Allow client side scanners over snapshot files Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 12
- 13. ResultScanner is main scan API Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 13
- 14. API (caution: not final yet) Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 14
- 15. To the future and beyond • HBASE-8691 High-Throughput Streaming Scan API • Can we bypass regionservers without taking snapshots? • Bypass memstore data, or stream memstore data, but read directly from hfiles • Secure reading from snapshots • Keep up with the updates at – https://issues.apache.org/jira/browse/HBASE-8369 Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 15
- 16. Thanks Questions? Enis Söztutar enis [ at ] apache [dot] org @enissoz Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 16