Storing and manipulating graphs in HBase
10 likes12,517 views
This document discusses storing and manipulating graphs in HBase. It provides an overview of graph theory concepts and different modeling techniques for storing graphs in HBase, including the adjacency matrix and adjacency list approaches. It also covers techniques for distributed traversal and querying of graphs stored in HBase using MapReduce jobs. Key tips discussed include implementing custom comparators and using utilities like MultiScanTableInputFormat and TableMapReduceUtil.
![Don’t get fancy with byte[]
class EdgeValueWritable implements Writable {
EdgeValue edgeValue
byte[] toBytes() {
// use strings if you can help it
}
static EdgeValueWritable fromBytes(byte[] bytes) {
// use strings if you can help it
}
}
groovy](https://image.slidesharecdn.com/storingandmanipulatinggraphsinhbase-updated-120524173208-phpapp02/85/Storing-and-manipulating-graphs-in-HBase-41-320.jpg)
![Querying by vertex
def get = new Get(vertexKeyBytes)
get.addFamily(edgesFamilyBytes)
Result result = table.get(get);
result.noVersionMap.each {family, data ->
// construct edge objects as needed
// data is a Map<byte[],byte[]>
}](https://image.slidesharecdn.com/storingandmanipulatinggraphsinhbase-updated-120524173208-phpapp02/85/Storing-and-manipulating-graphs-in-HBase-42-320.jpg)
![Do implement your own comparator
public static class Comparator
extends WritableComparator {
public int compare(
byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2) {
// .....
}
}
java](https://image.slidesharecdn.com/storingandmanipulatinggraphsinhbase-updated-120524173208-phpapp02/85/Storing-and-manipulating-graphs-in-HBase-59-320.jpg)
Storing and manipulating graphs in HBase
- 1. Storing and Manipulating Graphs in HBase Dan Lynn [email protected] @danklynn
- 2. Keeps Contact Information Current and Complete Based in Denver, Colorado CTO & Co-Founder
- 3. Turn Partial Contacts Into Full Contacts
- 6. Refresher: Graph Theory rt ex Ve
- 7. Refresher: Graph Theory Edg e
- 9. Tweets @danklynn retweeted “#HBase rocks” follows author @xorlev
- 10. Web Links http://fullcontact.com/blog/ <a href=”...”>TechStars</a> http://techstars.com/
- 11. Why should you care? Vertex Influence - PageRank - Social Influence - Network bottlenecks Identifying Communities
- 12. Storage Options
- 13. neo4j
- 14. neo4j Very expressive querying (e.g. Gremlin)
- 16. neo4j Data must fit on a single machine :-(
- 17. FlockDB
- 20. FlockDB No multi-hop query support :-(
- 21. RDBMS (e.g. MySQL, Postgres, et al.)
- 23. RDBMS Huge amounts of JOINing :-(
- 29. Adjacency Matrix 1 3 2
- 30. Adjacency Matrix 1 2 3 1 0 1 1 2 1 0 1 3 1 1 0
- 31. Adjacency Matrix Can use vectorized libraries
- 32. Adjacency Matrix Requires O(n2) memory n = number of vertices
- 33. Adjacency Matrix Hard(er) to distribute
- 34. Adjacency List 1 3 2
- 35. Adjacency List 1 2,3 2 1,3 3 1,2
- 36. Adjacency List Design in HBase e:[email protected] p:+13039316251 t:danklynn
- 37. Adjacency List Design in HBase row key “edges” column family e:[email protected] p:+13039316251= ... t:danklynn= ... p:+13039316251 e:[email protected]= ... t:danklynn= ... t:danklynn e:[email protected]= ... p:+13039316251= ...
- 38. Adjacency List Design in HBase row key “edges” column family e:[email protected] p:+13039316251= ... t:danklynn= ... at to W e?h p:+13039316251 e:[email protected]= ... st or t:danklynn= ... t:danklynn e:[email protected]= ... p:+13039316251= ...
- 39. Custom Writables package org.apache.hadoop.io; public interface Writable { void write(java.io.DataOutput dataOutput); void readFields(java.io.DataInput dataInput); } java
- 40. Custom Writables class EdgeValueWritable implements Writable { EdgeValue edgeValue void write(DataOutput dataOutput) { dataOutput.writeDouble edgeValue.weight } void readFields(DataInput dataInput) { Double weight = dataInput.readDouble() edgeValue = new EdgeValue(weight) } // ... } groovy
- 41. Don’t get fancy with byte[] class EdgeValueWritable implements Writable { EdgeValue edgeValue byte[] toBytes() { // use strings if you can help it } static EdgeValueWritable fromBytes(byte[] bytes) { // use strings if you can help it } } groovy
- 42. Querying by vertex def get = new Get(vertexKeyBytes) get.addFamily(edgesFamilyBytes) Result result = table.get(get); result.noVersionMap.each {family, data -> // construct edge objects as needed // data is a Map<byte[],byte[]> }
- 43. Adding edges to a vertex def put = new Put(vertexKeyBytes) put.add( edgesFamilyBytes, destinationVertexBytes, edgeValue.toBytes() // your own implementation here ) // if writing directly table.put(put) // if using TableReducer context.write(NullWritable.get(), put)
- 44. Distributed Traversal / Indexing e:[email protected] p:+13039316251 t:danklynn
- 45. Distributed Traversal / Indexing e:[email protected] p:+13039316251 t:danklynn
- 46. Distributed Traversal / Indexing e:[email protected] p:+13039316251 Pi v ot v e rt ex t:danklynn
- 47. Distributed Traversal / Indexing e:[email protected] p:+13039316251 Ma pReduce ove r out bou nd edges t:danklynn
- 48. Distributed Traversal / Indexing e:[email protected] p:+13039316251 Em it vertexes an d edge dat a gro upe d by the piv ot t:danklynn
- 49. Distributed Traversal / Indexing Re duc e key p:+13039316251 “Ou t” vertex e:[email protected] t:danklynn “In” vertex
- 50. Distributed Traversal / Indexing e:[email protected] t:danklynn Re duc er em its higher-order edge
- 51. Distributed Traversal / Indexing Ite rat ion 0
- 52. Distributed Traversal / Indexing Ite rat ion 1
- 53. Distributed Traversal / Indexing Ite rat ion 2
- 54. Distributed Traversal / Indexing Reuse edges created during previ ous iterat ions Ite rat ion 2
- 55. Distributed Traversal / Indexing Ite rat ion 3
- 56. Distributed Traversal / Indexing Reuse edges created during previ ous iterat ions Ite rat ion 3
- 57. Distributed Traversal / Indexing hop s req uires on ly ite rat ion s
- 58. Tips / Gotchas
- 59. Do implement your own comparator public static class Comparator extends WritableComparator { public int compare( byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { // ..... } } java
- 60. Do implement your own comparator static { WritableComparator.define(VertexKeyWritable, new VertexKeyWritable.Comparator()) } java
- 61. MultiScanTableInputFormat MultiScanTableInputFormat.setTable(conf, "graph"); MultiScanTableInputFormat.addScan(conf, new Scan()); job.setInputFormatClass( MultiScanTableInputFormat.class); java
- 62. TableMapReduceUtil TableMapReduceUtil.initTableReducerJob( "graph", MyReducer.class, job); java
- 65. Elastic MapReduce HFi les Copy to S3 Seq uen ceFiles
- 66. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles
- 67. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles
- 68. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles HFileOutputFormat.configureIncrementalLoad(job, outputTable) HFi les
- 69. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles HFileOutputFormat.configureIncrementalLoad(job, outputTable) HFi les HBase $ hadoop jar hbase-VERSION.jar completebulkload
- 70. Additional Resources Google Pregel: BSP-based graph processing system Apache Giraph: Implementation of Pregel for Hadoop MultiScanTableInputFormat example Apache Mahout - Distributed machine learning on Hadoop