Lost In The Clouds
- 1. Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
- 3. A free map of the planet Free geospatial data
- 4. Licensed under CC-BY-SA Created from Crowd-sourced contributions
- 6. 1,000+ contributors per day Used for: Wikipedia
- 7. Flikr
- 11. Mobile devices © Constantin Litvak
- 12. Coverage CC-BY-SA 2.0 © OpenStreetMap
- 13. The OpenStreetMap Database Server In the cloud
- 14. MySQL
- 15. Ways: 24,956,171
- 17. Tags: 3.6 billion highway=motorway
- 18. name=Broadway
- 19. foo=bar CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
- 20. Interfaces API For updates (1,000 users per day)
- 21. Queries of up to 0.25 ² degrees Weekly Planet Dump 100Gbyte XML file Changes Feed of changes
- 22. Daily / Hourly / Minutely
- 23. One file per minute (~30Kb compressed) CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
- 25. Want to be able to query the data in bigger chunks and to do it more selectively
- 26. Solution
- 27. Cloud based schemaless database with simple to use query interface and lots of indexes CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
- 28. XAPI Service GT.M or Cache
- 29. 250Gbyte database Nodes: 30Gb
- 30. Node tags: 115Gb
- 32. Ways: 21Gb
- 33. Way tags: 10Gb
- 34. Way indexes: 18Gb Apache + serverLink
- 35. Licensed as AGPL CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
- 36. XAPI Service In the cloud
- 37. Queries of up to 100 ² degrees
- 38. Query by tags and tag values
- 39. Scaleable
- 40. REST style interface http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] Import planet dump
- 41. Import minute diff CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
- 42. XAPI Service Geospatial Index – quadstrings adaabcdcabaadab
- 43. XAPI Service ^way(27016525)="adaabcdcabaadab" ^way(27016525,1)=296138118 ^way(27016525,2)=296138119 ^way(27016525,3)=296138120 ^way(27016525,4)=296138121 ^way(27016525,5)=296138118 ^waytag(27016525,"addr:housenumber")=2 ^waytag(27016525,"building")="yes" ^wayx("building","*","adaabcdcabaadab",27016525)="" ^wayx("building","*","adaabcdcabaadab",27028298)="" ^wayx("building","*","adaabcdcabaadab",27028299)="" ^wayx("building","*","adaabcdcabaadab",27028326)="" ^wayx("building","*","adaabcdcabaadab",27028327)="" ^wayx("building","*","adaabcdcabaadab",27035972)="" ^wayx("building","*","adaabcdcabaadab",27035973)="" ^wayx("building","*","adaabcdcabaadab",27035974)="" ^wayx("building","*","adaabcdcabaadab",27035975)="" ^wayx("building","*","adaabcdcabaadab",27035984)="“ <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> <nd ref='296138120'/> <nd ref='296138121'/> <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> <tag k='building' v='yes'/> </way>
- 44. Deployment Currently 5 instances: London
- 45. Paris
- 46. San Diego
- 47. Zurich
- 48. Amsterdam (Wikimedia-de) More servers needed: Internet connectivity
- 49. 300Gb disk storage CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
- 50. Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
- 51. Internals Debian
- 53. REST URI represents a resource
- 54. Xpath flavoring serverLink WebLink emulator GT.M or Cache POM AGPL
- 55. XAPI Future Load Sharing / balancing
- 57. JSON output
- 59. More servers
- 60. Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap