dataviz on d3.js + elasticsearch
4 likes5,903 views
The document discusses using Elasticsearch, D3.js, Angular.js, and Google Refine to create a full stack data visualization of open data from Bordeaux, France. It focuses on data from the CAPC contemporary art museum, importing the data into Elasticsearch for scalable search and then using D3.js, Angular.js, and Yeoman to build the front-end visualization with JavaScript. The goal is to make the data more accessible and understandable through interactive visualization.
![import data in
elasticsearch
• bulk api import for es is handy
• http://www.elasticsearch.org/guide/
reference/api/bulk.html
• we use templating in google refine to
export data to the correct format
• { "index" : { "_index" : "musees", "_type" :
"capc"} } [snipp]
mardi 19 mars 13](https://image.slidesharecdn.com/prez-bordeauxjs-pdf-130319055344-phpapp01/85/dataviz-on-d3-js-elasticsearch-6-320.jpg)
dataviz on d3.js + elasticsearch
- 1. dataviz on bordeaux open data + elasticsearch + d3js + angular.js + google refine mardi 19 mars 13
- 2. goal • full stack dataviz • front js centric with angular.js and sass • d3.js -> best lib for dataviz (a bit complex) • elasticsearch scalable search engine access form js ajax client • grab open data and reformat with google refine mardi 19 mars 13
- 3. philosophy • smarter and smarter browsers will grab a huge part of the stack • scalable NoSQL solution talk REST: super easy access • you always can enrich, refine, store, model data from anysource: yes you can ! mardi 19 mars 13
- 4. go open data ! • go to http://opendata.bordeaux.fr/ • we focus ou capc contemporary museum of bordeaux • http://opendata.bordeaux.fr/content/ collections-du-capc-musee-dart- contemporain • curl https:// bdxconfigogdi.blob.core.windows.net/ converteddata/capc.csv -o capc.csv mardi 19 mars 13
- 5. google refine • data many times isn’t perfect: humans input, unsuitable model... • with google refine you can tidy data, enrich with web services call and so on... • i think you should never think you don’t have the good data for the job • be smart and be creativ: you have everything you need, all the time ! mardi 19 mars 13
- 6. import data in elasticsearch • bulk api import for es is handy • http://www.elasticsearch.org/guide/ reference/api/bulk.html • we use templating in google refine to export data to the correct format • { "index" : { "_index" : "musees", "_type" : "capc"} } [snipp] mardi 19 mars 13
- 7. elasticsearch • scalable search engine • adding more power == adding more nodes • sharding • replication • fault tolerant mardi 19 mars 13
- 8. elasticsearch • store unstructured document (json) in indexes (NoSQL way) • talk REST (api) • advanced query langage • multiple analyzers (tokens, languages, etc...) • blazing fast ! • no alternativ solutions (to my advice) (and kimshy advice too ;) ) mardi 19 mars 13
- 9. yeoman • perfect tool for the new browser focus stack coming soon • yeoman init angular • yeoman init angular:route capc • yeoman install d3 • yeoman install jquery • yeoman server -> yeah !!! mardi 19 mars 13
- 10. angular.js • mvc framework for browser (js) • by google • rely better on html doc • cleaner controller handling than backbone (to my advice) mardi 19 mars 13
- 11. d3.js • best javascript lib for dataviz (to my advice) • slow learning curve • based on svg browser markup • data and dom oriented • generic (go low level with svg easily) mardi 19 mars 13
- 12. dataviz • make appears informations from pure data • you should explore, analyse and be creativ to grab the most value possible from the data: go as far as you can • tables (excell way) -> forgive • visualization will reveal your data: data is made to be know to take decision, manage, understand mardi 19 mars 13
- 13. open data • today, wide range of datas come to the public domain • again, without computing and vizualisation, data has a poor value • open data without data scientists and data visualization has no futur mardi 19 mars 13
- 14. futur • with low bandwith, storage and computing cost, you can grab open data from a lot of sources • you can put them in big data store and make analysis, relations between, with open source technologies • you can then share this to the word with data visualization on your website, blogs... mardi 19 mars 13 • amazing isn’t it ?? !!!!!
- 15. questions ? infos ? • https://github.com/mathieue • http://www.mathieu-elie.net/ • https://twitter.com/mathieuel • https://www.facebook.com/mathieuelietech mardi 19 mars 13