Doing data science with Clojure
3 likes6,486 views
The document discusses the integration of data science practices in Clojure, emphasizing quick analytics, effective collaboration, and the use of live coding environments. It highlights methodologies for persistent and searchable data sharing, composable data structures, and the use of functions over collections for data manipulation. Additionally, it addresses the wishlist for better tools and projects relevant to improving data science workflows in Clojure.
Doing data science with Clojure
- 1. Doing data science with Clojure @sbelak [email protected]
- 5. The analytics chasm Ideal. Almost real-time, can be done during brainstorming without disrupting flow < 2min < 20min project squeeze in somewhere in the day fail roadmap ahoy!
- 6. Think in distributions, not numbers
- 8. Sharing results • Have one canonical version that is always current. • Concentrate discussion in one place and make it searchable and persistent. • Include methodology (=code).
- 15. Code hidden, but can be expanded Questions, comments, & annotations Shareable Periodically re-run to keep it fresh #alderaan #sales #growth discoverability
- 16. Wishlist/TODO • Better editor (shaunlebron.github.io/parinfer/ ?) • Embedded REPL • Better exception reporting • Browsable data structures (tried and miserably failed: org-babel)
- 17. The tools
- 19. Data frame • Data tends to be heterogeneous • Clojure excels in structure manipulation/encoding
- 20. github.com/sbelak/huri • No data structures, just functions over collections • Composable (even DSLs — no macros!) • Reasonably fast (transducers <3) • Do-what-I-mean (auto-sort, liberal with inputs, …) • Minimal buy-in • Support reaching into nested structures everywhere
- 21. composable data structure based DSLs ->> and partial friendly Support reaching into nested structures everywhere vanilla vector of maps interoperability Provide curried versions where possible
- 22. Composability is key to quick iterating • Provide curried versions where possible • ->> and partial friendly • encode computation in structure (comp, some-fn, every-pred, data structure based DSLs, …) • consistent API
- 23. Catching errors early more context easier debugging faster iterating
- 24. <3 Bret Victor
- 25. Q: What about machine learning? A: farm it out to sklearn
- 26. huri.plot • DSL on top of ggplot2 (via gg4clj) • Targets Gorilla REPL • Follows the rest of Huri’s design philosophy • bar chart, scatter plot, line chart, box & violin plot, heatmap, histogram
- 28. Wishlist/TODO • (even) better structure manipulation (via Spectre?) • Interactive plots • More transducer-compatible (online) math functions • Optimizing ->> (rewrite code on the fly to do more with transducer composition)
- 29. Projects worth keeping an eye on github.com/thi-ng/geom github.com/yieldbot/vizard zeppelin-project.org github.com/aphyr/tesser github.com/nathanmarz/specter