Using Onyx in anger
2 likes1,416 views
The document discusses Onyx, a high-performance distributed computation system designed for various data processing tasks, including ETL and machine learning, implemented in Clojure. It emphasizes the principle of 'data is code' and highlights Onyx's capabilities for self-service analytics, workflow management, and automated view generation. Key takeaways include its production readiness, effective architecture for clean data processing, and the emphasis on interactive development through queryable data descriptions.
![Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]](https://image.slidesharecdn.com/usingonyxinanger-170325154002/85/Using-Onyx-in-anger-8-320.jpg)
![Catalogue
[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]
Vanilla Clojure function
(defn adder [n {:keys [x] :as segment}]
(assoc segment :x (+ n x))))
Plugins (I/O)
seq, async, Kafka,
Datomic, SQL, S3,
SQS, …
parameter
self-documenting](https://image.slidesharecdn.com/usingonyxinanger-170325154002/85/Using-Onyx-in-anger-9-320.jpg)
Using Onyx in anger
- 2. Onyxa masterless, cloud scale, fault tolerant, high performance distributed computation system … written entirely in Clojure
- 3. Onyx at • In production for almost a year • ETL • online machine learning • offline (batch) machine learning • ad-hoc analysis
- 4. Self-service infrastructure for data scientists
- 5. 1.Onyx at a glance 2.How Onyx rewired my brain 3.Putting “data is code” to work
- 6. 1.Onyx at a glance 2.How Onyx rewired my brain 3.Putting “data is code” to work Describing computation with data
- 7. Onyx at a glance
- 8. Job = [[:input :processing-1] [:input :processing-2] [:processing-1 :output-1] [:processing-2 :output-2]] [{:flow/from :input-stream :flow/to [:process-adults] :flow/predicate :my.ns/adult? :flow/doc "Emits segment if an adult.”}] workflow + flow conditions + catalogue[{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type :function :my/n 5 :onyx/params [:my/n]} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/doc "Writes segments to a core.async channel"}]
- 9. Catalogue [{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type :function :my/n 5 :onyx/params [:my/n]} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/doc "Writes segments to a core.async channel"}] Vanilla Clojure function (defn adder [n {:keys [x] :as segment}] (assoc segment :x (+ n x)))) Plugins (I/O) seq, async, Kafka, Datomic, SQL, S3, SQS, … parameter self-documenting
- 10. Computation entirely described with data data is code!
- 11. Everything can be run locally!
- 13. How Onyx rewired my brain
- 14. It’s not about scaling, but clean architecture
- 15. My goto architecture KafkaDB Events Onyx Onyx Onyx Persist all events to S3 • time travel • query with AWS Athena
- 18. Putting “data is code” to work
- 19. Interlude: queryable data descriptions with spec • s/registry, s/form • Build a graph (Datomic) Interact with your type system! code is data!
- 20. Case study: autogenerating materialised views Kafka Materialised views Events External data Automatic view generation • Event & attribute ontology • Manual (via spec) • Inferred • Statistical analysis (seasonality detection, outlier removal, …) Onyx Onyx Onyx
- 21. Automatic view generation 1. Walk spec registry 2. Apply rules 1. Define new view (spec) 2. Trigger Onyx job that creates the view ⤾
- 22. Code is data or data is code?
- 23. Takeouts
- 24. Onyx is production ready
- 25. Everything should be live and interactive
- 26. Computation graphs are a great way to structure data processing code
- 27. Queryable data and computation descriptions supercharge interactive development and are a great building block for automation