Apache Druid®: A Dance of Distributed Processes
0 likes309 views
This document summarizes the key components and collaborations in Apache Druid. It describes Zookeeper's role in coordination, the Overlord's role in task management, the Broker's role in query routing, and the Middle Manager's role in ingestion and indexing. It provides diagrams illustrating how these components work together to ingest and store distributed data, and answer queries in a scalable way.
Apache Druid®: A Dance of Distributed Processes
- 1. Apache®, Apache Druid®, Druid®, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
- 2. [email protected] 20 years in Enterprise Architecture CRM, EDRM, ERP, EIP, Digital Services, Security, BI, RI, and MDM BA Theology (!) and Computer Studies TOGAF certified Book collector & A/V buyer Prime Timeline = proper timeline #werk petermarshall.io
- 3. What the collaborations do Some principles Each collaboration Wrap-up Questions!
- 4. Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules
- 6. ● A job to do ● Compute to do the ingestion ● A place to store optimised data ● A question to answer ● Data to process! ● Compute to answer queries ● Somewhere to put the data that’s near to the query process ● Some rules to follow Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules
- 8. What the collaborations do Some principles Each collaboration Wrap-up Questions!
- 9. Zookeeper Overlord Ingest Spec Task Log S3, GCS, ABS, HFDS, HTTP, Local, MySql, Postgres, Druid Kafka Kinesis Middle Manager (Indexer)
- 11. Deep Store Zookeeper Overlord Middle Manager (Indexer) columnarise index & encode time-shardDeep Store
- 23. What the collaborations do Some principles Each collaboration Wrap-up Questions!
- 29. Zookeeper Coordinator Overlord Broker Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules
- 30. ★ A job to do ★ Compute to do the ingestion ★ A place to store optimised data ★ A question to answer ★ Data to process! ★ Compute to answer queries ★ Somewhere to put the data that’s near to the query process ★ Some rules to follow Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules
- 31. http://druid.apache.org Imply Distribution https://imply.io/get-started @druidio Add Apache Druid as a skill Apache Distribution https://github.com/apache/druid ASF Slack #druid Druid Community https://druid.apache.org/community/ Meetup Groups https://www.meetup.com/pro/apache-druid/ Google Groups Druid User Forum https://groups.google.com/
- 32. What the collaborations do Some principles Each collaboration Wrap-up Questions!