Exponea - Kafka and Hadoop as components of architecture
- 1. Kafka and Hadoop as components of architecture - Martin Strycek
- 2. Kafka Kafka is a distributed streaming platform.
- 3. Hadoop The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing
- 4. How Kafka and Hadoop got into Exponea?
- 5. How Kafka and Hadoop got into Exponea?
- 6. How Kafka and Hadoop got into Exponea ● We had our in memory database super fast, but in memory ● Our customers were scared that they will have to pay a lot ● They want to have freedom to run analyses on all data ● We had some troubles processing data
- 7. Kafka + MapR
- 8. Kafka + MapR ● We were appending data to files that contain jsons ○ HDFS does not support append ● We started using Kafka 0.8.2.1 ● We had no idea how to monitor the whole stack
- 9. Where we are now
- 10. How Kafka and Hadoop got into Exponea ● We are using Kafka to stream data to ● We have first Sparks jobs that are part of application ○ Recommendation ○ Predictions ○ Campaigns overview ○ Loading data to
- 11. ● We are using Oryx 2 ○ But we need multitenancy ● We have MapR ○ But it ships with different Spark version that Oryx 2 ● We are using Oryx 2 ○ But it works with different version with Kafka Recommendation
- 13. ● How about we use ● How about we create better local storage for ● We need another cluster for testing ○ Bare metal? AWS? Google Cloud? ● library to be usable in ● We want to do a workshop for all of you that want to try it out, but don’t have a place where. What next?
- 14. ● Freedom & responsibility ● Big impact ● Team ● Proficiency Exponea Culture } Global ambitions, best company to work for
- 15. Thanks!