Running Kafka and Spark on Raspberry PI with Azure and some .net magic
Download as PPTX, PDF0 likes189 views
The document discusses the setup and challenges of running Apache Kafka and Spark on Raspberry Pi devices using .NET and Azure, highlighting various edge scenarios and configurations. It details the installation process of .NET 6 and related tools, while addressing issues with performance, compatibility, and documentation. The conclusion emphasizes the potential of Raspberry Pi 4 as a platform for edge computing applications despite several limitations.
Running Kafka and Spark on Raspberry PI with Azure and some .net magic
- 1. 1 TOPIC Running Kafka and Spark on Raspberry PI with Azure and some .NET magic Marco Parenzan
- 2. 2 Thanks to our partners
- 3. Senior Solution Architect in beanTech Microsoft Azure MVP Community Lead 1nn0va // Pordenone 1nn0va After Hour (ogni martedì sera alle 21) https://www.youtube.com/c/Community1nn0va Linkedin: https://www.linkedin.com/in/marcoparenzan/ Marco Parenzan
- 6. 6 .NET for data(science) • .NET Spark • .NET Interactive (notebooks) • Azure Synapse Analytics • ML.NET… All good for IoT scenarios!
- 7. 7 .NET for ARM/edge .NET (Core) Jeff Geerling (You Tube)
- 8. 8 More ARM
- 9. ““Questo matrimonio non s’ha da fare…”
- 11. 11 Edge Scenario IoT Edge…quale è lo scenario equivalente OpenSource? Non voglio usare i container…voglio sfruttare una rete di rpi
- 12. 12 Apache Kafka and Apache Spark
- 13. 13 VNET The complete vision Edge00 PI3B+ Device(GPIO) Edge01 PI400/4 Kafka Edge02 PI4/4 Spark Edge99 IoTHub Nano00 ESP32WROOM Device//GPIO TP-LINK MR6400 4G Edge03 PI3B+ (MQTT Broker) VPN Private Network? Azure Edge 192.168.2.100 192.168.2.101 192.168.2.102 192.168.2.103
- 14. 14
- 15. 15
- 16. 16
- 17. 17 Enabled SSH and VNC DHCP with MAC reservation Issues: - networking: PI does not broadcast its name out of the box(?!?!?!) - Samba/NFS/… - Reaching (issues with Windows 10)
- 18. 18 Install .NET 6.0 on Raspberry PI Download https://dotnet.microsoft.com/download/dotnet/thank- you/sdk-6.0.100-rc.1-linux-arm32-binaries sudo mkdir -p $HOME/.dotnet sudo tar zxf dotnet-sdk-3.1.100-linux-arm.tar.gz -C $HOME/.dotnet echo 'export DOTNET_ROOT=$HOME/.dotnet' >> ~/.bashrc echo 'export PATH=$PATH:$HOME/.dotnet' >> ~/.bashrc source ~/.bashrc • On edge00 and edge03 (edge02 has 3.1 for .net spark) https://docs.microsoft.com/en-us/dotnet/iot/deployment
- 19. 19 Visual Studio Code It works! • And install VSCode (https://code.visualstudio.com/docs/setup/raspberry-pi) • sudo apt update • sudo apt install code • But...
- 20. 20 Minimal Visual Studio 2019 for NanoFramework in the 2022 era
- 21. 21 No Containers? • Not using containers • More simple manipulation of config files • Images not ready for ARM
- 22. 22 The scenario WordCount as a Sample (WCaaS ) Wordsignal, command, event (vote?) 19 can become «event1»..»event9» In device terms: a button on digital input for each «word»
- 23. 23 Edge00: PI3B+ device gw Produce events Send to Kafka Issues: • Kafka written in Java • But std .NET package uses native libkafka (C++) from Confluent that is available only for x86 • Require rebuilding (not found a ready rebuild) • Used a native implementation (not maintained) • Not tested on ARM • Not clean working • MQTT listener on kafka does not exist (only plugins controlled by kafka)
- 24. 24 Edge01: PI400 Kafka Collect events events with Kafka Install OpenJDK11 Issues • Consumes lot of memory (required PI4) • Issues with configuration (listeners, also on WSL) • Complicated configuration (SASL-SCRAM, certificate store)Industrial scenario (tampering)
- 25. 25 Edge02: PI4 Spark Works on PI3 (even if too slow!) Aggregating events Install OpenJDK8 for 2.4.8 (not 11) Issues • Issues(?) with 3.x (not really tested) • Alignment between .net spark with spark • Problems with watermarking that is fundamental on streaming
- 26. 26 Edge03: • Handle processed data and route to the cloud • Planned installing an MQTT broker (Mosquitto?) • Installed .NET 6 • Monitor files on a shared folder on edge02 • sudo mkdir /mnt/data • sudo apt-get install sshfs • sudo sshfs -o allow_other,default_permissions [email protected]:/home/pi/data /mnt/data • Issues • <empty>? • Broker not tested
- 28. 28
- 29. 29 Minus/Issues • PI3 is slowOnly PI4+ • PI4 is slow for desktop • Kafka is difficult to configure • Spark has a complicated processing/execution model • In general documentation is sparse and not homogeneus (PI/Kafka/Spark)
- 30. 30 Plus • .NET 6 works fine • Also Java works fine Kafka and Spark • PI4 is powerful • It can be a potential platform for Kafka and Spark on edge
- 32. Thank you Any questions? marcoparenzan marco_parenzan marcoparenzan https://github.com/marcoparenzan/kafka-spark-on-pi-with-dotnet