Building IoT infrastructure on edge with .net, Raspberry PI and ESP32 to connect to Azure
Download as PPTX, PDF0 likes395 views
The document discusses building an IoT infrastructure on the edge with .NET that connects devices like Raspberry Pis and ESP32s to Azure. It describes setting up a network of Raspberry Pi devices running .NET Core and connecting sensors to collect data and send events to an Apache Kafka cluster. The events are then aggregated using Apache Spark on another Raspberry Pi and the results routed to the cloud. Issues encountered include Kafka's Java dependencies, Spark's complex processing model, and lack of documentation around integrating Pi, Kafka and Spark. While the technologies work individually, configuring and integrating them presented challenges at the edge.
Building IoT infrastructure on edge with .net, Raspberry PI and ESP32 to connect to Azure
- 1. After Hour 2021 Internet of Things Building an IoT infrastructure on Edge with .NET that talks with Azure on Raspberry PI, ESP32, Mini PC Marco Parenzan
- 2. After Hour 2021 Internet of Things Marco Parenzan Senior Solution Architect in beanTech Microsoft Azure MVP Community Lead 1nn0va // Pordenone 1nn0va After Hour (ogni martedì sera alle 21) https://www.youtube.com/c/Community1nn0va Linkedin: https://www.linkedin.com/in/marcoparenzan/
- 3. After Hour 2021 Internet of Things Marco Parenzan marco_parenzan marcoparenzan marcoparenzan Marco Parenzan
- 4. After Hour 2021 Internet of Things
- 5. After Hour 2021 Internet of Things
- 6. After Hour 2021 Internet of Things “I’m not a data scientist! Or a BI Analyst!
- 7. After Hour 2021 Internet of Things “But I’m a .NET developer
- 8. After Hour 2021 Internet of Things .NET for data(science) • .NET Spark • .NET Interactive (notebooks) • Azure Synapse Analytics • ML.NET… • All good for IoT scenarios!
- 9. After Hour 2021 Internet of Things
- 10. After Hour 2021 Internet of Things “I’m not a field engineer
- 11. After Hour 2021 Internet of Things https://opcfoundation.github.io/UA-.NETStandard/
- 12. After Hour 2021 Internet of Things Real industrial edge/GW with PI
- 13. After Hour 2021 Internet of Things More ARM
- 14. After Hour 2021 Internet of Things .NET for ARM/edge • .NET (Core) • Jeff Geerling (You Tube)
- 15. After Hour 2021 Internet of Things Sono contento perchè... ...dalla sessione di... ...ho preso... Marco Dal Pino Devices on edge Mirco Vanini .NET e Raspberry PI Andrea Tosato Message broker Marco Minerva Minimal API Riccardo Zamana «Distributed edge»
- 16. After Hour 2021 Internet of Things ““Questo matrimonio non s’ha da fare…”
- 17. After Hour 2021 Internet of Things “ Sì….può….fareeeee!
- 18. After Hour 2021 Internet of Things Edge Scenario • IoT Edge…quale è lo scenario equivalente OpenSource? • Non voglio usare i container…voglio sfruttare una rete di rpi
- 19. After Hour 2021 Internet of Things Apache Kafka and Apache Spark Gateway (s)
- 20. After Hour 2021 Internet of Things
- 21. After Hour 2021 Internet of Things TP-LINK MR6400 4G LENTO!!!!!!!
- 22. After Hour 2021 Internet of Things Edge00 // PI400/4 PiOS Edge06 // Atom Z8350/2 Win10/64 Edge03 // Pi3B/1 PiOS/32 Edge02 // Pi3B/1 PiOS/32 Edge01 // Pi4B/4 PiOS/32
- 23. After Hour 2021 Internet of Things VNET The complete vision Edge00 PI400/4 Edge01 PI4/4 Kafka Edge02 PI3B+/1 Spark Edge99 IoTHub Nano00 ESP32WROOM Device//GPIO TP-LINK MR6400 4G Edge03 PI3B+/1 Broker) VPN Private Azure Edge 192.168.2.100 192.168.2.101 192.168.2.102 192.168.2.103 Edge06 Atom Z8350/2 AI NOT YET TO BE CHANGED
- 24. After Hour 2021 Internet of Things Enabled SSH and VNC • DHCP with MAC reservation • Issues: • networking: PI does not broadcast its name out of the box(?!?!?!) • Samba/NFS/… • Reaching (issues with Windows 10)Router!
- 25. After Hour 2021 Internet of Things Install .NET 6.0 on Raspberry PI wget https://download.visualstudio.microsoft.com/download/pr/1f85b038- 9917-4d0a-8485-5dc86510eec7/a7555924fe292c6c2140893f066abe65/dotnet- sdk-6.0.100-linux-arm.tar.gz sudo mkdir -p $HOME/.dotnet sudo tar zxf dotnet-sdk-6.0.100-linux-arm.tar.gz -C $HOME/.dotnet echo 'export DOTNET_ROOT=$HOME/.dotnet' >> ~/.bashrc echo 'export PATH=$PATH:$HOME/.dotnet' >> ~/.bashrc source ~/.bashrc • On edge00 and edge03 • edge02 has 3.1 for .net spark • And JDK • Edge01only JDK https://docs.microsoft.com/en-us/dotnet/iot/deployment
- 26. After Hour 2021 Internet of Things Visual Studio Code • It works! • And install VSCode (https://code.visualstudio.com/docs/setup/raspberry-pi) • sudo apt update • sudo apt install code • But...
- 27. After Hour 2021 Internet of Things Ubuntu o RaspiOS? • Ho provato Ubuntu 20 su PI400/4 64bit • Lento! • Sono tornato a RaspiOS/32 • Facile abilitare VNC e SSH
- 28. After Hour 2021 Internet of Things No Containers? • Not using containers • More simple manipulation of config files • Images not ready for ARM • Speed…not ready to wait more…
- 29. After Hour 2021 Internet of Things The scenario • WordCount as a Sample (WCaaS ) • Wordsignal, command, event (vote?) • 19 can become «event1»..»event9» • In device terms: a button on digital input for each «word»
- 30. After Hour 2021 Internet of Things Edge00: PI400 • Produce events • Send to Kafka • Issues: • Kafka written in Java • But std .NET package uses native libkafka (C++) from Confluent that is available only for x86 • Require rebuilding (not found a ready rebuild) • Used a native implementation • not maintained • Not clean working • MQTT listener on kafka does not exist (only plugins controlled by kafka)
- 31. After Hour 2021 Internet of Things Edge01: PI4/4 Kafka • Collect events events with Kafka • Install OpenJDK11 • Issues • Consumes lot of memory (required PI4) • Issues with configuration (listeners, also on WSL) • Complicated configuration (SASL-SCRAM, certificate store)Industrial scenario (tampering)
- 32. After Hour 2021 Internet of Things Edge02: PI3 Spark • Aggregating events • Install OpenJDK8 for 2.4.8 (not 11) • Issues • Issues(?) with 3.x (not really tested) • Alignment between .net spark with spark • Problems with watermarking that is fundamental on streaming
- 33. After Hour 2021 Internet of Things Edge03: PI3 • Handle processed data and route to the cloud • Planned installing an MQTT broker (Mosquitto?) • To implement real MQTT routing • Installed .NET 6 • Monitor files on a shared folder on edge02 • sudo mkdir /mnt/data • sudo apt-get install sshfs • sudo sshfs -o allow_other,default_permissions [email protected]:/home/pi/data /mnt/data • Issues • <empty>? • Broker not tested
- 34. After Hour 2021 Internet of Things DEMO
- 35. After Hour 2021 Internet of Things .NET NanoFramework • http://nanoframework.net/ • Open Source, from .net microframework • no AOT compiling • Interpreter • firmware containing IL code • Promising, not complete • No async/await and Tasks • API • All rewritten • Focused on devices, not Azure
- 36. After Hour 2021 Internet of Things ESP32-WROOM32 (1x10€, 3x23€)
- 37. After Hour 2021 Internet of Things ESP32 CAM – Waiting for driver (few weeks)
- 38. After Hour 2021 Internet of Things DEMO
- 39. After Hour 2021 Internet of Things
- 40. After Hour 2021 Internet of Things Minus/Issues • PI3 is slowOnly PI4+ • PI4 is slow for desktop • Kafka is difficult to configure • Spark has a complex processing/execution model • In general documentation is sparse and not homogeneus (PI/Kafka/Spark)
- 41. After Hour 2021 Internet of Things Plus • .NET 6 works fine • Works everywhere • Also Java works fine Kafka and Spark • PI4 is powerful • It can be a potential platform for Kafka and Spark on edge
- 42. After Hour 2021 Internet of Things Some links • https://github.com/dotnet/spark • https://docs.microsoft.com/en-us/dotnet/spark/tutorials/get- started?tabs=linux • https://code.visualstudio.com/docs/setup/raspberry-pi • https://dotnet.microsoft.com/download/dotnet/thank-you/sdk-3.1.413- linux-arm32-binaries • https://github.com/dotnet/spark/releases • https://docs.microsoft.com/en-us/dotnet/iot/deployment • https://mvnrepository.com/artifact/org.apache.spark/spark- streaming_2.12/3.0.1
- 43. After Hour 2021 Internet of Things GRAZIE!
- 44. After Hour 2021 Internet of Things https://www.youtube.com/c/Community1nn0va