Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to Monitor 1,000+ Solar Farms in Real Time with Sandy May and Richard Conway
1 like590 views
The document focuses on the use of Azure Databricks for monitoring solar farms, presented by Richard Conway and Sandy May of Elastacloud. It covers data acquisition from solar and renewable sources, Azure's architecture for data processing, and the orchestration of data to optimize energy production and balance within national grids. The presentation also discusses practical implementations, including data prediction models and reporting through Power BI.
Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to Monitor 1,000+ Solar Farms in Real Time with Sandy May and Richard Conway
- 1. Sandy May & Richard Conway, Elastacloud Azure Databricks Monitoring Solar Farms #SAISDD11
- 2. About Us • Richard Conway Founder and Director of Elastacloud, a UK Cloud Data Analytics Consultancy, Azure MVP/Microsoft Regional Director + Sandy May Cloud Big Data Surgeon • Microsoft Azure Gold Partner, Cloud Platform and Data Analytics, OSS Partner of the year 2015, Microsoft Partner of the Year nominee 2018 • Co-founder of UK Azure User Group, IoT and Data Science Innovators UK, UK Cloud Infrastructure User Group • Author of data science degree academy.microsoft.com • Running AzureCraft in UK annually • Contributors to open source, several Apache projects including Storm, Spark, Libcloud and Parquet • 50+ people, offices in London, Nottingham and Spain 2#SAISDD11
- 3. What we’ll cover in this presentation • A little about solar and renewables data acquisition • A little about Azure and how we design things • Using Databricks in a fun way • Orchestrating the future 3
- 4. The Solar Farm 4 images from mnn.com
- 5. 5 Courtesy Wikipedia – installed UK capacity ~ < 10GW
- 6. Solar Irradiance 6 Inverter W/m2 String String String DC to AC National Grid
- 7. Modbus Collection of Data 7 Inverter Portal Consumer Logging Get Data TCP Port 502 Things to note: - Inverter can generally only accept a single Modbus TCP connection - As such usage of Hub and Spoke Models are needed to relay to cloud - Modbus def example: 29;1;INV_EFFICIENCY;3;2;29;;;0.1;0;%;2 Register No. Min/Max/Avg Units Msg Type Per minute Redundant Consumer TCP Port xxx
- 8. Cloud Gateway Patterns from Farm 8 Inverter Modbus Collector ServerPyranometer Weather station Generally terrible router Cloud Gateway Export Meter Event Hub Message Stream Low spec AWS Server Data Orchestration API with 24 hour lag Event Hub Message Stream Takes around 8-9 hours!
- 9. All sources of information 9 Farm weatherbit.io darksky openweather Met office cloud cover suntimes Azure
- 10. Balancing Energy 10 production time - intraday shortfall surplus - Shortfalls lead to buy back to fulfil contracts - Surpluses need selling - Grid needs to be balanced at all times - Need to understand intraday pricing market to understand exposure - Insure risk through PPAs for lower returns - National Grid has balancing cost - Internal transactions need balancing too - Need to understand interconnects
- 11. Our Azure Architecture 11 Data Bricks Event Hubs Scada API for Actual Power & Recorded Irradiance Weather API’s (MetOffice, DarkSky & WeatherBit) 5 minute weather data streamed for over 1,200 solar farms in Europe; data used to create Solar Irradiance predictions, and Power output for farms using service Actuals collected for model training and evaluation Predictions and model performance recorded in Azure SQL Data Warehouse and reported on using Power BI