In this project, as a data engineer working in a smart agriculture company using a px4 drone, we will create a data pipeline that processes the data in this ulog file. Let's briefly talk about what we will do.  We will add the ulog file to a bucket in minio, and airflow will parse this ulog file for us, convert the data in it to csv format, and then write it to another bucket. Using Nifi, we will ETL these csv files and write them to opensearch. Later, we will present the data we obtained using the dashboard offered by Opensearch to the business units. These will be needed to process the data.

• Docker - Docker compose
• Python dependency(Pyulog,Minio)
• Apache Airflow
• Apache Nifi
• Opensearch
• Prometheus
• Grafana

docker compose up --build

MIT

Dillinger is currently extended with the following plugins. Instructions on how to use them in your own application are linked below.

	Container Name	URL
	Apache Airflow	http://192.168.1.149:8080/home
	Apache Nifi	http://localhost:8081/nifi/
	MinIO	http://192.168.1.149:9001/login
	Opensearch Dashboard	http://192.168.1.149:5601/app/home
	Prometheus	http://192.168.1.149:9090/
	Grafana	http://192.168.1.149:3000/?orgId=1