Kubernetes and real-time analytics - how to connect these two worlds with Apache Flink? - Albert Lewandowski, GetInData
0 likes114 views
The document discusses the integration of Kubernetes and real-time analytics through Apache Flink, highlighting concepts such as Kubernetes operators, custom resource definitions, and Flink's unique savepoints. It includes use cases, the deployment process on Kubernetes, and emphasizes the importance of observability through tools like Prometheus. Additionally, it provides practical insights for setting up a local environment and monitoring performance, along with opportunities for collaboration in the field.
Kubernetes and real-time analytics - how to connect these two worlds with Apache Flink? - Albert Lewandowski, GetInData
- 1. Kubernetes and real-time analytics How to connect these two worlds with Apache Flink? Author: Albert Lewandowski
- 2. © Copyright. All rights reserved. Not to be reproduced without prior written consent. About me ● Big Data DevOps Engineer - GetInData ● Focused on infrastructure, cloud, Big Data, AI, scalable web applications ● Certified Google Cloud Architect ● Certified Kubernetes Administrator
- 3. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Content ● Principles in the Big Data world on Kubernetes ● Why real-time data streaming? ● Different faces of Apache Flink. ● Flink and Kubernetes - real life scenarios. ● Observability of the platform. ● Quick start on your computer.
- 5. © Copyright. All rights reserved. Not to be reproduced without prior written consent. What is Kubernetes? Open-source platform for managing containerized workloads and services
- 6. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Kubernetes - Operators Method of deploying and managing app Automated provisioning of resources One setup for multiple environments Examples: pulsar-operator, postgres-operator, prometheus-operator
- 7. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Kubernetes - Custom Resource Definitions Defining custom APIs as add-ons Dynamic registration with Kubernetes API CRDs can be accessed with kubectl A CRD represents the desired state and an operator makes it happen.
- 8. © Copyright. All rights reserved. Not to be reproduced without prior written consent. What is Apache Flink? Flink is an open-source stream processing framework that supports both batch processing and data streaming programs Flink’s Savepoints are different from Checkpoints in a similar way that backups are different from recovery logs in traditional database systems. A savepoint is a consistent image of the execution state of a streaming job State of the Flink job
- 9. © Copyright. All rights reserved. Not to be reproduced without prior written consent. What is Apache Flink? Job Diagram State of Flink job Diagram
- 10. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Apache Flink Roadmap Source: Roadmap - Apache Flink
- 11. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Perception Business logic CI/CD Idempotency Reprocessing Explainability Monitoring Testing Serving Infrastructure Data Ingestion Security
- 12. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Reality Business logic CI/CD Idempotency Reprocessing Explainability Monitoring Testing Serving Infrastructure Data Ingestion Security
- 14. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Data Streaming vs. Batch Events 1 2 3 4 5 6 Batch Stream
- 15. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Use cases User activity Fraud detection Logistics Industrial IoT Location data Recommendations
- 16. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Use case - Kcell - Telecom company in Kazakhstan 500K Events / second 10M Subscribers 165K Events / s 10M Subscribers 22.5 TB / month 2020 2018 40 TB / month
- 17. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Use case - Kcell SMS events Voice usage events Data usage events Roaming events Location events Input Process Actions
- 18. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Use case - Kcell - some scenarios for Flink Balance Top Up Case If subscriber top-ups her balance too often in short period of time. We can offer her a less expensive tariff or auto-payment services. Fraud case in roaming Send an email to the anti-fraud unit if subscriber registered in roaming but his balance at the moment is equal to 0. This situation is impossible in standard case. Automatic SIM card activation Send an email to the anti-fraud unit if subscriber registered in roaming but his balance at the moment is equal to 0. This situation is impossible in standard case. Dealer Motivation Case Trigger bonus for a dealer when we discover that purchase happened attributable to him/her.
- 19. Apache Flink One tool, multiple versions
- 20. © Copyright. All rights reserved. Not to be reproduced without prior written consent. One tool, multiple languages Java 8 or 11 Python SQL Scala 2.11 or 2.12
- 21. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Where should I install? Standalone Kubernetes YARN cluster ● CICD process ● Service Discovery - monitoring with Prometheus ● Scalability ● Managing resources ● A/B Testing
- 22. © Copyright. All rights reserved. Not to be reproduced without prior written consent. High Availability of Flink JobManager level Storage level ● ZooKeeper ● Kubernetes (beta) ● High Availability of storage to/from which Flink writes/reads savepoints and checkpoints ● Performance of storage Job Strategy ● Data reprocessing policy ● How to deploy new job?
- 23. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Flink K8S Operator Kubernetes Operator for Apache Flink Ververica Platform Native Kubernetes - Apache Flink CRDs Yes Yes No No CICD Kubernetes API Kubernetes API REST API or Web UI Kubernetes API Installation Helm chart or raw Kubernetes manifests Helm chart or raw Kubernetes manifests Helm chart or raw Kubernetes manifests No need to install any component SQL Editor No No Yes No Dependencies No No Persistence volume for database Object storage for artifactory No Status beta beta production beta
- 24. Flink + Kubernetes = ? Overview in the article here
- 25. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Why Flink on Kubernetes? Simpler deployment process Flexible jobs management Simple Service Discovery - Prometheus Flexible testing
- 26. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Installation & Configuration Helm A package manager for Kubernetes CICD tool Example: Gitlab CI Kubernetes API Flink jobs
- 27. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Testing Incubating Mode A/B Testing Blue Green Deployment Production data Production job Job Incubating mode Separated output Standard output Dedicated TaskManagers Dedicated TaskManagers savepoi nt Proxy Flink Job #1 Flink Job #2 Result #1 Result #2
- 28. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Deployment process Versioning images Unit & Integration tests Git Flow Deployment process Monitoring
- 29. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Kubernetes aspects Dedicated namespaces Secured access to Flink (RBAC) Configuration files Storage for savepoints&checkpoints Resources Secrets Network performance
- 30. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Self-healing and autoscaling Scale based on metrics Flink restarts External tool for fixing Re-create cluster Automate manual tasks
- 31. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Job Cluster & Session Cluster Job Cluster Session Cluster Full set of Flink cluster for each individual job Standalone Flink cluster on Kubernetes Short running tasks Ad-hoc queries Long running tasks Separate images for different jobs
- 32. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Stories from production ● Automate in the beginning ● CICD pipeline is a must ● Verify JVM metrics ● Test different Flink configurations to get the best performance and no restarts ● Secure access to Flink jobs ● Get logs from Flink TMs and JMs
- 33. Local Setup
- 34. © Copyright. All rights reserved. Not to be reproduced without prior written consent. How to start locally? ● Minikube / Docker Desktop or any different local K8s env ● Ververica Platform ● Locally started Kafka cluster or use a Datagen APACHE FLINK KUBERNETES STREAMING SQL
- 36. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Observability Observability is about measuring how well internal states of the system can be inferred from knowledge of its external outputs (according to the control theory).
- 37. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Part One: Metrics Get metrics from environment and application - but how?
- 38. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Prometheus - Kubernetes-native solution open-source systems monitoring and alerting toolkit joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes a lot of exporters you can write your own easily mature ecosystem PushGateway, Blackbox, AlertManager, etc.
- 39. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Prometheus - simple or complex High Availability? Simple Complex Example solutions: Cortex (above), Thanos, M3DB
- 40. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Pull vs. push-based monitoring Pull Push Collector takes metrics Agents push metrics Workload on central poller increases with the number of devices polled. Polling task fully distributed among agents, resulting in linear scalability. Polling protocol can potentially open up system to remote access and denial of service attacks. Push agents are inherently secure against remote attacks since they do not listen for network connections. Flexible: poller can ask for any metric at any time. Relatively inflexible: pre-determined, fixed set of measurements are periodically exported.
- 41. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Prometheus - Stories service discovery simple on k8s limited security archived data how old data is required? monitor monitoring
- 42. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Part Two: Logs analytics 1. Get logs from app or environment. 2. Save logs. 3. Query them. 4. Make your system self-healing and discover what’s happening inside your platform.
- 43. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Logs analytics - which tool should I choose? Logs Analytics for Developers Logs Analytics for Business Loki ElasticSearch
- 44. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ELK vs. Loki ELK Loki + Promtail/Fluentd Indexing Keys and content of each key Only labels Query language Query DSL or Lucene QL LogQL Tool for data visualisation Kibana Grafana Query performances Faster due to indexed all the data Slower due to indexing only labels Resource requirements Higher due to the need of indexing Lower due to index only labels
- 45. © Copyright. All rights reserved. Not to be reproduced without prior written consent. What about alerts? Alerts signify that a human needs to take action immediately in response to something that is either happening or about to happen, in order to improve the situation.
- 46. Quick start
- 47. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Flink - Complex Event Processing Article. Codebase for example.
- 48. © Copyright. All rights reserved. Not to be reproduced without prior written consent. DevOps best practises Article.
- 49. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Kubernetes - first setup ● Minikube ● Kind ● Use Kubernetes service from public cloud provider like AWS, GCP, Azure during free tier
- 50. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Kubernetes + Flink - Operator $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/crd.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/namespace.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/role.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/role-binding.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/flinkk8soperator.yaml Verify if it works: $ kubectl -n flink-operator get po Run the example job: $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/examples/wordcount/flink-operator-custom-resou rce.yaml Verify if it is running and its status: $ kubectl get flinkapplication.flink.k8s.io -n flink-operator wordcount-operator-example -o yaml Requirements: Kubernetes cluster, kubectl
- 51. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Kubernetes + Ververica Platform Install Ververica Platform locally with Helm $ helm repo add ververica https://charts.ververica.com $ helm install vvp ververica/ververica-platform $ helm install vvp ververica/ververica-platform --set acceptCommunityEditionLicense=true Verify if Ververica is up $ kubectl get po Access the web user interface and REST API $ kubectl port-forward service/vvp-ververica-platform 8080:80 Do you want to test Flink SQL feature? Use Flink Faker (a data generator source connector) https://github.com/knaufk/flink-faker/ It requires changing used image for vvp-gateway. Requirements: Kubernetes cluster, kubectl, Helm
- 52. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Join Us! Data Engineer Spark, Kafka, Airflow, public cloud Link Backend Engineer Java / Scala, microservices Link MLOps Engineer MLOps tools, Python, public cloud Link DevOps / SRE GCP, Terraform, Prometheus Link
- 53. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Q&A
- 54. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Contact details [email protected] LinkedIn: https://www.linkedin.com/in/albert-lewandowski
- 55. Thank you for your attention!