Introducing MetalK8s, An Opinionated Kubernetes Implementation
0 likes684 views
Scality Metalk8s is an opinionated Kubernetes distribution designed for long-term on-premise deployments, emphasizing an out-of-the-box experience without complex user decisions. Built on Kubernetes with a focus on reliability and performance, it integrates critical services for storage and monitoring while addressing unique enterprise deployment challenges. The project aims for seamless operation across both cloud and on-prem environments, ensuring compatibility and ease of installation based on extensive experience from previous Scality solutions.
Introducing MetalK8s, An Opinionated Kubernetes Implementation
- 1. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS Nicolas Trangez - Technical Architect [email protected] @eikke
- 3. ONE PURPOSE GIVING FREEDOM & CONTROL TO PEOPLE WHO CREATE VALUE WITH DATA
- 4. 8 60+ 120+ 20+ ~10 GLOBAL CLIENT BASEGLOBAL PRESENCE 20+ OFFICES 200+ PEOPLE NATIONALITIES EUROPEAMERICAS AUSTRALIA JAPAN
- 5. OUR JOURNEY TO KUBERNETES Scality RING, S3 Connector & Zenko
- 6. Scality RING - Physical servers, some VMs - Only the OS available (incl. ‘Legacy’ like CentOS 6) - Static resource pools - Static server roles / configurations - Solution distributed as RPM packages, deployed using SaltStack - De-facto taking ownership of host, difficult to run multiple instances - Fairly static post-install On-premise Distributed Object & File Storage
- 7. Scality S3 Connector On-premise S3-compatible Object Storage - Physical servers, sometimes VMs - Static resource pools - “Microservices” architecture - Solution distributed as Docker container images, deployed using Ansible playbooks - No runtime orchestration - Log management, monitoring,... comes with solution
- 8. Scality Zenko - Deployed on-prem or ‘in the Cloud’: major paradigm shift - New challenges, new opportunities - Multi-Cloud Data Controller, must run on multiple Cloud platforms Multi-Cloud Data Controller
- 9. Scality Zenko - Embraced Docker as distribution mechanism - Some shared with Scality S3 Connector - For Cloud deployments, started with Docker Swarm - Ran into scaling, reliability and other technical issues - Decided to move to Kubernetes - Managed platforms for Cloud deployments, where available (GKE, AKS, EKS one day) - On-prem clusters Deployment Model
- 10. Scality Zenko - Homogenous deployment between in-cloud and on-prem - Various services provided by cluster: - Networking & policies - Service restart, rolling upgrades - Service log capturing & storage - Service monitoring & metering - Load-balancing - TLS termination - Flexible resource management - If needed, easily add resources to cluster by adding some (VM) nodes - HorizontalPodAutoscaler Kubernetes Benefits
- 11. OUR JOURNEY TO KUBERNETES MetalK8s
- 12. On-prem Kubernetes - Can’t expect a Kubernetes to be available, provided by Scality customer - Looked into various existing offerings, but in the ends needs to be supported by/through Scality (single offering) - Decided to roll our own
- 13. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS
- 14. OPINIONATED We offer an out-of-the-box experience, no non-trivial choices to be made by users
- 15. LONG-TERM Zenko solution is mission-critical, can’t spawn a new cluster to upgrade and use ELB (or similar) in front
- 16. ON-PREM Can’t expect anything to be available but (physical) servers with a base OS
- 17. Scality MetalK8s - “Stand on the shoulders of giants” - Scope: 5-20 physical machine, pre-provisioned by customer or partner - Built on top of the excellent Kubespray Ansible playbook - Use Kubespray to lay out a base Kubernetes cluster - Also: etcd, CNI - Add static & dynamic inventory validation pre-checks, OS tuning, OS security - Based on experience from large-scale Scality RING deployments - Augment with various services, deployed using Helm - Operations - Ingress - Cluster services - Take care of on-prem specific storage architecture
- 18. Scality MetalK8s: Cluster Services - “Stand on the shoulders of giants” - Heapster for dashboard graphs, `kubectl top`,... - metrics-server for HorizontalPodAutoscaler - Looking into k8s-prometheus-adapter - Ingress & TLS termination: nginx-ingress-controller - Cluster monitoring & alerting: Prometheus, prometheus-operator, Alertmanager, kube-prometheus, Grafana - Host-based node_exporter on all servers comprising the cluster, including etcd - Host & container logs: ElasticSearch, Curator, fluentd, Kibana - Considering switch to fluent-bit - All of the above gives a great out-of-the-box experience for operators
- 22. Scality MetalK8s: Storage - On-prem: no EBS, no GCP Persistent Disks, no Azure Storage Disk,... - Also: can’t rely on NAS (e.g. through OpenStack Cinder) to be available - Lowest common denominator: local disks in a node - PVs bound to a node, hence PVCs bound, hence Pods bound - Thanks PersistentLocalVolumes & VolumeScheduling! - Decided not to use LocalVolumeProvisioner, but static approach (for now) - Based on LVM2 Logical Volumes for flexibility - PV, VG, LVs defined in inventory, created/formatted/mounted by playbook - K8s PV objects created by playbook - May support whole partitions/drives depending on application need - Working with community on Dynamic Local Volume provisioning - Also using LVM2
- 23. Scality MetalK8s: Deployment - Based on years of years of experience deploying Scality RING at enterprise customers, service providers,... - Constraints in datacentra often very different from ‘VMs on EC2’ - No direct internet access: everything through HTTP(S) proxy, no non-HTTP traffic - Dynamic server IP assignment - Security rules requiring services to bind to specific IPs only - Fully air gapped systems: requires 100% offline installation - Non-standard OS/kernel - Integration with corporate authn/authz systems - Not all of the above supported yet, tackling one by one - Relevant patches to be upstreamed to Kubespray - Only support RHEL/CentOS family of Linux distributions - Support for Ubuntu and others can be community-driven, Kubespray supports them - RHEL/CentOS sometimes difficult targets for containers/Docker/Kubernetes
- 24. Scality MetalK8s: Ease of Deployment $ # Requirements: a Linux or OSX machine with Python and coreutils-like $ # Create inventory $ vim inventory/... $ make shell # Launches a ‘virtualenv’ with Ansible & deps, ‘kubectl’, ‘helm’ $ # Demo @ https://asciinema.org/a/9kNIpBWg4KiwjT5mNSrH0tmj9 $ ansible-playbook -i inventory -b metal-k8s.yml $ # Grab a coffee, and done
- 25. Scality MetalK8s: The road forward - Documentation: Install guides, Operations guides, Troubleshooting guides,... - Forward & backward compatibility requirements - Sizing numbers - Hardware & software compatibility testing - Security auditing & testing - Testing/CI: install, upgrade, downgrade, ‘monkey’,... - Also in very constrained environments - Delivery of fully-offline installation package - ...
- 26. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS https://zenko.io https://github.com/scality/metal-k8s @Scality | @Zenko