Kubernetes + Python = ❤ - Cloud Native Prague
5 likes3,744 views
The document discusses the integration of Python with Kubernetes at Zalando, highlighting the company's extensive use of Python and machine learning. It covers various implementation details, including running Python applications on Kubernetes, deployment practices, testing methodologies, and the use of custom Kubernetes operators. Additionally, it presents resources and tools for developers, emphasizing Zalando's contributions to open-source Kubernetes projects.
![12
IS IT FAST ENOUGH?
echo 'GET http://localhost:8080' | vegeta attack -rate 200 | vegeta report
Requests [total, rate] 3300, 200.06
Duration [total, attack, wait] 16.498s, 16.494, 3.261ms
Latencies [mean, 50, 95, 99, max] 3.185ms, 3.229ms,
4.182ms, 4.791ms,
21.153ms
Bytes In [total, mean] 39600, 12.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:3300
local run on my laptop, single Pod in kind cluster](https://image.slidesharecdn.com/2019-09-05kubernetespythonlove-cloudnativeprague-190906070430/85/Kubernetes-Python-Cloud-Native-Prague-12-320.jpg)
![25
INTERACTIVE CONSOLE
$ pip3 install pykube-ng
$ python3 -m pykube
Pykube v19.9.1, loaded "/home/hjacobs/.kube/config" with context "mycluster".
Example commands:
[d.name for d in Deployment.objects(api)] # get names of deployments in default namespace
list(DaemonSet.objects(api, namespace='kube-system')) # list daemonsets in "kube-system"
Pod.objects(api).get_by_name('mypod').labels # labels of pod "mypod"
Use Ctrl-D to exit
>>>](https://image.slidesharecdn.com/2019-09-05kubernetespythonlove-cloudnativeprague-190906070430/85/Kubernetes-Python-Cloud-Native-Prague-25-320.jpg)
![38
RUNNING FROM DEVELOPMENT ENVIRONMENT
github.com/zalando-incubator/kopf
And here we are! Creating: {'duration': '1m', 'field': 'value',
'items': ['item1', 'item2']}
[2019-02-25 14:06:54,742] kopf.reactor.handlin [INFO ]
[asf-preprocessing/kopf-example-1] Handler create_fn succeeded.
[2019-02-25 14:06:54,856] kopf.reactor.handlin [INFO ]
[asf-preprocessing/kopf-example-1] All handlers succeeded for
creation.
Name: kopf-example-1
...
Status:
create_fn:
Message: hello world
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Success 81s kopf Handler create_fn succeeded.
$ kopf run scripts.py [--verbose]
$ kubectl apply -f ../obj.yaml
$ kubectl describe -f ../obj.yaml](https://image.slidesharecdn.com/2019-09-05kubernetespythonlove-cloudnativeprague-190906070430/85/Kubernetes-Python-Cloud-Native-Prague-38-320.jpg)
![61
LABEL & CUSTOM COLUMNS
labelcols=application
customcols=CPU+Requests=join(', ', spec.containers[*].resources.requests.cpu)
uses JMESPath expression](https://image.slidesharecdn.com/2019-09-05kubernetespythonlove-cloudnativeprague-190906070430/85/Kubernetes-Python-Cloud-Native-Prague-61-320.jpg)
Kubernetes + Python = ❤ - Cloud Native Prague
- 1. Kubernetes + Python = CLOUD NATIVE PRAGUE 2019-09-05 HENNING JACOBS @try_except_
- 2. 2 ZALANDO AT A GLANCE ~ 5.4billion EUR revenue 2018 > 250 million visits per month > 15.000 employees in Europe > 79% of visits via mobile devices > 26 million active customers > 300.000 product choices ~ 2.000 brands 17 countries
- 4. 4 PYTHON • Very popular • Easy to learn • Huge community
- 5. 5 PYTHON
- 6. 6 PYTHON IN ZALANDO 61 (Go) vs 257 (Python) profiles 14k (Go) vs 52k (Python) internal GitHub hits Machine Learning / Data Science
- 7. 7 RUNNING PYTHON ON KUBERNETES
- 8. 8 RUNNING PYTHON ON KUBERNETES ⇒ ~114 MiB Docker image
- 9. 9 DEPLOYMENT MANIFEST Security "stuff" Never without a readinessProbe
- 10. 10 HELLO, WORLD kubectl apply -f deployment.yaml kubectl port-forward service/aiohttp-helloworld 8080:80 http localhost:8080 HTTPie
- 11. 11 HOW TO USE KUBERNETES 1. run "Hello World" 2. configure the rest of your app
- 12. 12 IS IT FAST ENOUGH? echo 'GET http://localhost:8080' | vegeta attack -rate 200 | vegeta report Requests [total, rate] 3300, 200.06 Duration [total, attack, wait] 16.498s, 16.494, 3.261ms Latencies [mean, 50, 95, 99, max] 3.185ms, 3.229ms, 4.182ms, 4.791ms, 21.153ms Bytes In [total, mean] 39600, 12.00 Bytes Out [total, mean] 0, 0.00 Success [ratio] 100.00% Status Codes [code:count] 200:3300 local run on my laptop, single Pod in kind cluster
- 13. 13 RUNNING PYTHON ON KUBERNETES
- 14. 14 "GRACEFUL" TERMINATION IN KUBERNETES 1. kubectl delete pod 2. SIGTERM + remove endpoints 3. grace period (default: 30s) 4. SIGKILL ⇒ Going down on SIGTERM leads to request errors kube-proxy/ iptables, Load Balancer update
- 15. 15 TESTING THE ROLLING UPDATE tests/e2e/test_update.py::test_rolling_update_no_signal_handling Elapsed: 41.60 seconds Successes: 2056 Errors: 0 Requests/s: 49.42 rps tests/e2e/test_update.py::test_rolling_update_with_signal_handling Elapsed: 15.16 seconds Successes: 816 Errors: 2 Requests/s: 53.95 rps
- 16. 16 ALTERNATIVE: THE PRESTOP TRICK ...
- 17. 17 TESTING WITH PYTEST AND KIND
- 18. 18 RUNNING PYTHON ON KUBERNETES
- 19. 19 TESTING WITH PYTEST AND KIND https://pypi.org/project/pytest-kind/
- 20. 20 USING KIND CLUSTER DIRECTLY https://pypi.org/project/pytest-kind/
- 21. 21 ACCESSING KUBERNETES FROM PYTHON
- 22. 22 ACCESSING KUBERNETES FROM PYTHON • Official Kubernetes client - github.com/kubernetes-client/python • Pykube-ng - github.com/hjacobs/pykube du -csh /../../lib/python3.7/site-packages/kubernetes/ 23M total du -csh /../../lib/python3.7/site-packages/pykube/ 160K total
- 24. 24 OFFICIAL CLIENT VS PYKUBE-NG • "Official" • Generated (Swagger Codegen) • Less "Pythonic" • urllib3 • Supports many auth methods • Heavy (23 MiB) • Python 2.7 and 3 • Kel project 2015 • Dynamic • More "Pythonic" • requests • Limited auth methods • Light (160 KiB) • Only Python 3
- 25. 25 INTERACTIVE CONSOLE $ pip3 install pykube-ng $ python3 -m pykube Pykube v19.9.1, loaded "/home/hjacobs/.kube/config" with context "mycluster". Example commands: [d.name for d in Deployment.objects(api)] # get names of deployments in default namespace list(DaemonSet.objects(api, namespace='kube-system')) # list daemonsets in "kube-system" Pod.objects(api).get_by_name('mypod').labels # labels of pod "mypod" Use Ctrl-D to exit >>>
- 26. 26 WRITING A SIMPLE CONTROLLER
- 28. 28 DOWNSCALING DURING OFF-HOURS DEFAULT_UPTIME="Mon-Fri 07:30-20:30 CET" annotations: downscaler/exclude: "true" github.com/hjacobs/kube-downscaler
- 29. 29 HOUSEKEEPING ● Delete prototypes after X days ● Clean up temporary deployments ● Remove resources without owner
- 30. 30 KUBERNETES JANITOR ● TTL and expiry date annotations, e.g. ○ set time-to-live for your test deployment ● Custom rules, e.g. ○ delete everything without "app" label after 7 days github.com/hjacobs/kube-janitor
- 31. 31 JANITOR TTL ANNOTATION # let's try out nginx, but only for 1 hour kubectl run nginx --image=nginx kubectl annotate deploy nginx janitor/ttl=1h github.com/hjacobs/kube-janitor
- 32. 32 EXTENDING KUBERNETES WITH OPERATORS
- 36. 36 KOPF: SIMPLE SPY-HANDLERS React on events from K8s API. Raw payload, no interpretation. Fire-and-forget, ignore errors. github.com/zalando-incubator/kopf
- 37. 37 KOPF: CONVENIENT CAUSE AND DIFF HANDLERS github.com/zalando-incubator/kopf
- 38. 38 RUNNING FROM DEVELOPMENT ENVIRONMENT github.com/zalando-incubator/kopf And here we are! Creating: {'duration': '1m', 'field': 'value', 'items': ['item1', 'item2']} [2019-02-25 14:06:54,742] kopf.reactor.handlin [INFO ] [asf-preprocessing/kopf-example-1] Handler create_fn succeeded. [2019-02-25 14:06:54,856] kopf.reactor.handlin [INFO ] [asf-preprocessing/kopf-example-1] All handlers succeeded for creation. Name: kopf-example-1 ... Status: create_fn: Message: hello world Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Success 81s kopf Handler create_fn succeeded. $ kopf run scripts.py [--verbose] $ kubectl apply -f ../obj.yaml $ kubectl describe -f ../obj.yaml
- 39. 39 "HELLO WORLD" EXAMPLE: CRD github.com/zalando-incubator/kopfgithub.com/zalando-incubator/kopf
- 40. 40 8 LINES OF PYTHON github.com/zalando-incubator/kopfgithub.com/zalando-incubator/kopf
- 41. 41 4 LINES OF DOCKER github.com/zalando-incubator/kopfgithub.com/zalando-incubator/kopf
- 44. 44 VMWORLD 2019 SESSION WITH KOPF github.com/embano1/kopf-operator-vmworld
- 45. 45 KOPF: FEATURES • Custom & built-in resources supported (CRDs, Pods, Services, etc) • Agnostic to API clients: kubernetes-client, pykube-ng, raw HTTP, etc • Immediate reaction to changes and events • Predefined behavioural patterns: • Simple spy-handlers for event watching • Advanced cause & diff detection for actual change tracking • Retry-until-success approach to handlers • Operator resilience: • Restores its state on restarts • Operator testing toolkit (minimally sufficient) github.com/zalando-incubator/kopfgithub.com/zalando-incubator/kopf
- 47. Kubernetes Web View "kubectl for the web"
- 49. 49 MOTIVATION: PENDING PODS INCIDENT ⇒ I want to see all "Pending" Pods across all clusters!
- 50. 50 MOTIVATION
- 51. 51 SOME USE CASES Support interactions with colleagues • share links • avoid human mistakes (wrong cluster login on CLI, "-n" param, ..) • deep link to resource YAML Incident response and investigation • find a resource by name across clusters to help troubleshoot • find pending pods across all clusters • help identify if a problem is only in one cluster or multiple ones
- 54. 54 SEARCHING FOR DOCKER IMAGE
- 57. 57 UPGRADE TO KUBERNETES 1.14 "Found 1223 rows for 1 resource type in 148 clusters in 3.301 seconds."
- 59. 59 MULTIPLE CONDITIONS kubectl -l application=coredns
- 60. 60 MULTIPLE RESOURCE TYPES kubectl get pods,stacks,deploys,..
- 61. 61 LABEL & CUSTOM COLUMNS labelcols=application customcols=CPU+Requests=join(', ', spec.containers[*].resources.requests.cpu) uses JMESPath expression
- 62. 62 URL STRUCTURE /clusters/{cluster}/namespaces/{namespace}/{resource-type} cluster: cluster aliases separated by "," or "_all" namespace: namespace name or "_all" resource-type: plural names separated by "," or "all" (like kubectl get all) https://kube-web-view.readthedocs.io/
- 63. 63 QUERY PARAMETERS FOR LIST VIEW sort: column name to sort by, optionally with ":desc" labelcols: additional label columns (comma separated) or "*" selector: label selector (like kubectl -l), will be passed to Kubernetes API filter: post-processing filter, either a simple string, Column=Value or Column!=Value hidecols: hide certain columns customcols: custom columns using JMESPath limit: number of rows to show https://kube-web-view.readthedocs.io/en/latest/features.html#listing-resources
- 64. 64 CUSTOMIZATION Nearly all aspects can be customized for your org via CLI options: • External Links (monitoring dashboards, ..) • Default Label and Custom Columns • Sidebar, HTML templates • OAuth2 integration kube-web-view.readthedocs.io/en/latest/customization.html
- 65. 65 LINKING FROM DEPLOYMENT STATUS
- 67. 67 TESTING WITH PYTEST-KIND AND PURE HTML • Pykube • aiohttp • pytest-kind • Jinja2 templates
- 68. 68 KUBERNETES + PYTHON? • Pykube • asyncio / aiohttp • pytest-kind • Jinja2 templates codeberg.org/hjacobs/kube-web-view/
- 72. 72 KUBERNETES + PYTHON = github.com/hjacobs
- 73. 73 KUBERNETES + PYTHON = code examples from this talk can be found in codeberg.org/hjacobs/kubernetes-and-python
- 74. 74 ZALANDO OPEN SOURCE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws AWS ALB Ingress controller github.com/zalando-incubator/kube-ingress-aws-controller External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando/postgres-operator Elasticsearch Operator github.com/zalando-incubator/es-operator Kubernetes Metrics Adapter github.com/zalando-incubator/kube-metrics-adapter