K8Guard - An Auditing System For Kubernetes

K8Guard
An Auditing System For Kubernetes
Medya Ghazizadeh
Linux Foundation ONS March 26 - 29, 2018
Los Angeles, California

Apache V2 Legal and Disclaimers !
Can: Commercial Use, Modify, Distribute, Sublicense, Private Use, Use Patent
Claims, Place Warranty.
Can NOT: Hold Liable, Use Trademark
Must: Include Copyright, Include License, State Changes, Include Notice.

K8Guard was built in the same retail store that
makes this

Pre-K8S Journey
● Migrated from run time configuration (Chef) to Netflix’s Immutable Pattern.
○ Convert chef cookbooks to RPMs
○ Prevent run time errors
● Use Spinnaker to deploy everywhere
● Canary Deployments and Chaos Monkey-ed everything.
● Promote 12 Factor App Design.

Kubernetes is like the best tetris player
for your infrastructure

Shiny Problem 1: * in Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test
spec:
rules:
- host: *
http:
paths:
- path: /foo
backend:
serviceName: s1
servicePort: 80

Shiny Problem 2: 15 GB Image size
Why is that a problem?
How to avoid big image sizes?

Shiny Problem 3: Using Externally hosted images
- Deployment failed image failed to pull from docker hub
- Malicious code in Un-Scannable images

Shiny Problem 4: Root/Privileged Containers

Shiny Problem 5: Mount a HostPath volume
“hostPath”: A hostPath volume mounts a file or directory from the kubernete’s
host node’s filesystem into your pod. This is not something that most Pods will
need, but it offers a powerful escape hatch for some very very rare applications.

Shiny Problem 6: Single Replica
I admit this is the most controversial one !
Some people just don't need failover.

K8S Clusters
Problem Before
Facing Black
Friday
Efficiency
Security
Stability
Availability

A Kubernetes Prepared for Black Friday !

Keep The Lights On Even in Disaster !

Solution
Inspired By
The-Twelve-Factor

I. Codebase II. Dependencies III. Config IV. Backing
services V. Build, release, run VI. Stateless Processes VII.
Port binding VIII. Concurrency IX. Disposability X. Dev/prod
parity XI. Logs XII. Admin processes

Solution 1 - WRONG !
Wake up when with a pager’s alert and figure out
why k8s cluster is slow or down in the middle of
the night.
Set up 1-1 meeting with app teams, preach for
them the twelve factor app design and best
practices. Sending long email explaining back
and forth to each app team’s which does a
mistake.

Solution 2 - Scan/Action/Report Automatically
An Automated way to keep the K8S clusters form violations.
- Detect developer’s violations,
- Notify and warn them of their violation
- Take action on them on the violations
- Generate Metrics and Searchable reports for audits.

First Lets Defining Violation Types
Image Size Efficiency 5 GB image size
Image Repo Security Downloading
image from a shady repo
Extra Capabilities Security Setting UID/GUID
Privileged Mode Security Root containers
Single Replica Availability Not 12-factor app
Invalid Ingress Security/Stability Having "*” in ingress
Mount Host Vols Security/Stability Mounting kubernetes system files
No Owner Security No owner
annotation for namespace
Required Entity Security/Stability Required pod not deployed
Required Annotation Security/Stability Required annotation for

Enforce your policy on K8S entities
1. Deployments,
2. Pods
3. Jobs/CronJobs
4. Daemonset
5. Ingresses
6. Namespaces

K8Guard - An Auditing System For Kubernetes

K8Guard Discover
● Discover service, when in messaging mode, finds violations and puts them on a kafka topic.
and also discover API mode, is able to serve without depending on kafka. you can hit the end
points to get JSON response.

K8Guard Action
WHAT KIND OF ACTIONS DOES IT TAKE?
● Notifies the namespace owner (email, hipchat, …).
● After X amount of notifications, it will do a hard action such as:
○ Scale bad deployments down to zero.
○ Suspend bad jobs.
○ Delete bad ingress
Note that there is a safe mode - which only notifies and does not do hard actions.

K8Guard Report
Report service will generate a human readable and searchable report of all the past violation
actions.

/metrics
Discover API generates 19 metrics which are accessible at /metrics, and you can hook a monitoring
system like Prometheus to collect them and then generate pretty Grafana dashboards.
Violation metric examples:
● The number of all deployments.
● The number of bad deployments.
● The number of all pods.
● The number of bad pods.
Performance metrics examples:
● The number of seconds took to return all images from Kubernetes api.
● The number of seconds took to return all deployments from Kubernetes api.

Why Golang?
- Statically linked libraries
- Ship Binary, and run it on Docker Scratch !
- Small language, New libraries,
- Strongly encourages good practices (error handling, formatting, ...)
- Kuberentes is written in Go itself.

Batteries Included !
Developer Friendly !
How many times you tried to run
a github project and built failed ?
Some projects not only don’t give
you the batteries, they won’t
even give you an idea what kind
of batteries you have to buy.
Just push the code and good
luck !

Makefile !
make help
make build-deploy-minikube

Open Sourcing K8S
Target embraces open source.
second open source project at Target. After
winnaker

«Только все знают всё»

Thank You Open source Contributions
- Daemonsets
- HelmCharts
- Bug fixes
- Reddis and minimal technoogy stack

Open Sourcing Challenges !
There are no bad pull requests! There are only a
bad merges !

Future of
K8Guard
- Engine Rule.
- Plugin System.
- Cloud Messaging.
- Delegate notification
- Consolidate
Technology Stack

Keep in touch, Make your first PR!
● https://github.com/k8guard
● https://k8guard.github.io
● Twitter: @medyadaily

K8Guard - An Auditing System For Kubernetes

More Related Content

What's hot (20)

Similar to K8Guard - An Auditing System For Kubernetes (20)

Recently uploaded (20)

K8Guard - An Auditing System For Kubernetes

Editor's Notes