5 steps to take setting up a streamlined container pipeline
0 likes303 views
The document outlines 5 steps to set up a container pipeline: 1. Use versioning and container registries like GitHub, Docker, and private registries to manage code versions and container images. 2. Use an orchestration engine like Kubernetes to manage and orchestrate container processes. Common options are AWS EKS, GCP GKE, and Oracle OKE. 3. Provision the Kubernetes cluster using scripts or Terraform on cloud infrastructure like OCI. 4. Implement container pipelines using tools like Oracle Container Pipelines to automate building, testing, and deploying containers. 5. Use Helm to package and deploy Kubernetes applications and integrate it into the CI/CD pipeline
5 steps to take setting up a streamlined container pipeline
- 1. Michel Schildmeijer 5 steps to set up a container pipeline
- 2. Me • From Amsterdam, the Netherlands • Lead Technologist at Qualogy • Oracle ACE since 2012 • Authored 2 books about WebLogic in 2011 • Started with UNIX in 1994 • Started with BEA technology in 2000 • Focus on containerization strategies, DevOps etc.
- 4. Investigate increasing error rate Increasing percentage of bugs Lack of continuous visibility in DevOps teams DevOps Challenges
- 5. Ingredients for a Cloud Native Container Pipeline • Versioning & Container Registry • Containers & Orchestration Engine • Provisioning • Container Pipelines • Packaging & Deployments
- 6. Setup a Cloud Native Container Pipeline 1 Versioning & Container Registry
- 7. Versioning & Container Registry • Versioning repo • GitHub for Git • Subversion • BitBucket • Container Registries • Docker • Vendor Registries • Private Registries: • OCIR: • Private,HA
- 8. Versioning & Container Registry • Use of Github authentication for Kubernetes • Use the webhook Module kubectl create -f https://raw.githubusercontent.com/oursky/kubernetes-github- authn/master/manifests/github-authn.yaml • K8S Webhook token authenticator for shareable access • Instruct the apiserver to use the webhook config --authentication-token-webhook-config-file --authentication-token-webhook-cache-ttl
- 9. Versioning & Container Registry • Use of Github authentication for Kubernetes • Use the webhook Module • K8S Webhook token authenticator for shareable access • Deploy keys per projects
- 10. Versioning & Container Registry kubectl create secret docker-registry dockerregsecret --docker- server=somewhere.com --docker-username=api.user' --docker- password=*********' --docker-email=‘[email protected] • Registry Secret in Kubernetes
- 11. Containers & Orchestration Engine 2 Containers & Orchestration Engine
- 12. Orchestration Engine - Kubernetes • Is a Framework for building distributed platforms • Manages & orchestrates container processes (docker) compute networknetwork storage
- 13. Orchestration Engine - Kubernetes • Various Cloud providers have a Kubernetes solution • Microsoft: Azure Kubernetes Engine(AKE) • Google: Google Kubernetes Engine(GKE) • IBM/ RedHat: OpenShift • Oracle: Oracle Kubernetes Engine(OKE) • Digital Ocean
- 14. OKE – Oracle Kubernetes Engine
- 15. Setup a Cloud Native Container Pipeline 3 Provision Kubernetes
- 16. • Scripted setup Kubernetes Engine • Any Cloud Infrastructure (OCI) • Terraform providers client Generate priv/pubkeys Set TF envars terraform plan terraform apply Setup compartment Verify tenancy& user details Setup API key K8S Orchestration Engine – Kubernetes - provision
- 17. Orchestration Engine – Kubernetes - provision TF init, plan and apply
- 18. Setup a Cloud Native Container Pipeline 4 Container Pipeline
- 19. Oracle Container Pipelines • Former Wercker CI CD • Docker-Native CI/CD Automation platform • For Kubernetes & Microservice Deployments. • OpenSource • Artifact can be a packaged Docker Container
- 20. Container Registry AD 1 AD 3 Node Pool K8S Cluster Node Pool BM VM Virtual Network PV AD 2 Kubernetes Exposed Service Orchestration Container Engine (Kubernetes) Pods LB Test Push Build Oracle Container Pipeline Flow GitHub Orchestration web API
- 21. Working group Can have many applications Users can be in many organizations Wercker Organization
- 22. Links to a project on Github, Gitlab, or Bitbucket ‘Contains’ the build workflows Has dependencies, environment, permissions Wercker Organization
- 23. Steps □ Isolated bash script or compiled binary for accomplishing specific automation tasks. Pipelines (pipeline consists of steps) □ Pipelines: a series of steps that are triggered on a git push or the completion of another pipeline. Workflows □ Workflows is a set of chained branched pipelines to form multi- stage, multi-branch complex ci/cd flows Concepts for building pipelines
- 24. • Added to wercker.yaml • Add Pipeline: deploy-to-cluster • Add K8S specifics Oracle Container Pipeline
- 25. Terraform Step - script: name: Set Terraform variables code: | export TF_VAR_my_var1=foo export TF_VAR_my_var2=bar - releasequeue/run- [email protected]: action: "plan" out_file: "output_file" state_stor_bucket: "test- terraform-state-bucket" state_stor_bucket_region: “eu-fran-1" state_stor_key: "terraform_state"
- 26. Steps: install packages, run tests, etc. env variables, SSH keys ‘build’ pipeline created by default ‘Pipelines’ concept is in the web UI and in the wercker.yml file Pipelines
- 27. Injected into repository creating app □ Automatically creates a ‘build’ pipeline and workflow with a hook to the repository Various templates □ Go, javascript, java, node, php, python, etc. Box: base Docker image Service dependencies □ each pipeline can have its own box and service dependencies Pipelines Steps/After-Steps Yaml file
- 28. Flexible linking of pipelines Requires pipelines in wercker.yml and configured in the web UI. □ Depends on yml file, but workflows are not in the yml file Variation based on branch Parallel processing – concurrency Pipeline workflow
- 29. Helm 5 Helm
- 30. Helm • Release and Package Management for Kubernetes • Helm is client, Tiller runs Serverside in Kubernetes • Can be integrated with CI build tools ( Maven, Jenkins, and Wercker) curl https://raw.githubusercontent.com/helm/helm/master/scripts/get > get_helm.sh
- 31. Helm helm init $HELM_HOME has been configured at /home/oracle/.helm. Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy. To prevent this, run `helm init` with the --tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation Happy Helming!
- 32. Helm kubectl create serviceaccount --namespace kube-system tiller serviceaccount "tiller" created kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin -- serviceaccount=kube-system:tiller clusterrolebinding "tiller-cluster-rule" created kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}' deployment "tiller-deploy" patched
- 33. Helm Basic Components .helm/ charts/ templates/ Chart.yaml values.yaml Basic info about name, description, version of app Templates for charts; tiller parses them with values Values for parsing templates to tiller
- 34. Helm workflow architecture Helm ChartHelm Chart tiller server K8s api-server Chart deployed Application
- 35. Add these all to wercker pipeline Setup the git repository Add application wercker_helm- chart-generate Don’t generate wercker.yml Set the proper variables Run the build
- 36. Add these all to wercker pipeline Setup the git repository Add application wercker_helm- chart-generate Don’t generate wercker.yml Set the proper variables Run the build
- 37. Add these all to wercker pipeline Setup the git repository Add application wercker_helm- chart-generate Don’t generate wercker.yml Set the proper variables Run the build