From java monolith to kubernetes microservices - an open source journey with jenkins x

Editor's Notes

• #3: We’re from activiti, we do open source business automation and business process management. We’re going to explain to you how we went from building our project as a monolith to microservices and how we found Jenkins-X and what we’re doing with it. We’ll talk about all the cool things Jenkins-X can do for you out of the box and how we’ve customised it.
• #4: Two years ago we on the Activiti project had a pretty standard setup for the time. One repository, one build job. One type of artifact - maven. Build process driven by features in CI (bamboo) Releases manually initiated
• #5: Microservices -> lots of different types of artifacts. Not just maven anymore. Not just maven & nexus -> decisions docker images: where to host? Where to run? Where should the nexus live? Should the docker registry come from the cloud provider? Use helm? Where to host helm charts?
• #6: Jenkins-X answers many of these questions. So Out of the box it is highly opinionated. It adds all the glue to run CI/CD well in Kubernetes and follow best practices Has a `jx create cluster` wizard - https://youtu.be/xN9eydj4SHQ?t=17 - concluding at https://youtu.be/xN9eydj4SHQ?t=146
• #7: We can see nexus and chartmuseum and so on https://youtu.be/xN9eydj4SHQ?t=213 JX is using Jenkins but isn’t just Jenkins. It’s Kubernetes-native
• #8: Wizards for apps as well as clients Jx import from cloned repo Adds e.g. dockerfile based on draft pack Adds jenkinsfile and creates pipeline
• #9: As well as importing existing apps, jx cli can create a new one from a template And give us a pipeline for it - so let’s see that https://youtu.be/xN9eydj4SHQ?t=250
• #10: So when I do `jx import`: Draft pack is applied Webhooks created on git repo Jenkins pipeline created based on Jenkinsfile (added by draft pack if not present) Main branch of pipeline runs (master by default) Version tag created and artifacts built. Maven artifact goes to nexus in the cluster Docker image built and goes to cloud provider registry Chart goes to chartmuseum in cluster PR goes to environment PR merged to environment and env chart deployed
• #11: JX creates namespaces out of the box for isolated environments. By default you get a staging environment and your apps get deployed there - https://youtu.be/xN9eydj4SHQ?t=186 Env represented by a git repository containing a helm chart. Big helm chart represents all the microservices = ‘umbrella chart’ or ‘environment chart’.
• #12: JX works in GitOps style - it uses git to represent environments. So if you were to lose an environment, you could get it back - everything is tracked JX puts as much in git as possible e.g. Jenkinsfiles So if your CI system dies, you don’t lose your pipeline definitions And a pipeline should run when git changes through webhook
• #13: CD approach in JX makes promotion as automated as possible, as much as possible So every commit to master is tagged, versioned and built And automatically promoted through PRs Emphasis with CD is on getting to prod, not sheltering code away Fixed promotion-ready versions, not temporary snapshots Promotion through automation, not code freezes for manual tests Only the production gate is manual by default (so continuous delivery, not continuous deployment)
• #14: To make sure the flow is crystal clear: Create a PR to a microservice (could here check it through preview env) Commit the change by merging it Pipeline runs based on webhook - it creates a fixed version tag and builds a docker image and helm chart version. Pipeline creates PR into the staging environment. If mergeable then the staging environment gets updated. There’s a similar promotion workflow from staging to production. https://youtu.be/xN9eydj4SHQ?t=454
• #15: Actually the previous slide skipped over preview environments but they are optional They are namespaces dedicated to an instance of a microservice for a PR By default you just get that version of the microservice there - but you can add more or link to services in the staging environment
• #16: Having JX propose answers for lots of tooling decisions helped us a lot It’s cool that all the whole workflow is kubernetes-native, from the use of gitops to build pods It showed us how we could do CD for k8s We wanted to be sure Activiti Cloud projects would able to do this For us to do it too we realised we had to depart from some defaults
• #17: We are a vendor building open source libraries and components - so we needed hosting to be more public So we had to override some things not in the wizards Specifically in secrets in the JX k8s configuration. We also wanted to run various types of end to end tests in different namespaces but we’ll come back to that bit later.
• #18: JX defaults are aimed at a building typical web apps. Our project structure more complex than typical web app. We’re a vendor producing libraries that people can build on top of. And we’re also building projects ourselves using those libraries. Our pipelines to need chains of deps.
• #19: One pipeline needs to update another pipeline or several. So not just an environment requirements file being updated by pull requests resulting from builds of applications. Need PRs for maven artifacts etc. PR targets could be other projects.
• #20: JX under the hood uses updatebot. A java project - a jar file that has a jenkins plugin and you can use it in your pipelines. Specify list of target/downtream repos in yaml file Call update with version and type of artifact It scans the target repos and sends the update there, creating PRs.
• #21: https://drive.google.com/file/d/1yM2DBP-1WeMpAkAovOp2QqJS_wjHbSh0/view?usp=sharing Some projects consumed by several projects e.g. ‘activiti-cloud-dependencies’. So its Jenkinsfile tells updatebot to push the version to the configured target repos Updatebot goes through all target repos, clones each, determines where to put the version and creates a pull request. This happens right after we’ve pushed the maven artifact - it is the same process for other types of artifact. PR gets evaluated on the other end - the target project has its own configuration to run tests against any pull requests. Here some of the downstream repos are controlled by different JX clusters.
• #22: Updatebot means lots of PRs. We don’t want to be merging those manually. And CD means reduce manual gates. So we needed good tests in each of the repos. mergify: PRs are automatically merged when tests and checks pass.
• #23: We build different sets of examples from our base libraries. Need to test interactions of components per example with E2E tests We represent each example with a helm cart, each with a dedicated pipelines. Branches for those pipelines dynamically create namespace per PR. E2E tests run against namespace and it is destroyed at the end. Kinda like PRs into a staging environment but envs created and destroyed dynamically. Like preview envs but without having to do service linking to talk to other services.
• #25: Lots of pipelines Multiple instances of apps with each app instance containing a bunch of different microservices and databases. So we’ve got some fairly powerful clusters running this. Jenkins-X uses an elastic pool of build pods to run the build jobs so it can scale up and down… but still needs a fair bit of power. We chose multiple clusters for isolation of the different parts of our project for clarity and independent upgrading - we could’ve run it all on one cluster from a resourcing point of view. JX can also be multi-tenanted.
• #26: With all these PRs is that occasionally you get little github blips. It’s so rare that you don’t normally notice it but we make loads of requests. So we’ve had to put retries in and make sure we’re handling the odd failure. General distributed systems considerations but here applied to the CI system - because it’s distributed.
• #27: A lot of activity going through these pipelines. A couple of times we have hit github’s rate-limiting. Currently we have a user dedicated to updatebot and we rarely hit the limit but if we grow much more then we’ll have to start using separate users for different concerns.
• #28: Some configuration appears in multiple places in our setup...especially helm charts. At the library layer very little repetition. Every app has a helm chart but some of the apps have pretty much the same chart. There are ways to deal with this but it requires thought and some tradeoffs.
• #29: We’re just using ‘jenkins-x classic’. There’s another version of jenkins-x you can choose if you want that is built on an even more powerful model, with knative to do run jobs on kubernetes in a way that scales much like serverless. https://medium.com/@jdrawlings/serverless-jenkins-with-jenkins-x-9134cbfe6870 We’re not quite ready to migrate to that yet but it’s certainly exciting and we’re keeping a close eye on it. We’re also keeping an eye on options for prod rollout strategies, esp istio - https://blog.csanchez.org/2019/01/24/progressive-delivery-with-jenkins-x/