DevOps Patterns to Enable Success in Microservices

© COPYRIGHT 2019 COVEROS, INC. ALL RIGHTS RESERVED. 1@armillz
DevOps Patterns to Enable Success with
Microservices
Richard Mills
DevOps Solution Architect, Coveros Inc.
rich.mills@coveros.com
@armillz

Who is this guy?
• Me: Mad-Software-Developer turned Mad-Software-Engineer
turned DevOps-Solution-Architect. Pragmatist. Particular
focus on tools and automation. CI, CD, Agile, DevOps …
what’s next?
• PS: Thanks for inventing the term “DevOps” to describe what I like
to do.
• Pays my bills: Coveros helps organizations accelerate the
delivery of secure, reliable software using agile methods. We
provide expert consulting, mentoring, and training in:
• Agile transformations, coaching, development
• Agile testing and automation
• DevOps and DevSecOps transformation and implementations
• DevOps engineering

Why is he here?
• Share some of my experiences (and failures) on a journey from “classic” enterprise monolithic
systems into the “utopia” of microservices architecture
• Show some patterns for delivering highly-buzzword-compliant, microservices-based software
applications
• Give you a reference to walk away with
• NOT: Explain fundamentals of DevOps (or Agile)
• NOT: Sell you on DevOps (or Agile), necessarily
WARNING: I do everything very fast. Try to keep up.

Microservices: utopia or nightmare?

Microservices are all the rage!
“Let’s take all of our busted monolithic systems and
split them up!” they said.
Successful microservices require a combination of
technical architecture, automation, and
development methodology that all relate closely to
Agile and DevOps
Without the right mix of these, things often don’t go
as well as you planned

Microservices in the non-ideal real world
• Federated services with hidden dependencies
• Independently changing applications, but ...
• Loosely coupled, unspecified, AND STILL-PRESENT dependencies
• Teams that don’t talk to each other
• Hide behind (poorly defined) APIs
• Freedom to move at their own pace
• Desire: Deploy small changes rapidly to production
• Problem: can’t figure out which versions work together, even in test environments
You have many things changing at different rates (fast vs. never) with
“decoupled” interfaces that are hard to verify independently

What you need
•Bring teams closer
•Continuously integrate services
•Deploy integrated services into various environments
•Frequently test still-monolithic system in pieces and as a whole
This looks suspiciously like a DevOps problem
Conclusion: your DevOps pipeline is critical to your success

Context:
Modern Microservices Example

Dozen-ish independent services acting as a federated product in a registration and workflow system
React/JS front-end UI, Java Spring Boot REST Services, PostgreSQL back-end
Service Architecture
NRPM
UI
REST API
DB
Page
Service
Case Mgmt
UI
REST API
DB
App Entry
UI
REST API
DB
V&A
UI
REST API
DB
Auth
UI
REST API
DB
Gateway
UI
REST API
DB
UI
PostgreSQL RDS instance in AWS
External
Services
REST API
REST API
REST API

DevOps Baseline – what do we mean?

A DevOps Solution Includes...
• Team structure and skill set across the delivery chain
• Infrastructure and tools to support automation and team collaboration
• Delivery pipeline to build, package, deploy, test for repositories and branches of code
• Test automation (lots and lots of it) capable of evaluating changes
... all wrapped up in a supporting Agile development methodology
Overall: the ability to deliver independent changes into a working
environment rapidly and reliability while maintaining a high quality
system

Pipeline Design: Patterns of Success

Pipeline defines delivery process
The software delivery process is automated through a CI/CD pipeline to deliver application
microservices into various test (and eventually production) environments
Continuous MonitoringContinuous Integration
BuildCode
Continuous Delivery
commit
DEV
TEST
PROD
Compile
Test
Package
Publish
Unit Test
Integration Test
Code coverage
Code Scanning
Static Analysis
Bugs
Vulnerabilities
Technical Debt
Package
Dependency
checking
Vulnerable
components
Deploy Test
Provision
Install
Configure
Vulnerability Scanning
Deployment verification
Smoke test
Manual Testing
- Exploratory
- UAT
Functional Testing
- Behavioral
- API
- Web UI
Non-Functional Testing
- Load/Performance
- Web Security
Platform
Vulnerabilities
Test Results Defects
Change Requests
Test Promote
Operate
Monitor

Pipeline links to branching strategy
•Strive for "main line" development
• Small changes are easy to build, test, deploy, AND FIX
• Use short, small feature branches for isolated changes
• Consider Github Flow (very simple), Git Flow (complex)
• Avoid "parallel release development" at all costs
•What problems does branching cause vs. solve?
•Delivery pipeline will align with branching

Pattern: minimalist branching delivery pattern
• Features – simple CI build with compile, static analysis, unit test
• Pull Request (merge request) – pre-merge with master, deploy to on-demand ephemeral
environment, run automated tests (more on tests later)
• Use peer review to investigate impact on other services
• Master – after successful merge, build and deploy to dev-staging environment, run more
tests, promote upwards
• Remember:
SMALLER IS BETTER

Foreshadowing: branch everything with the code
• Tenet of DevOps and CI/CD: version-control everything so you can automate
everything
• We will discuss a lot of different artifacts that match your application code
• Application code
• Build scripts
• Deployment scripts
• Database scripts
• Tests
• ... more
• All of these should be stored and branched as a related code base!
Stand by for station identification...

Pattern: independent pipelines
• Each service has it’s own pipeline (elsewhere: shared pipeline code)
• Example: use “Jenkinsfile” or localized build script configuration
• Major parts of the “independent” pipelines
• Compile
• Unit test
• Static analysis (security, quality, composition analysis)
• Packaging
• Integration testing ... Pre/post deployment
• Test deployment
• Is your service independently deployable?
• How much testing can you do?
• What are your dependencies? (and what will eventually depend on you?)

Pattern: pipeline should branch with code
• Your build/delivery process will evolve with the code
• Example: Build scripts ... historically always live with code (maven, gradle, ...)
• Your “pipeline” automation code should live with code, as well.
• Pattern: Use “pipeline as code” to define your pipeline automation
• NOT: Traditional complicated UI-driven multi-job monsters (more elsewhere)
• Store pipeline definition (e.g., “Jenkinsfile”) in repository with code
• New branches of code can have local build/test pipeline updates
• Similar for deployment and configuration scripts (more elsewhere)
• Ansible/Chef – playbooks and recipes change to address code changes (new
config parameters, etc.)
• Docker/Kubernetes/Helm – orchestration and deployment YAML for containers
adjust with code
Ideally, these live in the same code repository

Pipelines run in parallel
• Each of your independent service pipelines can (will) execute in parallel
• This can be very resource intensive – plan for it
• At some point, you need to deploy your code into an “environment”
• Will these be shared/mixed environments, or dedicated to the independent pipeline?
• Note: static environment can cause issues if you are changing multiple things simultaneously
• “Dynamic” environments become VERY beneficial...
App
A
W
B
C
D
App
A
W
B
C
D
Service “A”
Service “B”
App
A
W
B
C
D
- OR -

Pattern: dynamic on-demand environments
• Microservices can have high rates of change across many services
• This can continually disrupt shared/static environments as new services are updated
• The ability to launch a new environment with all supporting services becomes valuable
• Replicate current PROD (or other) version of services
• Deploy newly changed service into replica
• Test, test, test
• Seeding data becomes important for new copies
App A
W
B
C
D
App A
W
B
C
D
PROD Replica

Pipeline join points
• At some point, your pipelines come together and
services work in a combined environment
• You will need to deploy multiple services and make
them work together
• Need some snapshot of “versions” to know which
versions are where and how they are verified against
each other
• Options:
• extensions of independent pipelines
• ”join” jobs that deploy lots of things together
• Note: consider “locking” shared environment during
testing to avoid blips and test failures
App
A
W
B
C
D

Pattern: capturing a “product” version
• As your services progress through environments, it’s useful to know what versions of services
work with each other
• Your “promotion” mechanism might need to consider moving groups of services together
• Single service
• Group of known-good services
• This may identify your subsystem of coupled monoliths
• Capture versions of your services
• SHA, semver, whatever
• Pattern: typically capture in single “release inventory” file to feed to deployment automation scripts

Pattern: UI and REST service code integration
• Problem: UI invokes back-end services through APIs
• What happens when they need to change?
• How do you test these continuously?
• Similar to cross-service integration
• Scenario 1: Full-stack developers building UI and API
• Consider having UI and API code live together
• Aligns UI and API changes automatically
• Scenario 2: Separate UI and API teams (anti-pattern?)
• UI and API might change at different rates
• Need mechanism to alter/change APIs (deprecate, multiple versions, etc.)
• UI and API might be in different repositories
• This leads to cross-service testing issues (stay tuned for a few more slides...)
App
Aw
W
Bw
Cw
Dw
A
B
C
D

Ensuring quality (aka: testing)

Your efforts in DevOps (and microservices)
will fail without proper automated testing
and assessment that is fully integrated into
the pipeline.

Test factors to consider with microservices
• Take every opportunity you can to test the code
• This will enable rapid changes throughout business cycle
• Testing paths to consider
• Testing in isolation (totally in your control)
• verify what you think your contract is
• Testing with downstream dependencies (database, other services)
• What else do you need to work properly?
• Verify what you think other contracts are
• Testing with upstream dependencies (UI, other services)
• Who needs YOU to work properly?
• Verify what other people think your contract (actually) is
• ”The Pipeline” needs enough global knowledge to orchestrate this
Pathological case: what happens when APIs change unexpectedly or don’t
fulfill their (poorly specified) contracts?
App
W
B
D
A
C

Pattern: tests live with code and pipeline
•Tests for microservices should live with the microservice code
• Unit tests ... part of code tree (e.g., junit) ... Duh!
• Automated REST tests ... part of code repository (Postman, REST Assured, ...)
• UI tests ... part of UI repository (Selenium, ...)
•This allows you to easily match code version to test version
Pro tip: bundling tests in a docker container makes them easy for anyone
to execute

Challenge: cross-service and cross-branch testing
• Major problem: multiple services (e.g., UI or multiple APIs) need similar linked
changes
• How do I implement this?
• Code based solutions (feature flags, API versions, ...)
• Deployment based solutions (concurrent running service versions)
• More important: how do I test this?
• Will depend on your implementation
• May require the ability to link changes between services/repositories
App
A’
W
B
C’
D

Pattern: testing simultaneous changes
• Deploy and execute simultaneous branch and cross-service testing
• Example: specify multiple builds or deployment versions during test environment creation
• Build/deploy multiple changed versions into same environment
• Minimize concurrent versions to avoid integration problems
• Force the integration problems as soon as possible
• Anti-pattern: multiple versions of all APIs deployed everywhere
• How do you wire them together?
• Do you have to develop parallel versions? ß torture
App
A’
W
B
C’
D
App
A’
W
B
C’
D
A
?

Important Non-Pipeline Aspects

Importance of team structure and communication
• Teams must align with delivery process (and Conway’s law)
• Your teams are likely part of a shared product
• If not: great! Rock on with your independence!
• If so, don’t pretend to be separate
• Example: ever had problems with remote, isolated vendors?
• Communicate with each other!
• Regardless, you will likely have shared patterns (devops, testing, architecture) that can be
leveraged to avoid re-inventing the wheel in your enterprise

Example “DevOps” product team structure
Our team organization on a 50-ish person development project
Important for the vertical development teams to actually work with each other (painful at first)
Horizontal Teams
DevOps Engr
DevOps Lead
Platform Engr
Platform Lead
Platform Engr Security
Security
Security
Performance
Performance
Agile Vertical Team
Scrum master
DevOps Engr
Biz Analyst
Agile Tester
Front End
Full Stack
Agile Vertical Team
Scrum master
DevOps Engr
Biz Analyst
Agile Tester
Front End
Full Stack
Agile Vertical Team
Scrum master
DevOps Engr
Biz Analyst
Agile Tester
Front End
Full Stack
Agile Vertical Team
Scrum master
DevOps Engr
Biz Analyst
Agile Tester
Front End
Full Stack

Integrating with software architecture
• Not all software lends itself to easy build, test, and deployment
• Work directly with software architecture and development teams
• Meaningful, testable service architecture is critical
• Each software service must support:
• Rapid build
• Automated test (controllability, observability) – between services
• Data initialization
• Installation/configuration
• Monitoring/metrics
• Cross-service debugging
• Standards and "Definition of Done" should reflect this
• Development stories aren’t complete until all these things work
• Keep product value and cross-service features in mind

Take-aways
• Version control everything, automate everything
• Your pipeline implements your branching strategy
• Independent pipelines with join points
• Use dynamic environments (containers!)
• Store and branch everything together – code, tests, database, pipeline code
• Test in isolation, test with downstream, and test with upstream
• Use shared/central pipeline libraries to avoid duplication and enforce standards
• And don’t forget:
WORK TOGETHER, DAMMIT!

Questions?
Thank you!
rich.mills@coveros.com
@armillz
https://www.coveros.com/services/devops/

DevOps Patterns to Enable Success in Microservices

More Related Content

What's hot (20)

Similar to DevOps Patterns to Enable Success in Microservices (20)

Recently uploaded (20)

DevOps Patterns to Enable Success in Microservices