Dublin JUG February 2018 - Microservices in action at the Dutch National Police
0 likes146 views
The document discusses the Dutch National Police's adoption of a microservices architecture and DevOps practices. It describes their private cloud platform, use of 5 cross-functional teams to build applications, and event-driven microservices with technologies like Spring Boot and Kafka. It also covers their continuous delivery pipelines, automated deployments to OpenStack, and challenges of maintaining over 50 independent services.
Dublin JUG February 2018 - Microservices in action at the Dutch National Police
- 1. [email protected] Microservices in action at the Dutch National Police Bert Jan Schrijver @bjschrijver
- 2. Bert Jan Schrijver L e t ’ s m e e t @bjschrijver
- 3. Architecture and platform Frontend Methodology and culture Introduction Development and testing Build tools, deployments and running in production Challenges and looking ahead Outline W h a t ‘ s n e x t ? Backend @bjschrijver
- 4. The police protects the democracy, maintains the law and is the authority on the streets. Around 65.000 people work at the Dutch police, of which over 1500 IT professionals. Dutch National Police
- 5. CLOUD PLATFORM ANALYSE PATRONEN BIG DATA SECURITY5 DevOps teams are building high tech big data web applications in a private cloud environment. These applications support police related themes. Product line Cloud | Big Data | Internet
- 7. • 5 teams, separate backlogs • Overall planning at start of sprint • Minimal planning ritual • Usability tests as part of sprint • (Almost) no meetings • Phabricator as tool of choice Methodology
- 8. • Continuous Delivery & DevOps • Short feedback loops • Embrace change • Minimal dependencies outside team • Invest in people, not in products • Open, transparent, verifiable Culture Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg
- 9. • How to handle support, monitoring and (pro-active) maintenance during sprint? Make this a role that cycles through the team. “Operator of the day” and “Developer of the day” Culture Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg
- 11. • End-to-end security and encryption • Version control for everything • Horizontally scalable, no single points of failure • No runtime dependencies on external systems • Standardised naming • Right tool for the job (polyglot) Architecture
- 12. Architecture • This slide is intentionally left blank…
- 21. Source: https://www.google.com/about/datacenters • OpenStack private cloud • Ceph distributed storage • General cloud services for police organisation • 3000 managed desktops • Automation starts when hardware boots Platform
- 22. Source: https://www.google.com/about/datacenters • 100’s of physical servers • 14 physical disks per server • It’s not a matter IF stuff breaks but WHEN Platform
- 23. Source: https://www.google.com/about/datacenters • How to manage 100’s of physical servers, 1000’s of virtual servers and 1000’s of desktops? Infrastructure as code. Terraform for cloud, Puppet for servers, Ansible for desktops Platform
- 24. Frontend
- 25. • Angular 5.x, TypeScript, RxJS • Bootstrap, responsive design • Graceful degradation when backend fails • All console logs/errors go to centralised logging system Frontend
- 26. • How to decouple building a feature from release and deployment? Use feature toggles. Frontend
- 27. • How to prevent creating a monolithic frontend? Apply microservice principles to the frontend too. Create re-usable frontend components as standalone component libraries. Frontend
- 28. Backend
- 29. • Small in size, single responsibility • Runs in its own process • Independently develop, deploy, upgrade, scale • Has its own data store • Distributed by default • Potentially heterogeneous/polyglot • Light-weight communication Anatomy of a microservice
- 30. • Did we build microservices right from the start? Nope. • Why did we move to a microservices architecture? • Scalability: both for performance/load and development teams • Modularity: independent development and deployment of each part of the system • The ‘cool factor’ helped bit ;-) The path to microservices
- 31. • How did we move to microservices? • Split the existing system in modules (bounded contexts) - defined by business functionality • For each module, create a microservice • For every microservice, check if it now only serves 1 part of the domain • If so: cool. If not: repeat - split it up more. The path to microservices
- 32. • Spring Boot, Java 8, Maven • Stateless • 1 service in 1 jar on 1 JVM on 1 host (and 1 git project) • Minimal amount of shared code: • Security • Logging and metrics • Past: high available via load balancers • Present: service discovery Backend
- 33. Service discovery is an advanced pattern. Do you need it? Probably not. Do we need it? We think we do. Service discovery
- 34. Service discovery Why do we use service discovery? • HA & load balancing without single points of failure • Direct secure end-to-end client-service comm. • Kerberos: picky on DNS verification • Dynamic horizontal scaling • Resilience: automated response to failure • Location transparency • Zero downtime deployments
- 35. Consul cluster service-example0 service-example1 Gebruiker ui-example0 (webserver) consul-template sseserver Service discovery setup • Client-side discovery • Self-registration
- 37. • Feature branch based development • Master branch must always be releasable • Test environment on OpenStack runs the master branch • Feature branches only live locally and on the CI server Development
- 38. • How to locally run a system that consists of ~50 services? Run only the component(s) you work on locally. For other components, local env connects to test env on OpenStack Development
- 39. • Unit tests • Mutation tests • Service/integration tests: Spring boot integration, embedded in-memory data stores, REST assured • End-to-end test: Protractor • Load tests: Gatling Testing
- 40. • How to test feature branches in a microservices environment? Spin up branch version of component in a container on the CI environment. For dependencies, connect to test env. Avoid changing multiple components at once. Testing
- 41. Build tools, deployments and running in production
- 42. • Gitlab • Jenkins with Docker swarm slave nodes • Jenkins 2 pipelines • Nexus • Sonar Build tools
- 43. • How to manage >100 builds with a single team? Create modular, reusable build definitions. (See https://virtualjug.com/pipeline-as-code-building- continuous-delivery-pipelines-with-jenkins-2/) Build tools
- 44. • Every push to master is a release • Config embedded in executable jar • Deployments • currently: Rundeck and Puppet • soon: Nomad Deployments
- 45. • How to know when, what and which version(s) to deploy? • Minimise administration and think time. • Test environment: deploy component on commit • Everything from test -> acc during sprint • Everything from acc -> prod after sprint • Single component test -> acc -> prod when needed Deployments
- 46. • Logging and dashboards via Graylog • Metrics: • Spring Boot actuator • Distributed tracing with Zipkin • Grafana for graphs • Kafka stats via Burrow • Monitoring via Sensu and Flapjack Running in production
- 47. Photo: Dave Lehl Challenges and looking ahead
- 48. Challenges 01 Share as little as possible; prefer duplication over coupling. Sharing code between services 04Authentication and authorisation happen at every request. Find the balance between performance and security. Running stateless has a cost When moving fast, don’t forget to finish up before starting something new. Switching focus has a cost 06 Throwing something away and starting over can work out better than refactoring. Don’t be afraid to rebuild03 Microservices are not just for the backend. Modularity is just as important on the frontend. Monolithic frontend 02 Minimalize dependencies on other teams, or it will slow you down. Cross functional team composition is vital 05 and lessons learned @bjschrijver
- 49. Looking ahead Upgrades and fixes without users even noticing. 0-downtime deployments Our plans for the (near) future. @bjschrijver Cross-functional teams with vertical (full stack) responsibilities. Product teams Split the frontend in products and re- usable components. Modular frontend There is no silver bullet here, but useful tools and practices do exist. Automated security testing Get the teams the information they need, but only when they need it. Better dashboards and alerting
- 51. Thanks for your time. Got feedback? Tweet it! All pictures belong to their respective authors @bjschrijver