From Legacy to a Microservices Architecture
2 likes779 views
This document discusses Briefy's transition from a legacy architecture to a microservices architecture using various technologies like Python, ZCA, Pyramid, Angular 4, and Deis. It provides an overview of Briefy as a company that creates visual content at scale. It then describes the three pillars that Briefy's solution is based on: being API-centric, using microservices, and being event-based. The document outlines Briefy's tech stack, infrastructure, shared codebase, productivity tools, core services, and support services. It provides more details on Leica, Briefy's core workflow and scheduling solution, including its domain models, user interface, API, and tasks. Finally, it discusses how every action in
From Legacy to a Microservices Architecture
- 1. From Legacy To a Microservices Architecture Python, ZCA, Pyramid, Angular 4 & Deis
- 2. About me Ruda Porto Filgueiras Software Architect at Briefy http://briefy.co From Brazil, based in Berlin Twitter @rudaporto [email protected]
- 3. The Company
- 4. Briefy Team
- 5. Creating visual content at scale INPUT & IMPORT We define a briefing together with our Customers. After this we receive a listing with all items to shoot Hotels, Flats, Restaurants, etc. Receive Orders ASSIGNMENTS We connect Creatives assigning them to Orders in the location are available. Creative schedules the shoot with the third-party contact and execute it. Assign Creative VALIDATE & PACKAGING Creative upload assets and we validate and do QA to make sure we are following the briefing and specific requirements. We deliver the content to Customer: SFTP, GDrive, API. QA & Delivery
- 8. Overview
- 9. A solution based on three pillars API CENTRIC To serve distinct use cases using the same codebase, Briefy built its solution around REST APIs. Web, Mobile and customer integration were the planned consumers. MICROSERVICES Single responsibility and distinct life cycles for parts of the solution pointed Briefy to use a micro services architecture. EVENTS BASED Most of the communication between the services could be done asynchronously, so an event-based — with messaging — solution was implemented.
- 10. Code quality and reuse as principles TESTING Codebase testing with Continuous Integration and Continuous deployment CODE QUALITY Style guide and documentation for the entire codebase DEVOPS Use existing toolset to minimise devops work. CODE REUSE Shared packages/libraries Build upon existing OSS codebase
- 11. Tech stack INFRA Storage: GDrive & S3 Messages: SQS Cluster: Kubernetes 1.7.8 DEIS 2.18 API Python 3.6.3 Zope Component Architecture Pyramid 1.9 / Cornice SQLAlchemy 1.2 Postgresql 9.6 DEV TOOLS Repo: GitHub CI: TravisCI Logging: Logstash Reporting: Sentry / NewRelic / Slack WEB FRONTEND Angular 4 Typescript Generic views Generic Listings Generic CTAs
- 12. Infrastructure
- 13. AWS THREE ENVIRONMENTS Live: eu-central-1 Staging: us-east-1 Development: eu-west-1 SERVICES Message routed using SQS S3 for static data & assets Kubernets Cluster on EC2 Dies Workflow
- 14. Kops AUTOMATE Kubernets cluster creation with a single command line, including bastion and nat High availability, multiple SDN options MAINTAIN Cluster upgrade and change Export format templates to cloud formation and terraform
- 15. Kubernets KUBERNETES OVER EC2 One master, n minions DEIS ON TOP OF KUBERNETES Workflow solution similar to Heroku PaaS EC2 WITH AUTO SCALING T2 and M4 instances.
- 16. Deis Workflow DOCKER Automatic build and save images SIMPLE git push deis master TWELVE-FACTOR And also support legacy if necessary
- 17. Devops MONITORING WITH NEWRELIC Servers and applications REPORTS ON SLACK / EMAIL Deploy, errors and alerts posted on tech channel AUTOMATION WITH ANSIBLE AWS and Cloudflare integration
- 18. Code
- 19. Shared codebase BRIEFY.COMMON Events / Queue Base Models / Mixins Logging Workflow BRIEFY.WS REST API blueprints Validation Error handling BRIEFY.GDRIVE Common GDrive actions Create, Copy, Metadata, List, Download BRIEFY.S3 Common S3 actions Create, Copy, Metadata, List, Download
- 20. briefy.common QUEUE WORKER Queue worker to consume SQS messages and dispatch WORKFLOW Workflow engine to create solid and secure business processes EVENTS Event firing to internal component architecture and to SQS MODELS Base mixins and sqlalchemy models with versioning, local roles support and polymorphic types CACHE Cache object serialization on redis with dogpile.cache
- 21. briefy.ws FILTER Sort and filter listings automatically based on model fields RESOURCES Base classes to exposure models as API endpoints: REST, workflow transitions, versioning EVENTS Object lifecycle events firing SCHEMA Automatic colander schema generation for validation and deserialization SECURITY Resource factory to dynamically compute ACLs based on model workflow and local roles
- 22. Productivity tools CHECKSTYLE Linting rules and code style guide COOKIECUTTERS Templates to create new packages and services DOCKER IMAGES Python 3.6.3 Plone 5.0.8 Nginx / Varnish
- 23. Services
- 24. Core services MS.LAURE Automatic quality assurance and renaming of assets LEICA Workflow and scheduling solution. API-based CHOREOGRAPHER Events handling and routing. ROLLEIFLEX Authorisation and user management service. ALEXANDRIA Digital Assets management solution API-based
- 25. Support services CMS Content management backend for Briefy website. Using Plone CMS. THUMBOR Image server handling automatic cropping, resizing and image transforms BRIEFY.SLACK Service handling Slack notifications. BRIEFY.MAIL Service handling Mail notifications using Mandrill as the provider HOTELS CRM CRM solution to deal with Hotels acquisition for the City Packages initiative. LEADS Service handling leads and quotes coming from briefy.co website. SITEMAP Service that generates a sitemap.xml for CMS and static content. SEO helper API GATEWAY Gateway and router handling public accesses to Briefy API
- 26. Leica: from briefing to assets
- 27. The core of Briefy WORKFLOW & PERMISSIONS Every model have its own workflow HUNDREDS OF EVENTS Workflow transition, task execution, object creation and update. DATA MODELS Customer, Project, Order, Assignment, Creative and User profiles Being the scheduling and workflow solution for visual content creation, Leica is the core of Briefy operations.
- 28. Elements of Leica BRIEFY.FRONTEND Web application developed with Angular4, connects to Leica API. This app has 3 distinct users: Customer, Creative and Briefy team member, with distinct permissions. User Interface BRIEFY.LEICA Domain models, permissions, business logic and API endpoints. Codebase is well documented and 90% code coverage. API LEICA TASKS Scheduled tasks execute workflow transitions, generate reports and post KPIs. Share the same codebase as briefy.leica Tasks
- 29. Domain Models CUSTOMER Company requesting visual content creation. PROJECT A grouping of orders from the same customer. Usually attached to one contract (or agreement) ORDER A content creation request. When it is not confirmed, it is called Lead. Customer oriented. ASSIGNMENT A content creation execution schedule. Creative oriented. JOB POOL Groups available Orders to be offered to a group of pre-selected Creatives. CREATIVE A visual content creator - Photographer, videographer — profile. CUSTOMER USER Profile of a user attached to a Customer. BRIEFY USER Profile of a Briefy team member user.
- 30. Managing visual content creation at scale cost is complex Confidential Managing Visual Content Creation Is Complex
- 32. Every action, one event (1) ARIANA Assign a creative to an Order ORDER.WORKFLOW.ASSIGN 2017-07-27 15:46:10.03267 120 Times
- 33. Every action, one event (2) OPHELIE Validates a Creative PROFESSIONAL.WORKFLOW.VALIDATE 2017-07-31 15:55:11.436015 213 Times
- 34. Events trigger new actions ASSIGN. IS SCHEDULED Assignment is scheduled on Leica, generating an event EVENT IS ROUTED Actions are registered to this event, i.e.: Send email to Creative EMAIL SENT TO CREATIVE Using Mandrill, a message is sent to the Creative An event on Leica will be sent to Choreographer, there it will be routed to other micro services to execute new actions. This flow is asynchronous and all messages are sent using SQS queues. BRIEFY.LEICA BRIEFY.CHOREOGRAPHER BRIEFY.MAIL
- 35. Thank you! @rudaporto [email protected] +49 176 28607522 Rudá Porto Filgueiras