HAProxyconf 2019 - Criteo - Transitioning from Ticketing to LBaaS
0 likes505 views
The document discusses a transition from traditional ticketing systems to a more agile load balancing as a service (LBaaS) framework at Criteo, emphasizing cost-efficiency, operational agility, and control over infrastructure. It highlights user experience improvements, such as zero-configuration load balancers and enhanced observability, while detailing challenges and considerations around health checks and the disaggregation of load balancing services. The overarching goal is to enable efficient network management and service delivery across a massive scale of servers and infrastructure.
![41
Feedback
• Overall stability
• Small memory footprint
• Community + enterprise support
• "I'm investigating an issue we have with some fetch we're
adding on a custom tcp_info structure, and I'm wondering if
I might have discovered a broader issue[...]“
• -
• Deployed worldwide two hours after](https://image.slidesharecdn.com/haproxyconf2019-criteo-transitioningfromticketingtolbaas-191110141009/85/HAProxyconf-2019-Criteo-Transitioning-from-Ticketing-to-LBaaS-41-320.jpg)
HAProxyconf 2019 - Criteo - Transitioning from Ticketing to LBaaS
- 1. 1 Transitioning from Ticketing to LBaaS November 12, 2019 - Amsterdam William Dauchy @wdauchy Network-LB Team Lead Pierre Cheynier @pierrecdn Network-LB Senior SRE
- 2. 2 Criteo infrastructure in a nutshell • Focus on cost-efficiency, agility and regaining control on infra. software • Examples: • Commoditize hardware, challenge vendors by establishing direct ODM relationship • BIOS, BMCs and Switches OS: switch to OSS
- 5. 5 Network is not to be outdone • CLOS Matrix (RFC7938) • Network Services still in specialized racks
- 6. 6 User, 2016: My container is running, and so what? • Step 1: application is deployed at git push • Step 2: Come on, I’m missing some things here… • Public or private VIP? • DNS entries? • TLS certificates? • IPv6? • Traffic engineering? • Security policies enforcement? • Metrics? • Step 3: Don’t you want to .. fill a ticket?
- 7. 7
- 8. 8 API extensions for LB • Our team write DSL or API extensions • Same primitives everywhere • Linked to Consul registration • Features, not technologies (vendor agnostic) • Use Consul storage (KV/Metas) ► Ownership of app network config has been transferred!
- 9. 9 Health-checks • Business HC can be non-trivial • Multiple HCs on a service • Will all technologies have the support for running them … remotely? • Multiply sources of checks = lack of consistency
- 10. 10 Health-checks • Business HC can be non-trivial • Multiple HCs on a service • Will all technologies have the support for running them … remotely? • Multiply sources of checks = lack of consistency ► Consul as a state reference
- 11. 11 Health-checks • Business HC can be non-trivial • Multiple HCs on a service • Will all technologies have the support for running them … remotely? • Multiply sources of checks = lack of consistency ► Consul as a state reference (>=1.4)
- 12. 12 Health-checks • Business HC can be non-trivial • Multiple HCs on a service • Will all technologies have the support for running them … remotely? • Multiply sources of checks = lack of consistency ► Consul as a state reference (>=1.4)
- 13. 13 Health-checks • Business HC can be non-trivial • Multiple HCs on a service • Will all technologies have the support for running them … remotely? • Multiply sources of checks = lack of consistency ► Consul as a state reference (>=1.4)
- 14. 14 Here comes a Control-plane • Abstract internal dependencies • Resource reservation logic • External system provisioning • Give some controls to admins • Consume events, produce events • Device Provisioners: • One contract • N implementations (vendor/technologies)
- 15. 15 Here comes a Control-plane • Abstract internal dependencies • Resource reservation logic • External system provisioning • Give some controls to admins • Consume events, produce events • Device Provisioners: • One contract • N implementations (vendor/technologies)
- 16. 16
- 17. 17 User experience: metrology • Get network things done, within seconds. • Also, with great powers (…) • Diagnostic API endpoints provide errors cause • Subscribe to alerts on bad service health • Retrieve KPIs and logs autonomously
- 18. 18 User experience: Zero-config LB • Add a Consul http tag • Get a free LB service configured with sane defaults in seconds • Controlled namespace • Private visibility • TLS enforced with redirects • Self-diagnostics on errors
- 19. 19 User experience: Zero-config LB • Users are using it extensively • Remember East-West without LB? • Rate limit! • timeout tarpit 2s
- 20. 20
- 21. 21 Agility at scale • Ability to transparently switch LB vendor • Safe and progressive rollout • 50K servers • Between 50 and 100 LBs per datacenter • Easy and frequent platform upgrade • HAProxy deployments, several times a week • Transparent Linux kernel upgrades • > 530 deployments in less than two years(!)
- 22. 22 Incidents at scale • Doubling maxconn, what could go wrong? • Silent failing changes are not welcome • strict-limits, introduced in v2.1
- 23. 23 Probing the general service • End to end probing is providing fast feedbacks • Triggered many regressions and bugs • .. despite the simplicity of checks
- 24. 24 Log everything • http-response set-log-level silent if {rand(100) ge 1} • log 127.0.0.1 format rfc5424 local0 info • log-format-sd
- 25. 25 Observability example • TLS: %sslv - %sslc
- 26. 26
- 27. 27 Load balancing disaggregation • GeoDNS > L3 > L7 • “Hyper-converged” LB
- 28. 28 Load balancing disaggregation • GeoDNS > L3 > L4 > L7 • Support layer=L4
- 29. 29 Load balancing disaggregation • GeoDNS > L3 > L4 > L7 • Let L7 LBs become client of this • Translates into “L7 LB, on behalf of VIP X, asks for a L4 LB service” • L4 > L7 specifics • DSR: invest for ingress only ($) • Consistent hashing
- 30. 30 Load balancing disaggregation • GeoDNS > L3 > L4 > L7 • Let L7 LBs become client of this • Translates into “L7 LB, on behalf of VIP X, asks for a L4 LB service” • L4 > L7 specifics • DSR: invest for ingress only ($) • Consistent hashing
- 31. 31 Load balancing disaggregation • GeoDNS > L3 > L4 > L7 • Let’s redo this with layer=L3 • Translates into “L4 LB, on behalf of VIP X, asks for a BGP ECMP configuration” • Identify switch to peer with, configure it • L3 > L4 specifics • BGP ECMP => placement constraints • DDoS
- 32. 32 Load balancing disaggregation • GeoDNS > L3 > L4 > L7
- 33. 33 Load balancing disaggregation • Moved from Hyper-converged devices to commodity hardware • Consequences: • Cost efficient • LBs = plain servers in the CLOS matrix!
- 34. 34 Edge PoPs • GeoDNS > Edge PoP > L3 > L4 > L7 • Terminate TLS early = huge latency win • pool-purge-delay (HAProxy >= v2.0)
- 35. 35 Edge PoPs • GeoDNS > Edge PoP > L3 > L4 > L7 • Reach large population base, keep costs under control • Control plane role • Configure Geo DNS zones to closest PoP • Call third-party API • Configure HAProxy accordingly
- 36. 36 Edge PoPs • GeoDNS > Edge PoP > L3 > L4 > L7 • Reach large population base, keep costs under control • Control plane role • Configure Geo DNS zones to closest PoP • Call third-party API • Configure HAProxy accordingly
- 37. 37 GeoDNS on steroids • Adapt routing based on performance metrics, aka. traffic engineering • “Someone told me these subnets perform better using another AS path” • Build our own GeoDNS dataset? • Increase latencies by directing user bases to closest location with more accuracy • A decision-making tool? • Data-center location and topology assessment
- 38. 38
- 39. 39 Feedback • Loading more objects at runtime? • Several changes per second! • Never flush stats (at least some counters)
- 40. 40 Feedback • Overall stability • very high traffic pressure • Latency sensitive • Loosing requests during reloads is not acceptable! • Small memory footprint • TLS certificates at scale like a public hosting service • Share certificates between bind (in v2.1)
- 41. 41 Feedback • Overall stability • Small memory footprint • Community + enterprise support • "I'm investigating an issue we have with some fetch we're adding on a custom tcp_info structure, and I'm wondering if I might have discovered a broader issue[...]“ • - • Deployed worldwide two hours after
- 42. Thank you!