Hadoop & devOps : better together
Download as PPTX, PDF4 likes1,454 views
The document focuses on the integration of DevOps practices with Hadoop to automate and enhance deployment processes in big data environments. It highlights the importance of continuous integration, testing, and quality management through the use of various tools and configurations, facilitating collaboration between development and operations teams. Additionally, it covers aspects of protecting enterprise assets, easing Hadoop migration, and empowering both administrators and developers with better feedback and monitoring capabilities.
Hadoop & devOps : better together
- 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hadoop & DevOps : better together Maxime Lanciaux DevOps System Architect February 2017
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Automate & Speed up deployment
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automate & Speed up deployment Continuous integration workflow Dev + Ops Set of practices that emphasize the collaboration and communication of both software developers and information technology professionals while automating the process of software delivery and infrastructure changes. Automate deployment with tools to ensure quality
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automate & Speed up deployment Continuous integration + Build machine – Jenkins + Maven Configuration management tool (Hadoop + non Hadoop tools) – Ambari + Ansible Configuration management (configuration, projects, Jenkins pipeline) – Git Big Data Platform + Database + reporting – HDP + PostgreSQL + Grafana + Zeppelin Others – Custom log4J config, bash template, custom python scripts Typical set of tool You may want to use virtualization, containerization, continuous inspection, code collaboration tool as well
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automate & Speed up deployment Typical Architecture DN DN DN MN MN Database Reporting Edge Node Edge Node Configuration Management (Tag / Branches) Build Automation Continuous integration (Parameter, template) Configuration and Management platform + data Configuration / workflow / tests
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automate & Speed up deployment Typical Architecture Database Reporting Configuration Management Build Automation Continuous integration Configuration and Management platform Dev + DR Prod + Validation +
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automate & Speed up deployment Typical Organization HDFS directory for Dev, Validation, Production, Backup Hive databases and Kafka topics with naming convention HBase dedicated namespace YARN Capacity Scheduler with corresponding queue (Dev, Validation, Production) Ranger rules (Project, team, environment with groups) All automatically created during init of project (On Dev by user, on Prod by admin)
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automate & Speed up deployment On the Git side Codify everything Version everything Organize infrastructure as code
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Automate & Speed up deployment Enforce testing and quality
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enforce testing and quality Use of multi-tenant (1+n) Hadoop clusters – Which means naming convention and parameter / security Environments / Context
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enforce testing and quality Use of multi-tenant Hadoop cluster – Which means naming convention and parameter / security Environments / Context Dev Prex Prod Dev Dev
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enforce testing and quality Use of multi-tenant Hadoop cluster – Which means naming convention and parameter / security Environments / Context
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enforce testing and quality Use of multi-tenant Hadoop cluster – Which means naming convention and parameter / security Environments / Context
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enforce testing and quality Hadoop logs Monitoring of full stacks component Application logs Evolution of performance Code review with or without tools Test are include in the bundle (for non-reg) Massive use of Pareto
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Automate & Speed up deployment Enforce testing and quality Protect your enterprise asset
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Environments Configuration management Automation Impact analysis 5W Protect your enterprise asset Protect your enterprise asset Your company TMA #1 TMA #2 TMA #n DevOps
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Automate & Speed up deployment Enforce testing and quality Protect your enterprise asset Ease Hadoop migration
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Install, configure and deploy All test to be schedule / run Follow performance evolution Can be as easy as configuring access to a new edge node + configuration / password / SSH / certificates / security Multi-component environments Ease Hadoop migration A lovely hand
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Automate & Speed up deployment Enforce testing and quality Protect your enterprise asset Ease Hadoop migration Empower Administrator and Developer
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Developer to get a better feedback on their application Administrator to check / push back best practices Numerous action to be completely automated (ssh) Focus more high level task, with direct access to information Get directly access to template (no need to reinvent the wheels) Empower Administrator and Developer Same tools, same discussion
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Automate & Speed up deployment Enforce testing and quality Protect your enterprise asset Ease Hadoop migration Empower Administrator and Developer With full monitoring for Human
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved With full monitoring for human Per project, per environment, per Hadoop cluster or on all of them
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved With full monitoring for human Per project, per environment, per Hadoop cluster or on all of them
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved With full monitoring for human Per project, per environment, per Hadoop cluster or on all of them
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Automate & Speed up deployment Enforce testing and quality Protect your enterprise asset Ease Hadoop migration Empower Administrator and Developer With full monitoring for Human Demo
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved We are Recruiting ! Email me : [email protected]
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
Editor's Notes
- #29: Merci et question