Hadoop Summit 2013 : Continuous Integration on top of hadoop
4 likes3,655 views
Wisely Chen and Neal Lee presented on continuous integration on Hadoop. They discussed automating the software development process, including automatically running unit tests on code commits, performing tests on staging environments that mimic production, and deploying to production. They demonstrated tools for unit testing Hadoop jobs, including PigUnit and Vaidya for performance diagnosis, and how these tools can integrate with a continuous integration system to automatically run tests with each code change.
![Who I am
• Wisely Chen ( thegiive@gmail.com )
• Release manager of Yahoo![Taiwan] shopping and data team
• Love to promote open source tech at Taiwan
• Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007
• Puppet : PHPConf 2012 , RubyConf 2012
• Release Practice :Webconf 2013, Coscup 2012
Tuesday, June 11, 13](https://image.slidesharecdn.com/chenjune271100amroom210c-130715203801-phpapp01/85/Hadoop-Summit-2013-Continuous-Integration-on-top-of-hadoop-3-320.jpg)
![Who I am
• Neal Lee (@neal_lee)
• Data Engineer at Yahoo![Taiwan]
• Aiming to build up an easy use of self-service BI
platform connecting to Hadoop.
Tuesday, June 11, 13](https://image.slidesharecdn.com/chenjune271100amroom210c-130715203801-phpapp01/85/Hadoop-Summit-2013-Continuous-Integration-on-top-of-hadoop-4-320.jpg)
Hadoop Summit 2013 : Continuous Integration on top of hadoop
- 1. Continuous Integration on top of hadoop Wisely Chen & Neal Lee Tuesday, June 11, 13
- 2. Agenda • Who I am • Problem • Solution • Demo • Q&A Tuesday, June 11, 13
- 3. Who I am • Wisely Chen ( [email protected] ) • Release manager of Yahoo![Taiwan] shopping and data team • Love to promote open source tech at Taiwan • Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007 • Puppet : PHPConf 2012 , RubyConf 2012 • Release Practice :Webconf 2013, Coscup 2012 Tuesday, June 11, 13
- 4. Who I am • Neal Lee (@neal_lee) • Data Engineer at Yahoo![Taiwan] • Aiming to build up an easy use of self-service BI platform connecting to Hadoop. Tuesday, June 11, 13
- 5. Story 1 Tuesday, June 11, 13
- 6. Another Story Tuesday, June 11, 13
- 7. Yet Another Story Tuesday, June 11, 13
- 9. One click • Manual commit code to SCM • And DONE • Auto unit testing • Auto push beta for performance testing • Auto push to production grid • Auto trigger code Tuesday, June 11, 13
- 10. This feeling is 爽! Tuesday, June 11, 13
- 11. Continuous Integration Tuesday, June 11, 13
- 12. Continuous Integration • A software engineering practice • Maintain code repos • Automate the build • Make the build self-testing • Everyone commit to the baseline everyday • Every commit should be a build • Test in a clone of production environment • Make it easy to get the latest deliverables • Everyone can see the result of latest build • Automate deployment Tuesday, June 11, 13
- 13. We focus on • A software engineering practice • Maintain code repos • Automate the build • Make the build self-testing • Everyone commit to the baseline everyday • Every commit should be a build • Test in a clone of production environment • Make it easy to get the latest deliverables • Everyone can see the result of latest build • Automate deployment Tuesday, June 11, 13
- 14. CI flow 4. CI slave exec local UnitTest 7. CI slave exec Performanc 11. CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 10. Call 9. git tag 12. notify user Tuesday, June 11, 13
- 15. CI flow 4. CI slave exec local UnitTest CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI CI Master 1. Commit Code SCM 3. Call 5. Notify user Tuesday, June 11, 13
- 16. CI flow 4. CI slave exec local UnitTest 7. CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 8.Notify user Tuesday, June 11, 13
- 17. CI flow 4. CI slave exec local UnitTest 7. CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 9. Notify user Tuesday, June 11, 13
- 18. Unit Test 4. CI slave exec local UnitTest 7. CI slave exec Performanc 11. CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 10. Call 9. git tag 12. notify user Tuesday, June 11, 13
- 19. PigUnit • A simple xUnit framework • No cluster set up is required in local mode • Unit testing, regression testing, and rapid prototyping on the fly Tuesday, June 11, 13
- 20. Using PigUnit • Coding • Write PigUnit test case • Run local PigUnit test • Push to grid • Run Pig on grid • Get right result ! Tuesday, June 11, 13
- 21. Unit test is live doc • Unit test is runnable live doc • Pass test case and meet previous requirement Tuesday, June 11, 13
- 22. Performance Test 4. CI slave exec local UnitTest 7. CI slave exec Performanc 11. CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 10. Call 9. git tag 12. notify user Tuesday, June 11, 13
- 23. Vaidya • Rule based performance diagnosis of M/R jobs • Extensible framework • You can add your own rules • Write complex rules using existing rules Tuesday, June 11, 13
- 24. CI toolset CI slave exec local UnitTest CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid notify CI deploy deploy CI Commit Code SCM Call Vaidya BASH Tuesday, June 11, 13
- 25. CI is flexible • MapReduce can use MapUnit • Hive can use hive_test • Pig can use PigUnit Tuesday, June 11, 13
- 26. Github trigger CI Tuesday, June 11, 13
- 27. CI testing build pipeline Tuesday, June 11, 13
- 28. Testing Trend Tuesday, June 11, 13
- 29. DEMO Tuesday, June 11, 13
- 30. Conclusion • Auto testing will save your life • CI will boost your productivity • This process can feed in any platform Tuesday, June 11, 13
- 31. 謝謝大家 Tuesday, June 11, 13