Continuous integration with business intelligence and analytics
Download as ODP, PDF5 likes2,326 views
The document discusses continuous integration for business intelligence and analytics projects. It begins with an introduction of the speaker and overview of agile vs waterfall approaches. It then details the key aspects of continuous integration such as releasing early and often, bundling work into testable builds, and using an approval process. Challenges for applying these principles to BI projects are also outlined due to the disparate tools used and difficulties breaking work into independent chunks and testing components. The presentation concludes with discussing applying a regular software release process to BI through planning, coding, building, testing, releasing, and monitoring stages.
Continuous integration with business intelligence and analytics
- 1. 1 Continuous Integration For Business Intelligence And Analytics Alex Meadows Principal Consultant CSpring
- 2. 2 Agenda ● About Me ● Waterfall vs Agile ● Continuous Integration ● Software Release Process ● References Feel free to ask questions at any point in the presentation!
- 3. 3 Alex Meadows ● Worked in various industries (manufacturing, social/email marketing, consulting, software, agriculture) ● Open source agilest ● Loves data :)
- 4. 4 Let's Talk Releases... ●What's your team's release candance? ●How many bug releases go out afterwards? ●How much new development gets cut?
- 7. 7 Continuous Integration ● Release Early, Release Often! ● Bundle work together into cohesive “builds” – Server Configuration – Code – Database – Tests, tests, and more tests – Approval Process
- 8. 8 Continuous Integration ● Agile XP principal since 1994 ● Used in software everywhere* – Google ● Tests 100 million test cases every day ● Entire code base changes 50% monthly – Amazon, Yahoo, Microsoft, etc. * Used in mature agile software shops
- 9. 9 Software Release Process Source: http://www.360logica.com/blog/2014/06/continuous-integration.html
- 10. 10 Plan ● Break tasks out into manageable chunks – Can that ETL code be broken down by: ● Source/Target ● Table ● Process – Can that dashboard be broken down by: ● Object ● Type of data/visual/interaction ● Business must prioritize, team must commit
- 11. 11 BI Challenges ● Work is difficult to break apart – Most tools are designed for waterfall – Complex moving components
- 12. 12 Code ● Team works and develops “artifacts” – “Code” - Reports, ETL, Visualizations, Database changes, etc. – Documentation – Research
- 13. 13 Build ● Developed work is – Run through unit tests – Peer reviewed – “Compiled” - tested for readiness for QA – Pushed to QA
- 14. 14 BI Challenges ● Coordination across disparate tools, components ● Version control doesn't coordinate between tools ● Database version control is hard to introduce
- 15. 15 Test ● Verify all functionality, edge cases, and user requirements ● 10% manual, 90% automation work ● Automated gate keeper (pass/fail the build)
- 16. 16 BI Challenges ● Test tools not written for many components – Database – exists, but minimalist – ETL – database tools and etlTest (shameless plug ;) – Visualization – Selenium – Reports – mix of the above – Analytics – mix of the above
- 17. 17 Release/Deploy ● Coordinate and push to next environment ● Approvals act as gate keeping mechanisim ● Cycle starts back and the code phase as needed
- 18. 18 BI Challenges ● Coordination is possible, but challenging ● Tools might not provide integration points
- 19. 19 Operate/Monitor ● Release into Production ● Monitor processes, logs, etc for aberrations ● Bugs and other issues are added into backlog for future releases
- 20. 20 References