DevOps in an Embedded World
3 likes1,048 views
The document discusses the challenges of implementing DevOps practices in embedded software development, emphasizing the importance of continuous integration and delivery (CI/CD) for improving software release efficiency and defect resolution. It shares a case study from Navico, illustrating the impact of legacy architecture on product growth and the need for better tools and automation in testing processes. The author highlights lessons learned, including the need for cultural change, executive support, and the significance of automation in testing and software design.
DevOps in an Embedded World
- 1. DevOps in an Embedded World
- 3. Some of the products
- 4. Challenges in Embedded Software Development Hardware! Resource Constrained Deal with the performance implications Debugging Learn to debug and deal with - OS bugs - Hardware bug - UFO bugs Oscilloscope and JTAG is your friend Environment - Thermal - Moisture - Power consumption
- 9. Does CI/CD make sense for Embedded Software
- 10. The Principles High Frequency Increase the ability to release software to customers faster Reduce Overheads Reduce the work required to release features & products to the market Improve Defect Resolution Enable finding defects faster and as close to the developer as possible Automate Automate all inefficient manual tasks Reduce Response Times Enable developers to respond to defects faster
- 11. A Case Study
- 16. Navico R&D Centres our office
- 17. Our Challenges
- 18. 5,241,912Lines of Code 61Products and counting …
- 19. 12 12 12 19 19 23 23 28 32 48 50 61 48 49 50 51 52 53 54 55 56 57 58 59 Software Release Products Supported
- 20. “ Our developers never release code. Rather, it tends to escape, pillaging the countryside all around.
- 21. Complex Networks
- 22. Architecture ◍ Old Codebase ◍ Backwards compatibility with older hardware ◍ Continuing to support advancements in new hardware ◍ Desire to provide value to customers (new features on all products) Legacy Architecture
- 23. All of this meant ◍ Defects were found late ◍ Software stabilisation was taking longer ◍ Manual testing couldn’t keep up with product growth ◍ Costs were increasing (including Opportunity Costs) ◍ We were impacting our ability to innovate Time Cost of Finding and Resolving Defects
- 25. Does CI/CD provide Answers? High Frequency ◍ Reduce product release timeframes ◍ Get new features to customers faster Reduce Overheads ◍ Become more efficient ◍ Reduce costs Improve Defect Resolution ◍ Higher quality software ◍ Better brand perception ◍ More sales! Automate ◍ Become more efficient ◍ Faster turnaround times Reduce Response Times ◍ Prevent bugs on top of bugs ◍ Higher quality software
- 26. “ Continuous Delivery Sounds Great, But It Won’t Work Here
- 28. Our Plan Improve Tools & Architecture Seek integration and flexibility over compliance and process Improve Build Speeds Everything is depends on faster builds so fix this. Quality Control Incoming Code Validate the quality of incoming code automatically Automated Testing Manual testing can’t keep up, focus on automating as much as possible
- 29. “You can have data without information, but you cannot have information without data - Daniel Keys Moran
- 30. Previous VCS ◍ Proprietary tool with limited support or updates ◍ Extremely limited integration with other tools ◍ Lack of CI support ◍ No Code Review capability Improve Tools Bitbucket/Git ◍ Git! ◍ Integrated Code Reviews ◍ Significantly better integration with CI ◍ Integration fully supported with all tools
- 31. Improve Architecture ◍ Configurable codebase ◍ Fairly common HW platform ◍ Runtime feature flags
- 32. Improve Architecture ◍ Dependency matrix
- 34. Improve Architecture ◍ Dependency matrix ◍ Remove Cyclic Dependencies ◍ Eliminate Singletons ◍ Promote unit testability
- 35. Improve Branching ◍Many Branches – Long Lived – Mega Merges Testing Testing Testing Testing Testing Testing
- 36. Improve Branching ◍ Smaller Batch Sizes ◍ Easier to Merge/Test/Review ◍ Validation focused on a single branch ◍ Feature Flags Testing
- 38. 8 Hours For a Linux Platform Build 14+ Hours To build and package a release 2 Hours For each application build
- 40. Tools to look into ◍ ccache/clcache – for improving C++ compilation times ◍ IncrediBuild – faster parallel builds (tight integration in Visual Studio) ◍ distcc – Open Source (free) alternative to IncrediBuild
- 41. 0 1 2 3 4 5 6 Average Time To Build (Hours)
- 42. Test Automation
- 46. So how did we approach this? ◍ Start with micro tests - Unit Testing (Simulator) - Unit Testing (Real Hardware) ◍ Build Tools - Software Tools (N2K Simulator, Remote Control) - Hardware Tools (Repurpose/Build)
- 47. UI Test Automation ◍ Build or Buy ◍ Functional Testing vs API Testing ◍ Utilise HW Test Tools ◍ Execute on Real HW as well as Simulators ◍ BAT vs Full Regression
- 48. CI Pipeline ◍ Code Commit -> Pull Request ◍ Automated Build / Unit Tests (on HW) ◍ Merge to Master ◍ Daily Integration Builds on Master for - All HW platforms - All Application Variations ◍ Ready for QA.
- 49. What’s Next? ◍ Configuration as Code ◍ Code Quality Tools ◍ Simulate More Hardware ◍ Increase Analytics and Reporting ◍ Fully Simulated Test Environments for Dev ◍ Scale – From internal infrastructure to the Cloud ◍ Grow the team (We need you)
- 50. CI Pipeline ◍ Code Commit -> Pull Request ◍ Automated Build / Unit Tests/ Functional UI Tests ◍ Automated Architecture / Code Formatting Checks (SonarCube/clang) ◍ Merge to Master ◍ Daily Integration Builds on Master for - All HW platforms - All Application Variations ◍ Full Automated UI Test Coverage ◍ Ready for release.
- 51. Lessons Learnt ◍ Culture! ◍ Collect Data ◍ Get Executive Buy-in ◍ Change your Tools and Processes if needed ◍ Test Automation is key - Invest in HW - Simulate! - Virtualise ◍ Focus on Good Software Design for Everything