Securing the continuous integration
- 1. Securing the Continuous Integration Irene Michlin
- 2. People, process, and technology Step 1: make sure your CI does not harm your security Step 2: only then it can be used to improve your security People ProcessTechnology
- 4. Basic CI cycle Developer Development machine Version Control repo CI Server Write code + tests Pull latest code + tests Local build + test run Commit Poll Generate feedback Feedback Dashboard or gadget “Continuous Integration assumes a high degree of tests which are automated into the software”, Martin Fowler
- 5. Isolate your environment • Phishing link in email => keylogger installed => source code gone (or backdoor deployed) • Experimenting with development network => accounting department affected before EOY • Extra challenges: remote work or BYOD
- 6. Basic CI cycle Developer Development machine Version Control repo CI Server Write code + tests Pull latest code + tests Local build + test run Commit Poll Generate feedback Feedback Dashboard or gadget
- 7. Version control server • It has one job only – remove or disable everything else • No shared or generic accounts • Matching business process to close accounts
- 8. Basic CI cycle Developer Development machine Version Control repo CI Server Write code + tests Pull latest code + tests Local build + test run Commit Poll Generate feedback Feedback Dashboard or gadget
- 9. Integration build server • Who is responsible for keeping it up to date? • Where do external components come from? • Check vendor advice on compiler and linker options On September 18, 2014, RiskIQ detected credential-stealing malware being loaded onto users’ computers through a drive-by download at jQuery.com. Not too long ago, news broke about a malicious version of Xcode dubbed XcodeGhost that had been surreptitiously injecting malicious code into iOS developers’ apps. SK-CSIRT identified malicious software libraries in the official Python package repository, PyPI, posing as well known libraries. A prominent example is a fake package urllib-1.21.1.tar.gz, based upon a well known package urllib3-1.21.1.tar.gz.
- 10. Basic CI cycle Developer Development machine Version Control repo CI Server Write code + tests Pull latest code + tests Local build + test run Commit Poll Generate feedback Feedback Dashboard or gadget
- 11. Feedback mechanism • IoT electronic toys are notoriously insecure • Custom integration scripts - are you cutting corners?
- 12. Do no harm • Do not acquire CI components « by accident » • Not everything is secure out of the box • Dormant account today is an attacker-controlled account tomorrow - Compromising a staging continuous integration server (Jenkins) from a developer's machine, which then gave us access to the company's production Chef repository, allowing us to push a reverse shell to a production server. - Using access to a dev/staging Jenkins server to modify code in repositories after they've been through code review, which were then replicated to production. https://www.blackhat.com/us-17/briefings.html#developing-trust-and-gitting-betrayed
- 13. Treat people right • Don’t leave new starters to figure out CI rules on their own • Controls are safety net, not surveillance • Anyone on security team? • Help devs to automate “boring” things • Learn to say “yes” • Security goals must support business goals • Anyone in management? • Don’t abuse metrics
- 14. Stop blaming developers • Technology • People • Process
- 16. Code reviews • No change too small • Leave trivial checks to tools • Not a separate task, but in DoD for each task • Reject & rework is part of “normal”
- 17. Root-cause analysis • What happens to externally reported issues? • The first security feedback to introduce • What was missing in our CI process? => Improve
- 18. Chain of custody • Can you trust your release notes? • Has every “unit of work” in the release gone through all the checks? • Was it modified since “time of check”?
- 19. Reference implementations • Give dev teams libraries / tools that are secure by default • Leverage frameworks (e.g. output encoding) • Build your own • Potential areas to consider: • Managing secrets • Anything related to crypto • Authentication / authorization • SQL, file system access, shell exec() Core takeaway: • “<X> is hard to do securely, have to be aware of threats 1, 2, and …” • Build a secure by default implementation • Automate the tests
- 20. Static analysis
- 21. If you didn’t laugh at the previous one… • Static analysis – reason about code based on looking at it • Dynamic analysis – run code and observe how it behaves • Fundamental trade-offs: • Static analysis – high coverage, imprecise (false positives) • Dynamic analysis – code coverage is a challenge, but precise (the issues it finds are true positives)
- 22. Pick your poison Grep • Localized • Not syntax-aware • Regex Lint/AST • Syntax aware • Some semantics • Can express rules that use code constructs SAST • Builds detailed model of code • Complex rules that rely on control and/or data flow
- 23. Static analysis • Run on dev machine needs to be fast, low to no false positives • Ideally – integrated with IDE • Run in CI needs to provide fast feedback • Can be implemented with git hooks or as a part of the test suite • Integration with version control is useful • Integration with bug tracker is useful (but…) • Ability to enable analysis by defect type is very useful
- 25. “On commit” is your best friend • Automated coding standards checks • Code complexity / code duplication • Banned functions / APIs • Dynamic analysis • Static analysis • Fuzzing • https://www.nccgroup.trust/uk/our-research/securing-the-continuous-integration-process/ • https://www.nccgroup.trust/uk/our-research/best-practices-for-the-use-of-static-code-analysis- within-a-real-world-secure-development-lifecycle/
- 26. Points of contact Irene Michlin Principal Security Consultant M: +44 (0) 7972 333 148 E: [email protected] T: @IreneMichlin Thanks and credit to Clint Gibler - @clintgibler and Antti Vähä-Sipilä - @@anttivs https://www.nccgroup.trust/uk/our-research/securing-the-continuous-integration- process/ https://www.nccgroup.trust/uk/our-research/best-practices-for-the-use-of-static-code- analysis-within-a-real-world-secure-development-lifecycle/
Editor's Notes
- #3: On people, I only have one piece of advice - Hire the right people and treat them right. The rest of the talk is on process and technology.
- #4: The choice is enormous, not going to say “This is the recommended toolchain”. Not on the agenda, not on the hidden agenda. It’s rather what to look for in my tools, and how to put them together so that first of all I’m not harming my organisation’s security. And the secondary goal – I have this neat CI pipeline going, can I use it to improve my software’s security, amongst other things.
- #5: To ensure we are on the same page for the rest of the talk.
- #6: What happens in dev, stays in dev Physical or virtual, dev machines must be segregated from the main corporate environment.
- #7: Let’s talk about version control server
- #8: The choice of the version control pattern depends on multiple factors, such as frequency of releases, number of teams, structure of the code base etc. Generally speaking, patterns that require frequent commits to a shared branch are more suited for implementing CI. SSO is better than proliferation of accounts per each tool. RBAC with as much granularity as does not interfere with your work. Needs administrative component to terminate the accounts as required.
- #9: Integration server
- #10: It only needs R/O access to source control Set local repo for external libraries. Use scanners. Standard builds remove chasing the “it worked on my machine” problems
- #11: Feedback mechanism
- #12: Who is listening? Unless it is a low-tech mechanism “All the developers are in the same room and they look at a physical dashboard when they feel like it”, you have to consider the security of the mechanism. The tools are held together with what is often called “glue skills” – the developers are putting together ad hoc scripts, punching holes in firewalls, hardcode passwords. Even if the script itself isn’t dangerous, if it’s world-writable but runs at high privilege, it has a high potential to become dangerous. Glue skills are great to have in your team, but they can also be dangerous. Treat these scripts as first class citizens – under source control and under the same security standard as everything else. My example is not an attack on a particular tool, literally I’ve searched for “broken build gadgets” and picked a first example.
- #13: CI environments are rarely planned upfront. The tools are introduced one by one and sometimes people only learn about everything they have during an audit, which is a bit too late. Secure system lifecycle. The product choice and evaluation can be quick and informal, but it has to be a conscious choice. Installation and configuration – don’t enable features you don’t need Ensure account management lifecycle covers all the CI tools.
- #14: Luckily the industry is moving on from the “people are the weakest link” concept.
- #15: Luckily the industry is moving on from the “people are the weakest link” concept. ©Giuseppe Colarusso http://www.gianlucagimini.it/prototypes/velocipedia.html Via AREyouKIDDINGmi And now we are to level 2 of this game: how to actually do good in security terms using CI.
- #16: The figure above is based on several popular maturity matrices for continuous integration, as well as NCC Group experience with variety of clients. Won’t have time to go through them all, only couple. There is now understanding in the software development industry that security features are not enough for building a secure product. To achieve an acceptable level of overall security, the product needs all the features to be secure: the whole attack surface of the product must be considered from a security point of view, not only features that are directly security related, such as login functionality. The same is true for securing continuous integration. In the matrix above, the dedicated Security row lists explicit security related activities that must happen at some point towards CI maturity. However, every cell on the other rows needs to be evaluated for the security of its tools and processes as well.
- #17: Frequently asked “are manual code reviews waste of time”? Not if done well. If your decision has only one possible outcome, then why bother with the decision? It’s waste of time. Another source of waste of time in review is “saving up” to end of week/end of iteration. It’s all about quick feedback.
- #18: People like to rush into bug bounty programs Don't trouble trouble until trouble troubles you
- #19: Whether you release each change individually in CD, or in batches, the same questions apply
- #20: I’ve seen a company with a checklist for each api function – authentication, authorisation, input, output. Makes code reivew expensive, boring, and error prone. They’ve moved all the “boilerplate” to be given to developers for free. Test case that automatically checks for missing access controls and goes through the checklists
- #21: Everybody’s favourite, right? https://78.media.tumblr.com/tumblr_m09u09jyo31qzrlhgo4_250.gif
- #22: Either you love dogs very much, or you haven’t come across static analysis
- #23: There is a range of complexity in “light-weight” vs “heavy-weight” static analysis approaches From simple text matching to understanding code syntax, semantics, and can reason about data and control flow Each of these have strengths and contexts in which they are useful
- #25: The figure above is based on several popular maturity matrices for continuous integration, as well as NCC Group experience with variety of clients. Won’t have time to go through them all, only couple. There is now understanding in the software development industry that security features are not enough for building a secure product. To achieve an acceptable level of overall security, the product needs all the features to be secure: the whole attack surface of the product must be considered from a security point of view, not only features that are directly security related, such as login functionality. The same is true for securing continuous integration. In the matrix above, the dedicated Security row lists explicit security related activities that must happen at some point towards CI maturity. However, every cell on the other rows needs to be evaluated for the security of its tools and processes as well.
- #26: Don’t throw all the tools at the dev team at the same time. Fuzzers, static analysis, dynamic analysis all take time to learn and configure and initially will bring the productivity down, not up Start with “new” code, ensure your debt isn’t growing. No two journeys towards maturity are the same. It may make sense for your organisation to take a practice from Intermediate or Advanced before doing everything in Beginner. It meant to inform, not to restrict. Hard to come with good questions on the spot, so I won’t force you. I’m here today, obviously, and if you have questions later, find me on twitter or by email.