Downloaded 43 times
![Embrace Complete Continuous Deployment
End-to-End Continuous Deployment
• Configuration Management (Chef),
• Standardized Environment Images (Packer)
• Environment/Subsystem Orchestration Layer
• Production-like environments in Dev & Test
must include all components
Results
• Hardened base images become part of the
standard development process
• Continuous validation of secure configuration
• Holistic system validation & testing
#Sample Alert Logic Chef NodeDef
{
"name": "cloud-api-node",
"versions": {
"1.6.0": {
"vm_type": "squeeze64",
],
"install_phase": {
"run_list": [
"app-version-install@4.1.0",
]
},
….
Leverage Configuration
Management
Leverage Standardized
Environment Images
Build an Orchestration Layer
#Sample Packer Configuration
{
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access`}}"
,
"secret_key": "{{user `aws_secret`}}"
,
"region": "us-east-1",
"source_ami": "ami-de0d9eb7",
"instance_type": "t1.micro",
"ssh_username": "ubuntu",
"ami_name": "packer-ex {{timestamp}}"
}]](https://image.slidesharecdn.com/zjomtjsscrqcsfcwzgea-signature-866a95b932ab6b8bcf2016eeaf1f8c42b8cf1ca42309f6e54e601658daa49cca-poli-150306174329-conversion-gate01/75/Overcoming-Security-Challenges-in-DevOps-27-2048.jpg)
Uploaded byAlert Logic
Overcoming Security Challenges in DevOps
This document discusses taking a DevOps approach to security. It outlines how DevOps practices like automation, immutable infrastructure, and infrastructure as code can improve an organization's security posture by reducing vulnerabilities and ensuring consistent configurations. It also addresses some of the challenges of integrating security into DevOps environments and proposes moving towards software-defined security models that provide real-time visibility, automatic protection, and continuous assessment.
More Related Content
Similar to Overcoming Security Challenges in DevOps
![[muCon2017]DevSecOps: How to Continuously Integrate Security into DevOps](https://cdn.slidesharecdn.com/ss_thumbnails/mucon2017devsecopshowtocontinuouslyintegratesecurityintodevops-171106193735-thumbnail.jpg?width=640&height=640&fit=bounds)
![Governance, Deployment & Methodologies for Agentic Automation [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day2-251112135038-a386476d-thumbnail.jpg?width=640&height=640&fit=bounds)
Overcoming Security Challenges in DevOps
- 1. Taking a DevOpsApproach to Security Alert Logic & Chef discuss overcoming security challenges in DevOps Yvo Van Doorn Manager, Solutions Architects Chef Software Justin Criswell Cloud Solutions Architect Alert Logic
- 2. Speaker Introduction • YvoVan Doorn • Manager, Solutions Architects • Chef Software, Inc. • @yvov • www.linkedin.com/in/yvovandoorn • Justin Criswell • Cloud Solutions Architect • Alert Logic, Inc. • @JustinCriswell5 • www.linkedin.com/in/justincriswell
- 3. Session Overview More organizationsare embracing DevOps to realize compelling business benefits such as faster yet safer feature release cadence, increased application stability, and rapid response to shifting market conditions. However, security and compliance monitoring tools have not kept up and often represent the single largest remaining hurdle to Continuous Delivery. Topics covered in this session: •How DevOps Improves your Security Posture •Overcoming Challenges in DevOps Environments
- 4. DevOps Improves SecurityPosture
- 5. Mark Burgess, Fatherof Configuration Management Author of “In Search of Certainty” Configuration Management “We suffer sometimes from the hubris of believing that control is a matter of applying sufficient force, or a sufficiently detailed set of instructions.”
- 6. Automation and ConvergentInfrastructure “A system's desired configuration state can be said to be defined by fixed points. Most configuration management systems (e.g.: CFEngine, Chef, Puppet, PowerShell DSC) are based on this idea: they provide means to declare what must happen instead of requiring imperative workflows that prescribe what to do.” Mark Burgess, Father of Configuration Management Author of “In Search of Certainty”
- 7. The Birth ofDevOps
- 8. Driving toward ImmutableInfrastructure "This is what I call disposable computing. Throw away a broken process rather than trying to fix it. Machines can be made expendable as long as the total software is designed for it. Not much of it is today, but we´re getting there. Nature shows that this is a good way of scaling services." Mark Burgess, Father of Configuration Management Author of “In Search of Certainty”
- 9. Infrastructure as Code •Converge on a regular interval • Configuration Management is idempotent • All persistent changes must be in source control • Manual intervention discouraged • Out-of-band changes will be lost
- 10. Security & ComplianceImplications
- 11.
- 12. Test Driven Infrastructure CostsOf Finding Bugs in Production
- 13. Continuous Security Auditing & Compliance End-to-End Visibility Disaster Recovery& Business Continuity Remediation & Fast Resolution Continuous Detection/ Protection Automated Configuration & Scaling Security Posture
- 15.
- 16. Overcoming Security Challenges •Challenges for Security Technology & Practice Today - AWS Shared Responsibility Model - Challenges remain for Customers • Leveraging DevOps for Security - Best-Practices for blending DevOps with Security • Toward Software-Defined Security - Embracing new reality of AWS Cloud Infrastructure
- 17. AWS Shared ResponsibilityModel Customer Responsibility Foundation Services Hosts • Logical network segmentation • Perimeter security services • External DDoS, spoofing, and scanning prevented • Hardened hypervisor • System image library • Root access for customer • Access management • Patch management • Configuration hardening • Security monitoring • Log analysis Apps • Secure coding and best practices • Software and virtual patching • Configuration management • Access management • Application level attack monitoring • Network threat detection • Security monitoring Networks Cloud Service Provider Responsibility Compute Storage DB Network
- 18. 2014: Security TopCloud Pain Point Security Pricing/Budget/Cost Human Change Management Security of Data, Control of Data Locality, Sovereignty Compliance Migration/Integration Internal Resources/Expertise Management Lack of Internal Process Vendor/Provider Issues Organizational Challenges Contractual/Legal Issues Service Reliability/Availability Network Lack of Standards 4% 31% 17% 12% 11% 11% 10% 9% 8% 7.4% 7% 7% 7% 5% 5%
- 19. Application Security TechnologyChallenges Network Changes Host Identity Auto-Scaling Application Security Technology Challenges
- 20. Traditional Security/Compliance isSlow Mature DevOps Velocity is Fast Security Practice does not Keep Up Security at Odds with DevOps Velocity
- 21. InfoSec Ends UpBeing Marginalized “The problem for the security person who is used to turning around security reviews in a month or two weeks is they're just being shoved out of the game. There's no way with how Infosec is currently configured that they can keep up with that. So, Infosec gets all the complaints about being marginalized and getting in the way of doing what needs getting done.” Gene Kim, former CTO of Tripwire Author of “The Phoenix Project: A Novel About IT, DevOps” & “Helping Your Business Win”
- 22. Integrating Security withDevOps • Leveraging DevOps practice for better Security - Prevent attack vectors with Immutable Systems - Manage vulnerabilities with Base Images and Configuration Management - Robust Auditing and Centralized Log Collection - Adopt strategy of Phoenix Upgrades - Embrace end-to-end Continuous Deployment
- 23. Manage Vulnerabilities withBase Images Manage Vulnerabilities • Conduct normal vulnerability scanning • Identify Vulnerabilities that exist in Base Images versus Application specific packages • Remediate at appropriate level as part of Continuous Delivery process • Start with Hardened “secure by default” base Results • Less work, done more reliably • Patching fits naturally into Phoenix Upgrades • Continuous Delivery allow frequent scanning in test environments to have real value • Fixes potential vulnerabilities systematically
- 24. Prevent Attacks withImmutable Systems Build secure base images that are representative of your infrastructure system base Design file system layout to separate code from data, and lock down to minimum required permissions. Should expand to network as well Leverage SANS Checklist and CIS Benchmark resources for system level security best practices and guidance Leverage configuration management tools to standardized all software versions and configurations Design Secure Immutable Infrastructure
- 25. Centralize Robust Auditing& Logging # This file contains the auditctl rules that are loaded # whenever the audit daemon is started via the initscripts. # The rules are simply the parameters that would be passed # to auditctl. # First rule - delete all -D # Increase the buffers to survive stress events. # Make this bigger for busy systems -b 1024 -a exit,always -S unlink -S rmdir -a exit,always -S stime.* …… Implement Local Auditing #Sample syslog-ng configuration #Lots of configuration required ........ # Send *ALL* System Logs to Log Appliance destination df_log_appliance_forward { tcp("my-log-appliance" port(514)); }; log { source(s_all); destination(df_log_appliance_forward); }; Collect Important Logs Centralize Log Collection for Search and Filtering
- 26. Adopt Strategy ofPhoenix Upgrades Embrace Phoenix Upgrades • Stand up new instances, don’t upgrade • Route traffic between old and new instances • Rich service metrics and automate rollback • Advanced routing can enable selective rollout Results • Creates evergreen systems, avoiding configuration drift and technical debt • Enforces refresh of all system components as complete artifact, tested as a holistic system • Greatly reduces security risks when combine with immutable instances and configuration management
- 27. Embrace Complete ContinuousDeployment End-to-End Continuous Deployment • Configuration Management (Chef), • Standardized Environment Images (Packer) • Environment/Subsystem Orchestration Layer • Production-like environments in Dev & Test must include all components Results • Hardened base images become part of the standard development process • Continuous validation of secure configuration • Holistic system validation & testing #Sample Alert Logic Chef NodeDef { "name": "cloud-api-node", "versions": { "1.6.0": { "vm_type": "squeeze64", ], "install_phase": { "run_list": [ "[email protected]", ] }, …. Leverage Configuration Management Leverage Standardized Environment Images Build an Orchestration Layer #Sample Packer Configuration { "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access`}}" , "secret_key": "{{user `aws_secret`}}" , "region": "us-east-1", "source_ami": "ami-de0d9eb7", "instance_type": "t1.micro", "ssh_username": "ubuntu", "ami_name": "packer-ex {{timestamp}}" }]
- 28. Moving to SoftwareDefined Security • Significant Opportunity remains in front of us - Security remains a lot of work - Does not fit naturally into DevOps Roles • AWS ready to Accelerate Security Technology - Leverage end-to-end visibility available - Protect automatically with real-time configuration - Transform periodic assessment into real-time automated responses
- 29. Leverage End-to-End Visibility LeverageDetailed Visibility of AWS • AWS describe endpoints reveals environment • Cloud Trail shows incremental changes • Long-term AWS Config unifies this Information Results • Enables visualization of entire environment • Can reason directly about exploitability • View of environment always up-to-date • No need to coordinate application changes with security infrastructure
- 30. Protect with AutomaticConfiguration Automated Protection • Add security infrastructure on-demand to cover changes to environment • Autoscale that infrastructure based on traffic seen in that environment environment Results • Focus on application deployments and not configuration of security infrastructure • Implements least-privilege configurations • Seamless experience that “just works” with Security fitting naturally into DevOps practice
- 31. Transform Assessment tobe Real-Time Real-Time Protection • Assessment activities should become continuous • Continuous Assessment “just happens” as deployments occur • Zero-day coverage delivered automatically Results • No more periodic assessment cycle • Security posture of environment available immediately and automatically after deployments • Security matches DevOps Velocity!
- 32. Contact Us Justin Criswell CloudSolutions Architect Alert Logic [email protected] @JustinCriswell5 Yvo Van Doorn Manager, Solutions Architects Chef Software, Inc. [email protected] @yvov Thank you!
Editor's Notes
- #18 As recommended by AWS, this slide will contain a visual of the AWS shared security model, and define the scope of the security practices and technology that the rest of the presentation will address.
- #19 Customer see security as the top pain point for Cloud deployments. This is because as challenging as security is in traditional environments, the adoption of Public Cloud infrastructure and DevOps practices have moved faster than security technology and practice, even farther away from meaningful effectiveness.
- #20 Traditionally security technology have always been challenged by the divergent activities of network, systems and application changes, where the majority of customers continue to operate with manually maintained network diagrams. Even when changes are known, the impacts to visibility between systems is not clear because the information is never available in a single place Modern cloud environments present new unique challenges to understanding host identity and network visibility. No longer do hosts maintain the same IP address originally assigned when restarted, and with auto-scaling the identity of the individual host is no longer meaningful. All of this makes security technology unable to have a complete understanding of deployment environments and the exposure these new environments present to applications that are running therein
- #21 - Current security people, process and tools are oriented to provide value on the order of years, quarters and even months. Even the best compliance practices in PCI proceeds on this cadence because they are simply difficult to keep up on in the mix of tools and practice that exist today. - DevOps tools and techniques are moving faster and faster, providing mature DevOps organizations with the ability to deliver business value into production at an amazing rate. - The result is that security is effective on one in every 10 or even 100 of system configurations
- #23 So let’s turn our attention away from the challenges that face security technology and practice to concrete things that can be done to help bring effective security to your environment by leveraging DevOps tools and practices. By fully embracing end-to-end configuration management and automated continuous deployment you can gain a greater control of the security posture of your environment. I’m going to cover three key techniques that should be built into all systems: First key choice is to build systems in which most file systems are read-only, and writeable file systems are noexec/nosuid. This will severely limit the non-privileged attack surface of systems Second is the corollary of immutable systems – that applying changes to should be done by replacing nodes in load balanced groups, rather than performing upgrades in-place. This enables the use of immutable systems. Third is to configure strong auditing and centralize all logging in order to detect out-of-band changes that are made outside of the the continuous deployment systems Finally ensure that you use common base images and system configurations, in order to allow vulnerabilities to be addressed on-mass, rather than a set of one-off systems
- #24 This slide is meant to be a pre-cursor to the Outcomes effort. Currently in today’s “Software Defined Infrastructure” nothing exists that allows organizations to conduct vulnerability scans in a real meaningful way that is in “lock step” with continuous deployment of systems. Scanning today is infrequent at best and usually identifies numerous vulnerabilities that must be addressed Addressing vulnerabilities in-mass introduces considerable risk to existing stability and performance of the infrastructure A way needs to exist that will allow an environment to scan frequently and identify problems as they arise. Vulnerabilities (or configuration problems) can be quickly addressed before they “stack up” and introduce added risk
- #25 This slide will provide an example of constructing immutable system using config management and deployment tools Apply those standards to systems that can then be imaged and used as “golden images” during automated deployments. Use tools that enable creating machine images (Packer) which allow standardizing system deployment across different infrastructures (VMWare, EC2, etc.) Design systems so that executable code is cleanly separated from data at the file system level, and implement a permissions strategy for these file systems based on Least Permissions, including read-only and noexec/nosuid mount options. Use resources like the SANS Linux Security Checklist or CIS Benchmark as you build golden base images that form the foundation of your deployments. These provide a clear definitive answer to how you can easily ensure security of the systems within your infrastructure, and CIS even offers hardened AMIs for EC2. Leverage configuration management (Chef) to ensure that all software and configuration changes are standardized and repeatable. Following these guidelines will ensure that your infrastructures are built to be as immutable fashion, and will enable maximum leverage out of your security and compliance hardening efforts. All of this makes separation of duties easier to achieve when there are no longer environment specific configurations that are outside of configuration management tools being used by developers.
- #26 This slide will show how to integrate strong auditing (via linux auditd) and centralized logging as a basic attribute of all systems built using the configuration management tools. Enable auditing (auditd) on local systems and ensure configuration is controlled through configuration management (Chef). This example shows a snippet of the CIS benchmark recommendations for auditd configuration for CentOS/Redhat systems. Ensure syslog is offloaded Alert Logic has a nice tool…
- #27 This slide will demonstrate how to affect change by replacing nodes in a load balancing group instead of performing in-place upgrades. Embrace the concept of Phoenix Upgrades, systems should not be pulled along but reborn when new versions are available Technical debt begins to accumulate and replacing over upgrading resets the clock When combined with deployment of immutable systems and configuration management, it provides a path for releasing new technology (kernel upgrades, etc.) in complete package that can be fully tested prior to deployment
- #28 This slide will focus on how important it is to develop, test and deploy the same way in order to greatly reduce overall risk and introduced vulnerabilities Configuration management ensures that “Infrastructure” becomes defined as code and introduces the capability to deploy in many different environments (Laptop, Test Data Centers, Production, etc). Tools like CHEF provide us the ability to define infrastructure as code Standardizing Environment Images ensures that we are not overburdened with creating many different images that “might” work in all different environments. Tools like Packer allow us to create an image that can be build and deployed into many different environments (LXC, EC2, VMWare, etc..) Orchestration (We use an internally developed tool called Environment Manager) allows us to ensure we can deploy our systems and ensure they are configured in a repeatable fashion. This ensures that systems are deployed, installed and configured in the proper order to maximize repeatable experience for development, testing and production Combining all three with the ability to execute Phoenix Upgrades ensures that we can deploy infrastructure in a repeatable fashion AND we do so without incurring technical debt. This greatly reduces risk and overall introduction of new vulnerabilities due to misconfigurations
- #29 Starting from that foundation, we have the opportunity to go further to achieve end-to-end security in the AWS cloud infrastructure. DevOps tools and techniques are a start… and software defined infrastructure provides access to end-to-end visibility… but security tools really need to catch up. Security technologies need to advance in three very important ways: AWS provides significant visibility into network and system deployments, which should be leveraged directly to identify the configuration of whole environments Transform traditionally period assessment tasks normally performed on a weekly or monthly basis, into more focused change-driven responses that operate continuously Automatically reconfigure security protection services based on the real-time change to the environment Each of these represent important advances that Security Technologies need to adopt to truly begin providing security coverage for Cloud environments
- #30 The first challenge is for our security technologies to gain visibility to the complete end-to-end picture of deployed environments. What we need is for this level of visibility to be automatic and immediately react to changes. This video segment shows the automated discovery of an entire complex environment, including EC2 instances, subnets, VPCs, and internet gateways. <video segment starts, showing auto-discovery of an environment deployed in AWS> This type of automatic discover and complete end-to-end visibility is necessary for security technologies to have up-to-date information about the environment in which it operates <video stabilizes the view, and then begins reconfiguring additional VPC, subnets and some hosts> Now when changes are made to the environment, via a deployment of additional instances in this case, the view updates to show the new adjusted environment <video stabilizes after the addition of new instances>
- #31 The last challenge is to respond to these changes in the as-deployed environment to automatically reconfigure the security protection to match the environment. This is particularly important in dynamic auto-scaling environments where the size of the environment could grow 10 times or more in a matter of minutes. This video segment picks up with a simpler web stack environment running just a few instances, plus the security infrastructure instances that are currently protecting the web stack. <video segment starts, showing the web stack and security appliances protecting it> Now as traffic grows to the web stack, the ELB will start new instances to handle the traffic. <video continues to showing the increased traffic and auto-scaled application stack instances> As the web stack grows, the security infrastructure protecting it expands to handle the additional traffic. <video continues to show auto-scaled appliance deployments to handle the application traffic>
- #32 The last challenge is to respond to these changes in the as-deployed environment to automatically reconfigure the security protection to match the environment. This is particularly important in dynamic auto-scaling environments where the size of the environment could grow 10 times or more in a matter of minutes. This video segment picks up with a simpler web stack environment running just a few instances, plus the security infrastructure instances that are currently protecting the web stack. <video segment starts, showing the web stack and security appliances protecting it> Now as traffic grows to the web stack, the ELB will start new instances to handle the traffic. <video continues to showing the increased traffic and auto-scaled application stack instances> As the web stack grows, the security infrastructure protecting it expands to handle the additional traffic. <video continues to show auto-scaled appliance deployments to handle the application traffic>