openstack, devops and people
32 likes2,932 views
The document discusses DevOps practices at Kakaocorp. It provides background on Andrew Yongjoon Kong and describes some key metrics of Kakaocorp's OpenStack deployment. It then covers concepts like collaboration, affinity, tools, and scaling in a DevOps context. Specific examples at Kakaocorp include using GitHub, Jira, Chef, and Jenkins. It also introduces initiatives like KEMI for integration and DKOS for container management.
openstack, devops and people
- 2. KakaocorpAndrew. Yongjoon kong • Cloud Technical Advisory for Government Broad Cast Agency • Adjunct Prof. Ajou Univ • Korea Data Base Agency Acting Professor for Bigdata • Member of National Information Agency Bigdata Advisory committee • Kakao à Daum Kakao à Kakaocorp, Cloud Computing Cell lead Korean Edition
- 3. KakaocorpSome Numbers about kakao openstack 1563 tenants 632 pull request since 2014.9 251 VMs are created/deleted per day 8703 vms
- 4. KakaocorpSome information about kakao openstack openstack release from grizzly to Kilo 5 times upgraded total 4Regionpg,seoulv2,seoulv1,busan additional service Heat/Trove/Sahara
- 5. Kakaocorp Cloud, we do have SDN, not have O penflow, No others eth0 Compute node nova-compute neutron-linuxbri dge-agent neutron-dhcp-a gent Gateway 10.10.100.1 linux bridge vm IP:10.10.100.2/32 Routing Table 1 10.100.10.2/32 via 192.1.1.201 BGP 192.1.1.202 BGP Virtual Switch block Virtual Process block Neutron-l3-agent vlan vlan Virtual Router Service Route Table 1 Management Route Table 1 Practice Frugality to Boost Creativity
- 6. KakaocorpSPECIAL Network Resource • Generally, Known as /32 network. • No L2 (link) consideration needed anymore ( no subnet ) • With Dynamic Routing Protocol, it move every where. • Simple IP planning ( Just think of IP ranges ) • It’s very Atomic Resource, it keeps its IP after migration through zones 10.0.0.1 / 32 or IP 10.0.0.1 netmask 255.255.255.255
- 7. Kakaocorp Integrated Information Service Bus & EIP : Code Name KEMI Enterprise Integration • Topic based Data ETL • Can cover every computing res ource ( Physical Server, Virtual instance, Container, Public Clo ud ) • Abstracting “Data Center Infor mation layer” • Can make deep engineering ex perience over every resources. Physical Servers Virtual Instances Containers External Clouds Others (switches, logs) monitoring KEMI IMS (kakao CMDB API) ESB Rule Engine Notificati on ETL Data Center Information abstraction layer API predicting scheduling Openstack Heat Other Service API Data Center (or Service ) Management Activity control
- 10. KakaocorpCITE
- 11. KakaocorpDKOS
- 13. The Term : Devops • Devops is no longer composite of two words “Development + Operation” • Cultural movement that changes • how individuals think about their work, • values the diversity of work done, • supports intentional processes that accelerate the rate by which businesses realize value, and measures the effect of social and technical change • Devops is about finding ways to adapt and innovate social structure, culture, and technology together in order to work more effectively
- 14. Software Development Methodologies • Waterfall • Focus on sequential progression • Large amount of time spent in requirements and design phase.
- 15. Software Development Methodologies • Agile • lightweight and flexible than previous methods such as waterfall • Agile Manifesto • We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: • individuals and interactions over processes and tools • working software over comprehensive documentation • customer collaboration over contract negotiation • responding to change over following a plan
- 16. Software Development Methodologies • Scrum • Ken Schwaber and Dr. Jeff Sutherland presents • focuses on maximizing a development team’s ability to quickly respond to changes in both project and customer requirements. • Sprints ( 1~4 weeks ) • Daily standup • Restrospect • What happened • What went well/poorly ?
- 17. Devops Misconceptions • Devops only for Developers and System administrators • Devops is a team • Devops is a Job Title • Devops Means Doing All the Work with Half the People • Devops is about Tool/Automation
- 18. Devops Anti-Patterns • Blame Culture • Prefer not to blame • Silos • Prefer share or usable product • Root Cause Analysis • Prefer quick/frequent response
- 19. Devops for Pillars Collaboration Affinity Tools Scailing
- 20. Collaboration comprises • Asynchronous code review • Documentation • Updating issues and bug report • Demonstrating weekly progress • Regular status updates • Pairing
- 21. Collaboration: Kakao cloud • Asynchronous code review: Github Pull Request • Documentation: Source code or Wiki Page • Updating issues and bug report: • Jira Comment • Githup and Jira Integration • Demonstrating weekly progress: Sprint Review • Regular status updates: Daily Standup, Auto Testing • Pairing : Github Review/Merge
- 23. Collaboration: kakao cloud • Develop Github proxy. • Sending Githup event to Kakao talk
- 24. Affinity • Have to think about “What comprises Team” • Have to fight against Diversity • Diversity in Gerder/Race/Nationality/Age/Veteran Status/Religion/Familial Status is OK and inevitable • Way of Handling works • Measurement of success • Communication style in works • Leadership • The key is “From individiual to Teams”
- 25. Tools Local Development Environment Version Control Artifact Management Automated Test Development Tools
- 26. Tools: kakao cloud Local Development Environment KFIELD Version Control Github Artifact Management Chef + Cookbook version Automated Test Jenkins’ Github PR Trigger + Rake+Kitchen + (openstack/docker) Development Tools
- 28. Tools: Version Control, Github • Pull Request based commit history management • No Self Merge rule Applied
- 29. Artifact Management : Chef /Cookbook version • Two Species source code • One is Openstack Code ( nova, neutron , cinder ….. ) • One is Openstack Deploying Code ( chef , infra-automation) • From Chef code, it uses openstack’s certain branch • Use cookbook metadata for artifact version. • Automatic Engaged after testing and merged
- 30. Automated Test &Build • 3 category • On Demand Automated test and build • Scheduled Test and build • Triggered Automated Test and Build • More category • Smoke testing • Regression Testing • Usability Testing • A/B Testing
- 31. Automated Test &Build : Jenkins Test • Triggered • Sometimes use on-demand test , “Test this please” comment on github pull request
- 33. Tools is accelerator (in good/bad way) • Tool is about which programming languages, IDEs, text editor, shell, configuration management solution, or chat program to use • Standardization of Tools • Improve communication • Set boundaries • Repair understanding within the scope of the devops compact • Tools Impacting Communication • Tool can reduce( or add ) the friction
- 34. References • Effective devops (O’reilly) • Test Driven infrastructure with chef (O’reilly) • Infrastructure as a code (O’reilly) • 클린 코드 Clean Code ( Acornpub) • 리눅스 네트워크의 이해 Understanding Linux network internals (Acornpub) • Personal Experience( andrew, just joke)