Continuously Deploying Culture: Scaling Culture at Etsy - Velocity Europe 2012
- 1. Continuously Deploying Culture Scaling Culture at Michael Rembetsy, Director of Operations Engineering (@mrembetsy) Patrick McDonnell, Senior Operations Engineer (@mcdonnps) October 4, 2012 Velocity Europe
- 2. About Us • Operations team at Etsy totals twelve • Engineering totals 125 • Company totals about 350 • 15 million members in over 150 countries • Over 800,000 sellers • 1.4 billion page views monthly
- 3. Disclaimer A + B != Culture
- 5. Silos and Barriers • Etsy employs 30 to 35 • About half are engineers • Siloed culture, barriers to collaboration in engineering • “Sprouter” • Designed to prevent engineers from directly touching databases
- 6. Management Shake-up • Maria Thomas promoted to CEO • Understands that community is of utmost importance • Begins to prioritize culture that supports community • Chad Dickerson brought on as CTO • Starts to bring focus to engineering team
- 9. Opening Communications • fix.etsy.com • First time exposing engineering to community • Scheduled maintenance posted ahead of time • Outage follow-ups • Culturally, technical debt needs to be paid off
- 11. Opening Communications • fix.etsy.com • First time exposing engineering to community • Scheduled maintenance posted ahead of time • Outages followed upon • Culturally, technical debt needs to be paid off
- 12. 2008 Takeaways • WTF did I get myself into? • Low or no impact projects • Downtime is accepted / needed • Community is most important • Lucky to make it through holiday season
- 13. 2008 End of Year Snapshot • Gross Merchandise Sales: $87.3 million • Visits: 163 million • Unique Visits: 47.7 million
- 14. 2008 Action Items • Gain support from the top and bottom to change culture • Increase transparency both within the organization and to the public • Pay back technical debt as soon as possible
- 15. Sea Change • Major shift in culture • People leave or are let go • Too many remotes • Teams centralized in Brooklyn • Need “DevOps” culture before remotes are manageable
- 16. Building the Foundation • DevTools team created • Develops first iteration of Deployinator • Stabilizes the deploy chain through dev, QA, and prod environments • Infrastructural overhaul • Move from lighttpd to Apache starts • Added monitoring and graphing • Better metrics exposure to non-technical divisions
- 19. Building the Foundation • DevTools team created • Develops first iteration of Deployinator • Stabilizes the deploy chain through dev, QA, and prod environments • Infrastructural overhaul • Move from lighttpd to Apache starts • Added monitoring and graphing • Better metrics exposure to non-technical divisions
- 20. Hiring Push • Only two operations engineers and too few local developers • Concentration on sourcing talent to keep pace with growth • Move from Downtown Brooklyn to larger space in DUMBO • More accessible to non-Brooklynites • More mature tech and art community
- 21. Internal Improvement • Stand-up meetings • Time-consuming, but necessary to improve communication • Inter-team collaboration • Ops involved in decision-making for dev projects and vice versa • Internal stability much stronger, damage control period ends • Network solidified, infrastructural foundation allows for future growth
- 22. Stability Arrives • Everyone wants to collaborate from top down and bottom up • Not just upper management pushing down • Everyone shapes culture, suggestions always welcome • People happy to come to work, contribute beyond their job description • No more scheduled downtime, site remains up as much as possible • Master database purchased as a stopgap for holiday capacity
- 23. 2009 Takeaways • Foundation solidified • Technology • Human capital • Beginning of “DevOps” culture • Berlin Wall falls
- 24. 2009 End of Year Snapshot • Gross Merchandise Sales: $177 million (102.6%) • Visits: 320 million (96.8%) • Unique Visits: 92.9 million (94.8%) • Page Views: 6.45 billion
- 25. 2009 Action Items • Stabilize the parts of your organization which create thrashing • Hire staff that will make a difference • Think about collaborative processes • When dealing with staffing constraints, prioritize projects by impact • Get things done (later “Just Ship”)
- 26. Renewed Energy
- 28. .etsy.com
- 29. 40 30 20 10 Very end of 2009 Today
- 30. Standardization Effort • PHP • Use it and nothing else • Everyone should be able to read and rewrite your code • MySQL becomes database of choice • Backfill of PostgreSQL tables to MySQL shards begins
- 31. If it Moves, Graph it • Graphing tools • Ganglia, Graphite, FITB, Weathermap • Monitoring • Nagios, Naglite • Increased focus on work/life balance • We removed alerts! WHAT?
- 34. If it Moves, Graph it • Graphing tools • Ganglia, Graphite, FITB, Weathermap • Monitoring • Nagios, Naglite • Increased focus on work/life balance • We removed alerts! WHAT?
- 36. Management Ideals • Accept failures but not lower standards • Доверяй, но проверяй (trust, but verify) • Blameless post-mortems • Welcome one-on-ones (http://bit.ly/cCWMqr) • Career planning • Happy company = happy community
- 37. Engineering Processes • Developer on call • A/B Testing • Prototypes • Feature flags and ramp-ups • Schema Change Thursday
- 38. 2010 Takeaways • Paring down the number of technologies used for development • Focus on technical visibility throughout the organization • Developers take responsibility for code release • Freedom to hire as needed • New focus on work life balance
- 39. 2010 End of Year Snapshot • Gross Merchandise Sales: $307 million (73.4%) • Visits: 534 million (66.7%) • Unique Visits: 147 million (58.2%) • Page Views: 9.25 billion (43.5%)
- 40. 2010 Action Items • Don’t guess at what’s wrong; graph it, monitor it, and find out • Empower developers with responsibilities • Have clear, documented standards and practices • Keep human management a priority
- 41. The Reaping • Death of non-standard technologies • Mongo, Scala, CoffeeScript, etc. removed from production systems • Sprouter is eradicated • No more Python • Removal of a major barrier to full developer accountability • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)
- 43. The Reaping • Death of non-standard technologies • Mongo, Scala, CoffeeScript, etc. removed from production systems • Sprouter is eradicated • No more Python • Removal of a major barrier to full developer accountability • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)
- 44. Organizational Change • Management becomes more engineering focused • Chad becomes CEO • Kellan is promoted to CTO • John is promoted to SVP of Technical Operations
- 45. Technical Contribution • Big push to contribute to open source and technical community • Deployinator, Statsd, Logster, many more • Three annual goals for every engineer • Speak at a conference • Write a blog post • Open source code
- 46. Still Scaling
- 49. And Still Scaling Culture • Engineering culture decides git is better for our working style than svn • Product development efforts are refocused on high-impact products such as search ads • Ops works to improve signal-to-noise ratio of alerts and develops internal tools such as Schemanator • Focus on security • SCRAM team, Security and Fraud team • Check out Nick Galbreath’s talk at DevOpsDays Austin (http://slidesha.re/IMaavq)
- 50. Happy Holidays • Game days to test failures before they happen • Capacity planning is easier • More dashboards (framework now on GitHub!) • Improved weekly financial reporting
- 52. Happy Holidays • Game days to test failures before they happen • Capacity planning is easier • More dashboards (framework now on GitHub!) • Improved weekly financial reporting
- 53. 2011 Takeaways • Year of the tool • Statsd, Deployinator, Supergrep v2, Logster, Schemanator, FITB • DevTools adds three engineers • Automation reaches maturity, few surprises • Focus on security allows PCI infrastructure to be completed in 6 weeks • Engineering matures, both platform and people
- 54. 2011 End of Year Snapshot • Gross Merchandise Sales: $526 million (71.4%) • Visits: 895 million (67.7%) • Unique Visits: 237 million (61.1%) • Page Views: 12.9 billion (39.3%)
- 55. 2011 Action Items • Senior management at a tech company should be technology- focused • Implement configuration management and automation as soon as possible (saves headaches later) • As technical staff increases, continue to focus on projects that matter • New technical challenges do not dictate cultural shift
- 56. Massive Growth • Explosive growth in hiring • Major organizational changes around product • Increased focus on community • High-impact products (shipping labels, gift cards) • Ramp-up of internationalization projects • Unique engineering challenges force non-standard technology • Redis • Virtualized CI test slaves
- 57. Spreading Information • Invitation to other teams to stop by the office for a chat • Ops becomes more involved in external informational exchange • Code as Craft blog posts • Rapid scaling of personnel, no one knows everyone • Remotes come in to Brooklyn more often
- 60. Work in Progress • Developer boredom curbed by allowing transfers between teams • Even between divisions (such as engineering -> product) • Developers now read data from prod databases for development • Removes one of the last anomalies between development and production • Front End Performance team forms to minimize site load times
- 61. 2012 Takeaways • Organizational resilience allows for total focus on community • Improvement on international product • Increased transparency to the public • Focus on informational exchange, both internally and externally • Open source all the things
- 62. 2012 Action Items • Know when not to try something • Focus on performance early • Allow dynamic allocation of resources • Don’t allow size to dictate culture
- 63. Future Predictions • “Wait till you hit 500 people, that’s when everything falls apart” • Etsy will not fall apart, but will change • Management supports and responds to cultural shifts • Freedom of movement between job functionalities • Trust that change is for the better
- 64. Future Predictions • Will create better and innovative ways of communicating as we scale • Building tools is the Etsy way • Continue to influence corporate culture • B Corp • Employee happiness
- 65. Office Hours Sponsor Pavilion Table A 2:00 to Last Call at the Bar Michael Rembetsy (@mrembetsy) Patrick McDonnell (@mcdonnps) https://github.com/etsy