DevOps : It's Made of People

Editor's Notes

• #2: Hi I’m Dave Benjamin.\nI specialize in agile development of software applications.\nOver the last few years I’ve been particularly interested in making good on agile’s promise of quick delivery, even in the enterprise. Its been interesting.\n
• #3: I apologize if this whole discussion doesn’t apply here.\nI’ve spent time delivering applications in larger enterprises, and again and again I see the same bitter conflict around deployment.\nBut this may not be true here, here may be a beautiful place of sweetness and light.\nBut just in case...\n
• #4: Most agile teams (pathologically) want to please the business by releasing the most valuable features to production early, and often.\nThey do this because the business pays them for features. Now.\n
• #5: Ops is responsible for production.\nKeeping it running, taking the considerable heat when there are problems, rebuilding it after big explosions, its all about responsibility.\n(optional: ITIL is about responsibility.)\nThey are usually seen as a cost center, but we’ll come back to that.\n
• #6: You can see the potential for conflict between Dev and Ops.\nThere is a tendency for BOTH sides to see the other as small-minded and arrogant.\nAnd every time something goes terribly wrong, there seem to be entertaining anecdotes on both sides convincingly blaming the other.\n
• #7: When Dev & Ops really get to fighting they can lose sight of the fact that to the rest of the organisation (literally, the business end) doesn’t really care about this conflict.\nThey want it now, they want it safely. Is that too much to ask?\n(Optional) I have three sons. When I listen to them argue loudly about who’s turn it was to put the bins out last night, I don’t care. I just know I’m going to be skulking around all this week sneaking bags into the neighbors bins. I blame them all.\nBut, if just the cultural divide was bad enough, it gets worse...\n
• #8: Most big firms manage their money through a mixture of Capital Expenditure for projects (CAPEX) and Operational Expenditure for, well, Operational things (also called BAU, Business As Usual).\nUsually there is a lot of CAPEX, which injected to make the company fly like a bird.\nUsually OPEX is grudgingly spent, and there is a good career reducing this number.\n
• #9: But CAPEX is fleeting.\nProjects appear, wave their arms grandly, deliver great things, jam them into production under the duress of the deadline, and then wrap up.\nThere is a lot of pressure to spend project money on features, not on sustainable practices, or supportability, things that, say, OPS may be quite interested in.\n
• #10: Ops usually lives on a thrifty OPEX budget, and whatever CAPEX it can charge projects.\nWhat usually happens is that any change activity is tied to some sort of chargeable amount, a “gate fee” if you will, so that they both claw back some money from the rich projects, and hopefully force the project to behave responsibly.\n
• #11: Invariably you reach a point where there is a huge divide.\nNot only does the Dev team have a mountain of paperwork & process to deal with, they struggle to figure out even how to approach it.\nThey tend to ignore it until the last minute, and then hopefully the threat of a slipped deadline will cause the business to step in, kick down the door, and accept mountains of risk to get it live.\n
• #12: Dev teams often look to the PM to solve the delicate choreography of getting the release through to production.\nBut in many places the PM has enough on their hands simply making sure everyone is paid and all the stakeholders are happy.\nThere isn’t enough left for them to get into the nuances of whether the disaster recovery plan really does require updating.\n
• #13: So, from a dev perspective, what do you do?\nThis isn’t a 12 step program, but always a good first step is to map out the current process.\nYou can use Lean techniques, like value stream mapping, or you can simply make big charts of target environments, whats in them, and what process it takes to get them in place.\nWrite it big, put it on the wall. Get everyone thinking about it, why it is, and how it can improve.\n
• #14: I often think the basic food group of agile is the Retro.\nGet a meaningful portion of the team to talk to a meaningful portion of the OPS team.\nDon’t aim for a big workshop, OPS is frequently too busy, and it is often a bad vibe.\nTry tacking on 15 minutes to each Dev/Ops meeting to maintain a list of good/bad/puzzling things.\nIdentify actions that will resolve the bad things, and, really really important here...\n
• #15: DO IT.\nI can’t overemphasise the importance of building trust between groups by saying what you’re going to do, and then doing what you said.\nA completed list of “10 things that really bug us” is worth 10 walls of butcher paper and sticky notes of blue sky thinking.\n
• #16: Depending on your existing culture (sorry if I’m pessimistic, its just where I’ve been) you have to respect that the Dev/Ops divide has been in place for a long time.\nThe act of ramming countless projects down their throats will leave a deep mistrust.\nIt will take a while to wear down. Don’t be disheartened or insulted.\n
• #17: On the other hand, don’t be too pessimistic.\nPeople are smart, people know things. If something works, they’ll get behind it.\n
• #18: I oversaw something like a dozen or so yellowpages.com.au production deployments, one of the busiest .com.au sites.\nAfter each production deploy I’d go to the Coles downstairs and buy an armload of candy bar bags (around $30 or so).\nI’d go around the ops area dropping off bags and having a chat with each group, saying thanks, talking about the cool new features now live, and asking how the deploy went.\nDuring one of these “Candy Bag Retros” our site crashed, really hard.\nAfter a three day weekend of intense work, we managed to resolve what was a curly java memory management issue. Not a forehead slap, but a legitimately hard to find performance issue.\nWe had OPS on side from minute one, and there was really no time wasted shouting and fingerpointing. A really good result.\nI like to think the candy, and the conversations around it, helped.\n
• #19: There is an excellent Slideshare presentation from two guys from Flickr.\nCalled something like “10 deployments a day at Flickr” its one of the things that really started the “DevOps” discussion (movement? discussion).\nBut it reminds me, and I apologize, that I’ve been talking about people here, and not about technology.\nSorry.\nYou probably came for a bracing array of technical options, which I’m happy to talk to, but to save you time I can briefly summarize the core message of DevOps with regard to technology...\n
• #20: Automate your builds (you do that), automate your tests (you do that?), automat your deployments to systest, automate your generation of release notes and change documents, automate your UAT, STAGE, and dare I say it, PRODUCTION deployments.\nAutomate the configuration of your machines (be they virtual or not).\nKeep your automation scripts and whatnot in source code control, test them and improve them.\n
• #21: But do all this automation to make things repeatable and trustworthy.\nIf it hurts, automate it until it stops hurting (“hurt” usually means “risk”).\nMake sure automation is created collaboratively, Dev shouldn’t force Ops to accept black box automation, Ops shouldn’t force Dev into additional requirements.\nWork together, make the automation serve to build trust and reduce pain.\n
• #22: Thats the opening remarks, there’s heaps more to discuss, but I want to let you guide the discussion from here on.\n
• #23: \n
• #24: \n