Dev Ops @ Envato
 ... or how we found out there’s a name and a support
group for all the stuff we already did to let a team of 8
deploy heaps of times a day to a Ruby on Rails app that
   has scaled up to around 20 million requests a week
                  without an ops team.
John Barton

 @johnbarton

john@envato.com
Envato?

http://envato.com
Stock Marketplaces
         &
 Tutorial Network
Marketplaces
Like iStockPhoto but for
  other creative niches
           or
  eBay for digital goods
Tuts+ Network
Big Blog Network for
   education in creative
fields.... and linkbait (ducks)
Big traffic and interesting in
it’s own right, but that’s for
    @ryanfaceryan to talk
            about
Why?
2 things recently made us
 think we might be doing
something a bit different
Neal Ford @ YOW!

 “Rails can Scale”
200k+ daily requests
         vs
 2m+ daily requests
Dev Ops Melbourne

“Continuous Deployment”
1 Deployment per week or
         fortnight
            vs
  5 Deployments per Day
The Marketplace
August 2006
One Marketplace
FlashDen
Rails 0.13b
No Users
No Traffic
March 2011
9 Marketplaces
ActiveDen (nee FlashDen)
       AudioJungle
      ThemeForest
        VideoHive
      GraphicRiver
      CodeCanyon
        3D Ocean
    Tuts+ Marketplace
       PhotoDune
Rails 2.3.11
       687,830 Users
20 Million Requests Weekly
One Codebase and One
Production Environment
... and well gosh darn it we
    just deploy to that site
 whenever we darned well
           feel like it
As much a product of time
and circumstances as good
 decisions and hard work
The “Golden Age”
3 Developers
30 minute feedback cycle
deploy, discuss on forums,
       deploy again
Decision 1:
Preserve everything good
about the the startup days
  for as long as we can
Commodity Hosting
Notice I’m not saying
“Cloud Computing”
You can't
trust “the
cloud” but
  you can
trust “the
 cloud to
  be “the
  cloud”.
Decision 2:
  Conservative Platform
choice so we don’t have to
     sweat the details
Corporate AntiPatterns
We’ve all been doing dev
 long enough to see this
stuff screwed up over and
         over again
Decision 3:
Don’t do all that stuff
So what do we actually do?
“Culture of respect &
trust, good attitude toward
          failure...”
Ted Dzubia
“How about "culture of
     stop fucking up"?”

 Ted Dzubia
http://teddziuba.com/2011/03/devops-scam.html
Ultimate Responsibility
"The fault, dear
Brutus, is not in our
    QA or Ops,
 But in ourselves."
Test Driven Development
           vs.
        QA Team
Test Driven Infrastructure
           vs.
        Ops Team
Both as a team and as
individuals we own our
work from when we are
    asked to do it...
... until is is demonstrably
error free and performant
         in production
Everyone is in the (paid) on
       call roster
Everyone takes a turn at
Level 2 Customer Support
Want those jobs to be
       easier?

   Stop fucking up.
Process
LEAN / TPS Principles

...without the process
You cannot write code any
faster than you can deploy
      it to production
Long running projects?
A. B. C.

Always Be Cmerging

  (the c is silent)
Dark Launch
Feature Flags
Private Beta
Community
We do trip up running this
           fast
But through years of
 openness with our users
via our forums and owning
     up to our mistakes
... we’ve ended up with a
 (relatively) sympathetic
        community
Time Zones both help and
         hurt
Traffic peaks during US day
  means that if things go
wrong we’re usually asleep
But it makes it very easy to
deploy during our business
          hours
Dev vs. Ops
Our Solution:
Don’t have Ops
Outsource commodity
       platform bits:
 virtualisation/cloud, have
rackspace take care of db/
         mailserver
Ensure the dev team has
the skills to take care of the
              rest
Keep the stack as Vanilla as
         possible
Virtualised servers in our
      own sandbox.

Cloud Flexibility - Cloud
   Shit-ness = WIN
Automate Configuration
    Management
Don’t modify live boxes,
tear down and build again
      from scratch
Take advantage of an
  individual's talents, but
   don’t rely upon them

ie. don’t accidentally create an ops guy
Performance & Scaling
Not as big a deal as
 everyone thinks
Shared-nothing load
balanced app servers + out
of request queue workers

   not rocket surgery
Measure, deploy, measure
again and then tweak or
        rollback

    New Relic FTW
There is no code faster
    than no code.
Caveats
We rely on our growth
following a steady exponent
   for this approach to be
             viable
We made a conscious
 tradeoff between delivery
time and production snafus
We’ve traded off platform
 innovation for product
       innovation

   (no MongoDB, etc)
Auditors aren’t big fans of
one team with access to
 everything, but you can
 mitigate these problems
Question?

Dev Ops @ Envato