![Without Gargoyle
SWITCHES = {
# enable my_feature for 50%
'my_feature': range(0, 50),
}
def is_active(switch):
try:
pct_range = SWITCHES[switch]
except KeyError:
return False
ip_hash = sum([int(x) for x
in ip_address.split('.')])
return (ip_hash % 100 in pct_range)
If you use Django, use Gargoyle
Wednesday, June 22, 2011](https://image.slidesharecdn.com/continuousdeployment-110622042605-phpapp01/85/Pitfalls-of-Continuous-Deployment-12-320.jpg)
![Using Graphite
# statsd.py
# requires python-statsd
from pystatsd import Client
import socket
def with_suffix(key):
hostname = socket.gethostname().split('.')[0]
return '%s.%s' % (key, hostname)
client = Client(host=STATSD_HOST, port=STATSD_PORT)
# statsd.incr('key1', 'key2')
def incr(*keys):
keys = [with_suffix(k) for k in keys]:
client.increment(*keys):
Wednesday, June 22, 2011](https://image.slidesharecdn.com/continuousdeployment-110622042605-phpapp01/85/Pitfalls-of-Continuous-Deployment-37-320.jpg)
![Setting up Sentry (1.x)
# setup your server first
$ pip install django-sentry
$ sentry start
# configure your Python (Django in our case) client
INSTALLED_APPS = (
# ...
'sentry.client',
)
# point the client to the servers
SENTRY_REMOTE_URL = ['http://sentry/store/']
# visit http://sentry in the browser
Wednesday, June 22, 2011](https://image.slidesharecdn.com/continuousdeployment-110622042605-phpapp01/85/Pitfalls-of-Continuous-Deployment-43-320.jpg)
Pitfalls of Continuous Deployment
- 1. DISQUS Continuous Deployment Everything David Cramer @zeeg Wednesday, June 22, 2011
- 2. Continuous Deployment Shipping new code as soon as it’s ready (It’s really just super awesome buildbots) Wednesday, June 22, 2011
- 3. Workflow Commit (master) Integration Failed Build Deploy Reporting Rollback Wednesday, June 22, 2011
- 4. Pros Cons • Develop features • Culture Shock incrementally • Stability depends on • Release frequently test coverage • Smaller doses of QA • Initial time investment We mostly just care about iteration and stability Wednesday, June 22, 2011
- 5. Painless Development Wednesday, June 22, 2011
- 6. Development • Production > Staging > CI > Dev • Automate testing of complicated processes and architecture • Simple > complete • Especially for local development • python setup.py {develop,test} • Puppet, Chef, simple bootstrap.{py,sh} Wednesday, June 22, 2011
- 7. Production Staging • PostgreSQL • PostgreSQL • Memcache • Memcache • Redis • Redis • Solr • Solr • Apache • Apache • Nginx • Nginx • RabbitMQ • RabbitMQ CI Server Macbook • Memcache • PostgreSQL • PostgreSQL • Apache • Redis • Memcache • Solr • Redis • Apache • Solr • Nginx • Nginx • RabbitMQ • RabbitMQ Wednesday, June 22, 2011
- 8. Bootstrapping Local • Simplify local setup • git clone dcramer@disqus:disqus.git • ./bootstrap.sh • python manage.py runserver • Need to test dependancies? • virtualbox + vagrant up Wednesday, June 22, 2011
- 9. “Under Construction” • Iterate quickly by hiding features • Early adopters are free QA from gargoyle import gargoyle def my_view(request): if gargoyle.is_active('awesome', request): return 'new happy version :D' else: return 'old sad version :(' Wednesday, June 22, 2011
- 10. Gargoyle Deploy features to portions of a user base at a time to ensure smooth, measurable releases Being users of our product, we actively use early versions of features before public release Wednesday, June 22, 2011
- 11. Conditions in Gargoyle from gargoyle import gargoyle from gargoyle.conditions import ModelConditionSet, Percent, String class UserConditionSet(ModelConditionSet): # percent implicitly maps to ``id`` percent = Percent() username = String() def can_execute(self, instance): return isinstance(instance, User) # register with our main gargoyle instance gargoyle.register(UserConditionSet(User)) Wednesday, June 22, 2011
- 12. Without Gargoyle SWITCHES = { # enable my_feature for 50% 'my_feature': range(0, 50), } def is_active(switch): try: pct_range = SWITCHES[switch] except KeyError: return False ip_hash = sum([int(x) for x in ip_address.split('.')]) return (ip_hash % 100 in pct_range) If you use Django, use Gargoyle Wednesday, June 22, 2011
- 13. Integration (or as we like to call it) Wednesday, June 22, 2011
- 14. Integration is Required Deploy only when things wont break Wednesday, June 22, 2011
- 15. Setup a Jenkins Build Wednesday, June 22, 2011
- 16. Reporting is Critical Wednesday, June 22, 2011
- 17. CI Requirements • Developers must know when they’ve broken something • IRC, Email, IM • Support proper reporting • XUnit, Pylint, Coverage.py • Painless setup • apt-get install jenkins * https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu Wednesday, June 22, 2011
- 18. Shortcomings • False positives lower awareness • Reporting isn't accurate • Services fail • Bad Tests • Not enough code coverage • Regressions on untested code • Test suite takes too long • Integration tests vs Unit tests • SOA, distribution Wednesday, June 22, 2011
- 19. Fixing False Positives • Re-run tests several times on a failure • Report continually failing tests • Fix continually failing tests • Rely less on 3rd parties • Mock/Dingus Wednesday, June 22, 2011
- 20. Maintaining Coverage • Raise awareness with reporting • Fail/alert when coverage drops on a build • Commit tests with code • Coverage against commit di for untested regressions • Drive it into your culture Wednesday, June 22, 2011
- 21. Speeding Up Tests • Write true unit tests • vs slower integration tests • Mock 3rd party APIs • Distributed and parallel testing • http://github.com/disqus/mule Wednesday, June 22, 2011
- 22. Mule • Unstable, will change a lot • Mostly Django right now • Generic interfaces for unittest2 • Works with multi-processing and Celery • Full XUnit integration • Simple workflow • mule test --runner="python manage.py mule --worker $TEST" Wednesday, June 22, 2011
- 23. Deploy (finally) Wednesday, June 22, 2011
- 24. How DISQUS Does It • Incremental deploy with Fabric • Drop server from pool • Pull in requirements on each server • Isolated virtualenv’s built on each server • Push server back online Wednesday, June 22, 2011
- 25. How You Can Do It # fabfile.py from fabric.api import * def deploy(revision): # update sources, virtualenv, requirements # ... # copy ``current`` to ``previous`` run('cp -R %(path)s/current %(path)s/previous' % dict( path=env.path, revision=revision, )) # symlink ``revision`` to ``current`` run('ln -fs %(path)s/%(revision)s %(path)s/current' % dict( path=env.path, revision=revision, )) # restart apache run('touch %(path)s/current/django.wsgi') Wednesday, June 22, 2011
- 26. How YOU Can Do It (cont.) # fabfile.py from fabric.api import * def rollback(revision=None): # move ``previous`` to ``current`` run('mv %(path)s/previous %(path)s/current' % dict( path=env.path, revision=revision, )) # restart apache run('touch %(path)s/current/django.wsgi') Wednesday, June 22, 2011
- 27. Challenges • PyPi works on server A, but not B • Scale • CPU cost per server • Schema changes, data model changes • Backwards compatibility Wednesday, June 22, 2011
- 28. PyPi is Down • http://github.com/disqus/chishop Wednesday, June 22, 2011
- 29. Help, we have 100 servers! • Incremental (ours) vs Fanout • Push vs Pull • Twitter uses BitTorrent • Isolation vs Packaging (Complexity) Wednesday, June 22, 2011
- 30. SQL Schema Changes 1. Add column (NULLable) 2. Add app code to fill column 3.Deploy 4.Backfill column 5. Add app code to read column 6.Deploy Wednesday, June 22, 2011
- 31. Updating Caches • Have a global version number • CACHE_PREFIX = 9000 • Have a data model cache version • sha1(cls.__dict__) • Use multiple caches Wednesday, June 22, 2011
- 32. Reporting Wednesday, June 22, 2011
- 33. It’s Important! Wednesday, June 22, 2011
- 34. <You> Why is mongodb-1 down? <Ops> It’s down? Must have crashed again Wednesday, June 22, 2011
- 35. Meaningful Metrics • Rate of tra c (not just hits!) • Business vs system • Response time (database, web) • Exceptions • Social media • Twitter Wednesday, June 22, 2011
- 36. Standard Tools Nagios Graphite Wednesday, June 22, 2011
- 37. Using Graphite # statsd.py # requires python-statsd from pystatsd import Client import socket def with_suffix(key): hostname = socket.gethostname().split('.')[0] return '%s.%s' % (key, hostname) client = Client(host=STATSD_HOST, port=STATSD_PORT) # statsd.incr('key1', 'key2') def incr(*keys): keys = [with_suffix(k) for k in keys]: client.increment(*keys): Wednesday, June 22, 2011
- 38. Using Graphite (cont.) (Tra c across a cluster of servers) Wednesday, June 22, 2011
- 39. Logging • Realtime • Aggregates • History • Notifications • Scalable • Available • Metadata Wednesday, June 22, 2011
- 40. Logging: Syslog ✓ Realtime x Aggregates ✓ History x Notifications ✓ Scalable ✓ Available x Metadata Wednesday, June 22, 2011
- 41. Logging: Email Collection ✓ Realtime x Aggregates ✓ History x Notifications x Scalable ✓ Available ✓ Metadata (Django provides this out of the box) Wednesday, June 22, 2011
- 42. Logging: Sentry ✓ Realtime ✓ Aggregates ✓ History ✓ Notifications ✓ Scalable ✓ Available ✓ Metadata http://github.com/dcramer/django-sentry Wednesday, June 22, 2011
- 43. Setting up Sentry (1.x) # setup your server first $ pip install django-sentry $ sentry start # configure your Python (Django in our case) client INSTALLED_APPS = ( # ... 'sentry.client', ) # point the client to the servers SENTRY_REMOTE_URL = ['http://sentry/store/'] # visit http://sentry in the browser Wednesday, June 22, 2011
- 44. Setting up Sentry (cont.) # ~/.sentry/sentry.conf.py # use a better database DATABASES = { 'default': { 'ENGINE': 'postgresql_psycopg2', 'NAME': 'sentry', 'USER': 'postgres', } } # bind to all interfaces SENTRY_WEB_HOST = '0.0.0.0' # change data paths SENTRY_WEB_LOG_FILE = '/var/log/sentry.log' SENTRY_WEB_PID_FILE = '/var/run/sentry.pid' Wednesday, June 22, 2011
- 45. Sentry (demo time) Wednesday, June 22, 2011
- 46. Wrap Up Wednesday, June 22, 2011
- 47. Getting Started • Package your app • Ease deployment; fast rollbacks • Setup automated tests • Gather some easy metrics Wednesday, June 22, 2011
- 48. Going Further • Build an immune system • Automate deploys, rollbacks (maybe) • Adjust to your culture • CD doesn’t “just work” • SOA == great success Wednesday, June 22, 2011
- 49. DISQUS Questions? psst, we’re hiring [email protected] Wednesday, June 22, 2011
- 50. References • Gargoyle (feature switches) https://github.com/disqus/gargoyle • Sentry (log aggregation) https://github.com/dcramer/django-sentry (1.x) https://github.com/dcramer/sentry (2.x) • Jenkins CI http://jenkins-ci.org/ • Mule (distributed test runner) https://github.com/disqus/mule code.disqus.com Wednesday, June 22, 2011