Tricks & challenges developing a large Django application
25 likes13,978 views
The document discusses tricks and challenges for developing large Django applications. It outlines three tricks: 1) Signing data to pass through untrusted sources without tampering; 2) Adding a cache version field to models to invalidate cached queries; 3) Using NoSQL databases like Redis and Solr for denormalizing data to allow flexible queries across entities. Redis sets are used to track relationships like followers and event attendees for recommendations.
![Lanyrd.com
./manage.py push_static
Minifies JavaScript and CSS
Renames files to include sha1(contents)[:8]
Pushes all assets to S3](https://image.slidesharecdn.com/tricks-and-challenges-110622100732-phpapp02/85/Tricks-challenges-developing-a-large-Django-application-29-320.jpg)
Tricks & challenges developing a large Django application
- 1. Tricks & challenges developing a large Django application Simon Willison EuroPython 2011, 22nd June http://lanyrd.com/sfwcq
- 2. Lanyrd.com
- 3. Lanyrd.com Definitive database of Professional Events and Speakers
- 4. Lanyrd.com Definitive database Social event recommendation of Professional Events Event networking tools and Speakers Archive of slides, notes and video
- 7. Lanyrd.com Pass data through an untrusted source with confidence that it hasn't been tampered with
- 8. Lanyrd.com Signing uses Unsubscribe" links in emails lanyrd.com/un/ImN6VyI.ii0Hwm7p71DEcGfaVzziQaxeuu Signed cookies "You are logged in as simonw" without hitting the database
- 9. Lanyrd.com Signing in Django 1.4 from django.core import signing signing.dumps({"foo": "bar"}) signing.loads(signed_string) response.set_signed_cookie(key, value...) request.get_signed_cookie(key)
- 11. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Django conferences Django Django events looking for participants coverage 1 Django event is looking for participants 52 videos Most recent added 3 weeks ago ON NOW EuroPython 2011 52 slide decks 19 Most recent added 4 Italy / Florence hours ago 19th–26th June 2011 Django Plone Pyramid Python Twisted 3 audio clips Most recent added 1 week ago 27 write-ups SEPTEMBER DjangoCon US 2011 6 Most recent added 1 2011 United States / Portland 6th–8th September 2011 week ago 11 handouts Django Open Source Python Most recent added 18 hours ago 17 PyCON FR 2011 3 notes France / Rennes Most recent added 10 17th–18th September 2011 hours ago Django Python By country OCTOBER PyCon DE 2011 Ireland 1 4
- 12. Lanyrd.com class Conference(models.Model): ... cache_version = models.IntegerField(default = 0) def save(self, *args, **kwargs): self.cache_version += 1 super(Conference, self).save(*args, **kwargs) def touch(self): Conference.objects.filter( pk = self.pk ).update( cache_version = F('cache_version') + 1 )
- 13. Lanyrd.com {% cache 36000 conf-topics conference.pk conference.cache_version %} <ul class="tags inline-tags meta"> {% for topic in conference.topics.all %} <li><a href="{{ topic.get_absolute_url }}">{{ topic }}</a></li> {% endfor %} </ul> {% endcache %}
- 14. Lanyrd.com Bulk invalidation from django.models import F topic.conferences.all().update( cache_version = F('cache_version') + 1 )
- 15. Trick #3: NoSQL for denormalisation
- 16. Lanyrd.com NoSQL = Not Only SQL
- 17. Lanyrd.com But we like joins! Slides from a session on Lanyrd show up... Under "coverage of this conference" Under "slides by Simon Willison" Under "slides about Django"
- 18. Lanyrd.com Solution: denormalise data to Redis and Solr
- 19. Lanyrd.com Commands Clients Documentation Community Download Issues Redis is an open source, advanced key-value store. It is often What people are saying referred to as a data structure server since keys can contain Comparison of CouchDB, Redis, MongoDB, Casandra, Neo4J & strings, hashes, lists, sets and sorted sets. strings hashes lists sets others http://j.mp/l32SqM via @DZone Learn more → @__NeverGiveup Oh YAY, oui tu me redis ! *-* Hm, on s'rejoint à Try it Download it 14h au bahut ? :o Ready for a test drive? Check this interactive Redis 2.2.10 is the latest stable version. JE L REDIS JE FOLLOW BACK SUR @Fuckement_TL tutorial that will walk you through the most Interested in legacy or unstable versions? important features of Redis. Check the downloads page. une question : "How to use ServiceStack Redis in a web application to take advantage of pub / sub paradigm" http://t.co/EOgyLU1 #redis #web Nice - Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Membase vs Neo4j comparison http://bit.ly/l32SqM from @kkovacs More... Sponsored by This website is open source software developed by Citrusbyte. The Redis logo was designed by Carlos Prioglio.
- 20. Lanyrd.com Our redis sets... simonw-follows:{144,21345,12328...} europython-attendees:{344,21345,787...} redis.sinter( 'simonw-follows', 'europython-attendees' )
- 21. add a conference you are signed in as simonw, do you want to sign out? Lanyrd.com calendar conferences coverage profile search EuroPython 2011 You're speaking The European Python Conference AT THIS EVENT 19 –26 JUNE 2011 Florence in Italy 97 attending http://ep2011.europython.eu/ @europython PEOPLE View the schedule on Lanyrd #europython 80 tracking PEOPLE Save to iCal / iPhone / Outlook / lanyrd.com/ccdpc (short URL) GCal TELL YOUR FRIENDS! Tweet about this event 119 speakers Andreas Alan Anna Schreiber Franzoni Ravenscroft Topics @onyame @franzeur Django Andrew Alessandro Anselm Kruis Godwin Dentella Plone @andrewgodwin Pyramid Andrii Alex Martelli Antonio Cuni @antocuni Python Mishkovskyi @mishok13 Twisted Ali Afshar Armin Rigo Armin Edit topics
- 22. Lanyrd.com apache > lucene > solr Search the site with Solr Search Main Wiki Powered by Lucid Imagination Last Published: Sat, 04 Jun 2011 12:23:42 GMT About Welcome Who We Are Welcome to Solr Documentation PDF Resources What Is Solr? Related Projects Get Started News May 2011 - Solr 3.2 Released March 2011 - Solr 3.1 Released 25 June 2010 - Solr 1.4.1 Released 7 May 2010 - Apache Lucene Eurocon 2010 Coming to Prague May 18-21 10 November 2009 - Solr 1.4 Released 20 August 2009 - Solr's first book is published! 18 August 2009 - Lucene at US ApacheCon 09 February 2009 - Lucene at ApacheCon Europe 2009 in Amsterdam 19 December 2008 - Solr Logo Contest Results 03 October 2008 - Solr Logo Contest 15 September 2008 - Solr 1.3.0 Available 28 August 2008 - Lucene/Solr at ApacheCon New Orleans 03 September 2007 - Lucene at ApacheCon Atlanta 06 June 2007: Release 1.2 available 17 January 2007: Solr graduates from Incubator 22 December 2006: Release 1.1.0 available 15 August 2006: Solr at ApacheCon US 21 April 2006: Solr at ApacheCon
- 23. Lanyrd.com add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Search We found 467 results for “django” FILTER BY django Search type Sessions 209 Topic: Django Coverage 190 ON NOW 1 conference FUTURE 5 conferences Conferences 60 PAST 51 conferences Books 5 Topic: GeoDjango Topics 2 PAST 1 conference People 1 Practical Django Projects By James Bennett PUBLISHED June 2009 FILTER BY past or future Django Geek Past events 55 @djangeek Future events 5 SLIDES
- 24. Lanyrd.com add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Your contacts' calendar yours 24 contacts 182 Simon We've found 182 conferences your Twitter contacts are Willison interested in. Your profile page TODAY Café Scientifique: Exploring Attend 21 the dark side of star Track formation with the Herschel From our blog Space Observatory Welcoming Sophie United Kingdom / Brighton Barrett to team 21st June 2011 Lanyrd Astronomy Science Today we have a very special announcement (and for once, 4 contacts tracking it's not a new feature!) We would like to welcome the super-wonderful Sophie Barrett to the Lanyrd team. 21 Usability Professionals' Attend Session schedules in Association – International Track your calendar Conference You can now subscribe to event schedules in your calendar of United States / Atlanta
- 25. CSS Maintainable CSS Maintainability @Natbat on Twitter 1 write-up Design Process Internet Explorer Debugging Lanyrd.com Added 7 months ago 16 CONFERENCES spoken at Show and tell Personal Projects 2 links Most recent added 7 14 CONFERENCES Web Standards Crowdsourcing months ago involved with SEE ALL COVERAGE WildlifeNearYou RSI Community 11 items in total Event Planning Geek Night User Experience Browse by year Usability Usability Testing JavaScript FAVOURITE COVERAGE See 6 items that Natalie Downe has 2011 ! 2010 ! 2009 ! Progressive Enhancement favourited 2008 ! 2007 ! 2006 ! 2005 Unobtrusive JavaScript Flexible design Planning Artifacts Social Software Book credits Appears with TECHNICAL REVIEWER Motivation CSS3 CSS Layout The Ultimate CSS Simon Willison Reference Project Management Dev Fort 9 times TECHNICAL EDITOR Rapid Application Development CSS Mastery Nigel Crawley 5 times Reinier Zwitserloot Full speaking history for Natalie Downe 4 times PAST CONFERENCES FUTURE CONFERENCES Neil Crosby 4 times spoken at speaking at Cristiano Betta Five Pound App #22 London Insites Tour 4 times Brighton, United Kingdom Evening Matt Westcott 30th March 2010 London, United Kingdom 4 times 19th July 2011 CSS3 Wizardry Gareth Rushgrove Workshop 4 times FUTURE CONFERENCES Brighton, United Kingdom 29th January 2010 attending
- 26. Trick #4: Hashed static asset filenames in CloudFront
- 28. Lanyrd.com Benefits Far futures expiry headers Cache-Control: max-age=315360000 Expires: Fri, 18 Jun 2021 06:45:00 -0000 GMT Guaranteed updated CSS in IE Deploy new assets in advance of application Old versions stick around for rollbacks
- 29. Lanyrd.com ./manage.py push_static Minifies JavaScript and CSS Renames files to include sha1(contents)[:8] Pushes all assets to S3
- 31. Challenge #1: HTTP requests
- 32. Lanyrd.com Talking to an API? What if it fails to load? What if it takes 30 seconds to load?
- 33. Lanyrd.com A user gives you a URL... urllib.urlopen(url).read() ? what if it's a huge file? what if it's a slow loading tarpit? what if it's the URL to your private, firewalled Solr instance? http://10.0.1.1/solr/select/?q=secrets
- 34. Lanyrd.com Safe URL consumption Connection timeouts Logging and Profiling Host validation HTTP caching / if-none-match / etc django.httpclient ?
- 35. Challenge #2: Profiling and debugging production systems
- 36. Lanyrd.com Debugging in development rocks! django_debug_toolbar = awesome assert False = instant useful 500 page import pdb; pdb.set_trace() adds a command line debugger to your running application!
- 37. Lanyrd.com DEBUG = False DEBUG = "BLIND"
- 38. Lanyrd.com from django.views.debug import technical_500_response import sys class UserBasedExceptionMiddleware(object): def process_exception(self, request, exception): if request.user.is_superuser: return technical_500_response(request, *sys.exc_info())
- 39. Lanyrd.com mysql-proxy Very handy lua-customisable proxy for all of your MySQL traffic Worst documented software ever log.lua - logs out ALL queries https://gist.github.com/1039751
- 40. Lanyrd.com django_instrumented (Unreleased) code I wrote for Lanyrd Collects various runtime stats about the current request, stashes a profile JSON in memcached Writes out the profile UUID as part of the HTML A bookmarklet to view the profile
- 41. Lanyrd.com
- 42. Lanyrd.com Django improvements DEBUG as a global setting is an anti- pattern More low-level hooks for measuring, well, pretty much everything Live profiling tools (as a third-party product)
- 44. Challenge #3: Zero downtime deploys
- 45. Lanyrd.com Challenge: deploy upgrades, including database upgrades, without downtime
- 46. Lanyrd.com Deploy DB changes separately from code changes Make backwards compatible schema changes We deploy code to a none-web server just so we can run "./manage.py migrate" (you are using South, right?) Use symlinks for instant rollback to previous code
- 47. Lanyrd.com Read-only mode If you can flip your site in to read-only mode, upgrades become a lot easier Flip to read only mode Replicate DB to a new machine Run migrations, then test Switch app servers to new database (On EC2, fire up an entire new web cluster) Feature flags: finely grained version of this
- 48. Lanyrd.com The lesson we keep on learning
- 49. Lanyrd.com Global settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0)
- 50. Lanyrd.com Global settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0) DEBUG TIME_ZONE Middleware?
- 51. Lanyrd.com Settings should be modifiable at run-time
- 52. Thank you! http://lanyrd.com/sfwcq [ [ We're hiring a developer/web operations person in London, come and talk to us!