OtherScreen Architecture

Editor's Notes

• #2: \n
• #3: \n
• #4: To learn lessons, it’s vital to identify engineers or scientists that you can learn them from.\n\nIn my case, Ben Black, CEO of Boundary, is such a scientist. He has a track record, sense of humor, and the ability to describe difficult concepts in a way I can understand. The message he carries is, in many cases, grim.\n\nHere he shares the truism that traffic will break every web application, there are no exceptions. (By the way, Richard McDougall is Application Infrastructure CTO at VMware, a $40b company.)\n
• #5: Depending on the size of your intended audience, the story gets even worse.\n\nThe world of open source software is a wonderful and amazing world, and there have never been so many tools freely available to engineers working on difficult problems. At the same time, the cutting edge has never been so sharp.\n
• #6: If you were at my last Skookum Tech Talk, I outlined some of the challenges of today’s Ruby community. Carl Lerche, co-author of bundler, one of the libraries I discussed in that talk, likened a recent tiff to “arguing what color to pain the door when half the house is collapsing from structural damage.”\n\nProgrammers can be so dramatic!\n
• #7: So here’s my over-dramatization!\n\n\n\n
• #8: So that’s all been pretty much a downer. What can I do about it?\n\nI believe that accepting into your job some elements of technical leadership is important. If you are a developer, you should be thinking about being a senior developer. If you are a senior dev, start looking at managing risk across multiple projects or products as a director.\n
• #9: Execution risk: you won't be able to build what you want\nVision risk: you won't build the desired thing\nAdoption risk: you can't attract customers to what you've built\nSupport risk: you will write something you can't keep running\nMaintainability risk: your code will reach a point where you can't change it easily\nVendor risk: you partner with the wrong people\n\n
• #10: Usage patterns encompass everything about the users’ behavior. Times of day, concentration of actions, are they reading or writing, do they cause reads or write to happen, which of their messages need to be broadcast, how many items are we talking about, what pages query or sort on which data, etc.\n\nYour tools are everything that you’ll use in your runtime environment to provide the user experience.\n\nConsidering them together is the fun part. I’ll now use this framework to talk about what we’ve done at Otherscreen to architect the experience you’ll have during the demo.\n
• #11: Heroku helps us address a risk. Because we aren’t spending time on maintaining and setting up the runtime environment, we can spend that time on writing the application. That’s direct treatment.\n
• #12: The selection of Heroku as a vendor opens Otherscreen to other risks. EC2 is prone to large outages, and the most recent major outages have both had isolated reports of data loss. At this time in the application’s lifetime, we think this exposure is worth the risk.\n\nHeroku has been hosting Rails applications as long as anyone, and our application is not challenging to host there. We are comfortable with their profile, and it doesn’t hurt that I have a couple of phone numbers over there...\n
• #13: The mobile experience is a vital part of our product. We anticipate high levels of mobile traffic from those enjoying our product at sports bars or their hosts’ living rooms. The decision about whether to write a native client or use the mobile web was very relevant for us.\n\nAt this time, we’re going to go with the mobile web, because of the better risk profile for us. We evaluated a middle path, but chose not to go that direction at this time. We’ll get into why we have the ability to revisit this decision more easily, later in the life of our company, later in this presentation.\n\n\n
• #14: We need to get our questions to end users in real time, and there could be a lot of them. Mass broadcast strategies can be tricky and are hard to get right. The “easy” ways to implement mass broadcast can introduce latency, poor performance, or worse. \n\nWe’re using HTML 5‘s WebSockets and Pusher to accomplish this. This offloads a core technical concern to a vendor, a decision that should never be made lightly.\n\nFor Otherscreen, however, it allows us to defer a significant development effort because of the difficulty of the task. By allowing Pusher to do what they are good at, especially for now, we’re also ensuring we can work on developing a product we can market instead of something that simply works.\n
• #15: That decision isn’t without risk. WebSockets are not yet widely implemented, and there have been recent setbacks with the protocol that are only now getting addressed by implementors.\n\nPusher is also a small startup themselves, and we project as if we might become their biggest client sooner rather than later. Bryan Helmkamp, CTO at Efficiency 2.0, shared some great advice with me about vendors: “Never be the first, the last, the biggest, or the smallest.” I think that’s great advice. We mitigated some of this risk by reaching out to them and beginning conversations this early in the process, before our mass-market launch.\n
• #16: A common approach to AJAX in Rails world involves shipping HTML across the wire to perform direct DOM manipulation.\n\n\n\nAs a result of this and other considerations, we decided to speak JSON as a data transport language and begin creating a real API of our own for our clients to consume. This decision has big implications, and makes these browser clients a very important part of the user experience.\n
• #17: Because of our mobile web focus, it’s difficult to maintain the same DOM for laptop and the mobile experience. Additionally, the introduction of WebSockets, with its message passing paradigm, is outside of the typical Rails engineering request-response wheelhouse. \n\n\n
• #18: As a result of this and other considerations, we decided to speak JSON as a data transport language and begin creating a real API of our own for our clients to consume. This decision has big implications, and makes these browser clients a very important part of the user experience.\n
• #19: Using backbone.js allows us to maintain “known good” model states on the client-side, aids in providing the proper interactivity for API writes, and is open to WebSocket messages updating model state and updating client appearance and interactivity. \n
• #20: As a result of making this separation, it clears up what language and system is responsible for what, which means less code and cleaner code-- always a good thing. \n
• #21: So, conceptually, we’ve dealt with getting the questions in front of clients. We run into a thorny problem when it comes to dealing with the answers, which is that running Rails can be expensive. \n
• #22: The blocking behavior of its clients means that if you anticipate spiky request patterns, you have to overprovision to ensure clients spend most of their time on the CPU rather than waiting around for a worker to service their request. Boo!\n
• #23: Here is where our decision to commit to an API early also helps. The technical decision on how to service individual API endpoints can be made later. We can use Sinatra, EventMachine, or even another language that Heroku supports like Clojure or node.js to address our concurrency problems at the application layer.\n\nKnowing what your runtime is good at and where it struggles is one of the most important things to understand as an architect, whether you have that title or not.\n
• #24: Let’s walk through what happens when play progresses and we now know the correct answer to a question we’ve posed. We’ve got a lot of answers that we need to mark as correct, scores to update, accuracy ratings to recalculate, etc.\n\nWe do this sort of work in the background, outside of the request-response lifecycle. This has positive effects on both the codebase and the user experience.\n
• #25: And, for our particular use case, the gifts of deferred work can keep on giving.\n\nAfter the work of doing the scoring is complete, all our clients are interested in that updated information. Fortunately, this time, when faced with a wave of client requests, we can be better prepared. The leaderboard information is the same for all users, and we can use HTTP caching to ensure that we only respond once. In our case, the background worker can do some of this work before a client even knows they’re interested in a new leaderboard.\n\n\n\n\n
• #26: The HTTP specification has a lot of goodies, and caching is one of the big ones. If you fully understand how to leverage HTTP caching, your application server need not produce the same response twice-- intermediate caches ought to be able to return everything else. Quite a challenge but an incredible payoff!\n\nIt’s trivial to see that serving every request from RAM is the most performant way to serve responses, and proxy caches are some of the most efficient open source software available in terms of RAM usage. It’s a great scalability story.\n
• #27: Thanks for listening!\n