Seda an architecture for well-conditioned scalable internet services

Editor's Notes

• #3: Internet hits translate to several I/O and network requests -> enormous load on underlying resources As of 2001: Yahoo gets 1.2 billion hits per day, AOL gets over 10 billion page views per day
• #5: SEDA = Staged Event-Driven Architecture
• #6: Static -> dynamic: Extensive computation and I/O Changing service logic: increasing engineering complexity and deployment General purpose platforms: Not specially engineered platforms
• #7: Replication – cannot scale to orders of magnitude Traditional OS and concurrency – brittle under large loads Traditional OS focuses on giving providing transparency through virtual machines. Thread switching has high overhead, threads have large memory overhead. Internet applications need massive concurrency and better control over resource usage … better control makes a big difference at the margin of excessive load.
• #8: Stages are robust building blocks subject to thresholding and filtering according to load Allows informed scheduling and resource-management decisions, including request reordering, filtering, and aggregation. Dynamic resource throttling allows control over resource allocation and scheduling of components Demonstration: High performance HTTP server (… remember CSE222a?? … multiple threads vs Select statement?)
• #9: Well-conditioned: output latency is determined by the length of the queue/pipeline This property holds regardless of the number of stages in the pipeline, subject to queueing/dequeing times In a non-pipelined design, clients wait for entire operation to complete before moving on to next one.
• #12: Overheads: cache and TLB misses, scheduling overhead, and lock contention Threads = multiprogramming … virtualization hides global resource management SPIN, Exokernel, Nemesis = examples of OS attempts to solve this
• #13: Apache, IIS, Netscape Enterprise Server, BEA Weblogic, IBM WebSphere Bad case: cached static pages (cheap) vs large pages not in cache (expensive)
• #14: Flash, thttpd, Zeus, JAWS Complex scheduler … hard to maintain … complex FSM maintenance, too … modularity difficult to achieve Needs helper threads that do blocked I/O Has well-conditioned, graceful degradation --
• #16: Sets of event queues … promote modularity Seems to be a generic class of solutions
• #17: Support massive concurrency = event-driven execution where possible Simplify construction of well-conditioned services = shields the application from details of scheduling and resource management … supports modular construction, support for debugging and profiling Enable introspection = applications analyze the stream to adapt behavior to load … prioritize and filter services to support degraded service under load Support self-tuning resource management = tune resource parameters to load … e.g., allocate threads to a stage based on load instead of hard-coding it apriori
• #18: Threads pull events from queue, schedule events on downstream queues, and waits for more Controller adjusts resource allocation and scheduler dynamically Note that threads are stage resources, not task resources. Thread count can be dynamically allocated based on load. Stages can run in serial or parallel, or both. Event handler can implement its own scheduling policy irrespective of how the system filled the queue
• #19: Set of stages separated by queues Private thread pool per stage Each state can be independently managed Stages can be run in serial or parallel Each stage can be independently load conditioned … more threads for heavy loads (thresholding) Important point: queues can be finite, which means that a stage can fail to queue an event … meaning next stage is very busy. Stage can make a decision to block (backpressure), or drop event (load shedding) and take some remedial action Question: should two modules communicate via method call or queue? A queue system promotes modularity, isolation, and load management … cost: latency Important point: debugging, billing, memory usage, and queue profiling can occur by attaching processors between stages and queues
• #21: Thread pool controller Adjusts the number of threads executing within each stage Periodically examines queue and adds threads if queue exceeds some threshold … or removes them if they’re idle for some amount of time Batching controller Adjusts the number of events processed by each invocation of the event handler (i.e., a batching factor) This increases throughput due to cache locality Tries to strike a balance … large batching factors can degrade *overall* performance … so try to select a small batching factor … kind of like a PLL *** Controllers are a great way of enforcing performance policy ***
• #22: This is for the Haboob web server Thread pool adjusted based on length of corresponding queue Queue length sampled every 2 seconds, thread added to pool if queue exceeded 100 entries … max 20 threads Threads are removed from the pool idle for more than 5 seconds AsyncFile used a threshold of 10 queue entries to exaggerate the behavior
• #23: Shows single stage generating events at an oscillating rate When the output rate increases, controller decreases batching factor … and visa versa *** The point: controllers allow application to adapt to changing conditions regardless of the particular algorithms used by the operating system … or the application ***
• #25: They’re eating their own dog food. Completed I/O goes back into the next stage’s queue
• #26: Clients issue bursts of 8KB packets Server responds with ACK for every 1000 packets Assume GB ethernet and all Linux boxes Slight degradation for SEDA is due to non-scalability in Linux network stack Threaded implementation stops at 512 connections due to Linux thread limitations SEDA implementation had 120 threads to handle socket writes.
• #29: Apache and Flash often perform faster, but the tails are fierce. Haboob gives consistent response time under load
• #31: Highly Scalable!