Graphene – Microsoft SCOPE on Tez

Editor's Notes

• #2: We are here to talk about how we are looking to power SCOPE with Tez.
• #3: We will do a quick overview of Cosmos and SCOPE Then we will talk about the role job manager plays in the system How we are looking to fix some of the problems that we have by leveraging Tez Anupam will dig a little deeper into design of Graphene He will be talking about the challenges in front of us and why we need your help to take Tez to the next level
• #4: A Microsoft-internal platform for building big-data applications​ ​ Used across Microsoft by Bing, Azure, Windows, Office for data mining and analysis. Available externally as Azure Data Lake Analytics Lets user focus on transforming data to gain insights while we focus on operating the platform at lower COGS.
• #6: SCOPE is the main scripting language for Cosmos. Targeted for large-scale data analysis. You could run a script over 1GB, TB, or a PB and we handle scaling that. SQL like language that allows C# and .NET devs to get started easily. On the right is a sample SCOPE script. In this case we are reading a TSV file and running a select statement on that, add a new column, and output that as a new file. Users can easily define their own functions and even implement their own versions of operators like extractors, processors, and outputters. Users just write the scripts thinking it is going to run on a single machine and we scale it out on the cluster. This means that the nitty-gritties of dealing from failures and retries is not something a user should worry about
• #7: Users submit a SCOPE script from VS using Scope studio plugin. The script goes through Cosmos Job Service and FE where it is compiled by Scope compiler. Compiler produces an AST representation of the script along with the codegen DLLs for user code and other artifacts. Optimizer makes decisions about execution plan, parallelism, and generates an algebra. Job manager, which is us, parses the algebra and starts executing the DAG on the cluster. As part of the execution JM launches Scope engine on the tasks which provides implementations of many standard physical operators. JM gives the Scope engine input paths to read and outputs to produce. Typically outputs of one vertex become input to some other vertex and DAG execution continues. --- SCOPE compiler and optimizer are responsible for generating an efficient execution plan and the runtime.
• #9: So what are the responsibilities of the Job Manager? DAG execution JM is the central and coordinating process for all processing vertices within an application. The primary function of the JM is to construct the runtime DAG from the compile time representation of a DAG and execute over it. The JM schedules a DAG vertex onto the cluster nodes when all the inputs are ready. JM can also do dynamic updates to the graph like a pod level aggregation or build a broadcast tree. Fault tolerance The Job Manager monitors progress of all executing vertices. Failing vertices are re-executed a limited number of times and if there are too many failures, the job is terminated. JM also detects slower tasks in a vertex and reexecutes them elsewhere on the cluster.
• #10: Scheduling When a task is ready then JM looks for a machine in the cluster to run the task upon. The global cluster load information used by each JM is provided through the cooperation of two additional entities in the system: a Resource Monitor (RM) for each cluster and a Process Node (PN) on each server. The RM aggregates load information from PNs across the cluster continuously, providing a global view of the cluster status for each JM to make informed scheduling decisions. It also enforces token limits.. Users typically give a job some tokens to run. Each token amounts to 2 cores and 6GB. JM ensures that the resources used by the job never exceed the allocated number of tokens.
• #11: When the job finishes then the JM finalizes the outputs so they become visible to the user. It also supports some custom metadata operations like catalog updates.
• #14: Go through details. Explains design and implementation decisions for Graphene and how we use Tez
• #15: 1m Once we decided to implement SCOPE AM using Tez. We decided upon certain ground rules or guiding principles we would use to accomplish this goal. Hitesh already gave us an idea about the scale of Cosmos and SCOPE workloads, and how critical they are for Microsoft’s business. The compiler, optimizer, execution engine and tooling will be minimally changed, in order to allow for a staged transition. Tez has very powerful set of APIs to allow any system to plugin. We will be using these extensibility points as much as possible. Finally, for features that we feel the need to add to Tez, we will be working with the community and making them work generally for all Tez users as much as possible. With these ground rules set we started working on porting Scope to run on top of Tez.
• #16: 3m The need to seamlessly upgrade from current job manager to graphene implies that graphene should be a drop-in replacement for current job manager. As Hitesh showed, doing this at Cosmos scale while being the backbone of Microsoft’s analytics need implies least perturbation. This meant that the SCOPE AM on Tez had to mimic existing job manager kind of behavior. Graphene has 4 unique integration point in Cosmos SCOPE stack not native to Tez. This introduction of our guiding principles and integration points will be helpful to understand our implementation and the rationale behind our design choices.
• #17: 5m
• #18: 6m
• #19: 7m
• #20: 8m
• #21: 10m Bring learnings from Job Manager back to Tez.
• #22: 12m
• #23: 13m
• #24: 15m
• #25: 16m
• #26: 16m