ScaleFast Grid And Flow
1 like337 views
The document describes Scalefast's grid and flow platform, designed for efficient grid computing and parallel processing of large datasets. It outlines the platform's capabilities in centralizing business processes, automating job distribution across computers, and facilitating process orchestration with built-in schedulers and error recovery features. Case studies illustrate the platform's effectiveness in reducing processing times and resource constraints for hedge funds and banks, showcasing significant improvements in operational efficiency.
ScaleFast Grid And Flow
- 1. Grid and FlowBy Robert [email protected]
- 2. The OfferingA distributed, stable and well synchronised platform for grid computing.A methodology, toolset and library for deploying an efficient, co-ordinated and parallel platform.
- 3. Current Operational ChallengesProcessing of large datasets open to high rates of failureProcessing takes a long time to completeProcessing is often executed sequentiallyTechnology bottlenecks e.g. 32 bit software often only supports between 2 and 3 GB RAM64 bit processes can hog available resourcesMost tools can’t exploit multi core configurationsDedicated hardware allocated to accommodate the maximum processing loadDifficult to audit and trace processing problems
- 4. With ScaleFast you can ...Centralise all business processesDefine hierarchical processes with step inter-dependenciesParallelise the running of processes and stepsRe-run failed processes from any point.Automatically split large process stepsMake use of multi core/processor computersDistribute jobs across multiple computersMake use of user workstations and other idle computing resources
- 5. ScaleFast GridDistributed Computing GridA distributor and worker nodesImplements map/reduceWorkers can run on user workstations or dedicated infrastructureCan be easily deployed to a cloud platformSupports the native running of Python, Java and .Net
- 6. ScaleFast FlowProcess Workflow EngineProcesses are made up of individual jobs which have inter-dependenciesThe output of one job can be the input of the next jobProcesses have notifications based on success or failureFlow has a built in scheduler which can be triggered by:Time with multiple time zone supportUser InterfaceAPIProcesses can be restarted from any point of failureProcesses can be made up of sub processes
- 7. Common Use CasesReporting and data processingStabilising processes that fail due to resource constraintsSpeeding up processes that take a long time to runImprove and/or balance resource utilisation Process orchestration and schedulingCo-ordinating processes with event based synchronisationParameter and data flow between process stepsCentralisation and versioning of processesReducing support administration with full process auditingGeneral processing and application developmentAny application/process that would benefit from parallelismRisk Management and PL ProcessingDistributed Computations
- 8. Case Study 1 – Hedge FundAfterBeforeTrade volumes of 10 000 per day
- 10. Reporting taking longer to run each day
- 11. System support occupies a fulltime resource with additional assistance frequently required
- 12. Overnight failures push EOD processing to t+2 (SLA at t+1 am)
- 13. Fund considers:Adding head count with full time EOD support resourcesPurchasing additional hardwarePurchasing a scheduling and automation productProcess failures reduced significantly
- 14. Fine grained audit trail of all processes
- 15. EOD processing time reduced from 8 hours to 50 minutes
- 17. No additional headcount requiredCase Study 2 – Hedge FundAfterBeforeFixed Income Risk ProjectNew trading system implementation for Risk Management and P&LRequirement to take all Risk processing in houseAfter trading system implementedEOD process become more complex and onerousReports begin to failFund considers:Head count requirement in supporting new trading systemPurchasing of hardware for additional processingProcess failures reduced significantly
- 18. Fine grained audit trail of all processes
- 19. EOD processing time reduced from 4 hours to 30 minutes
- 21. Risk Engine built on top of Grid and Flow.
- 22. Scenario analysis report with 50 scenarios across 2000 positions runs in under 5 minutesCase Study 3 – BankAfterBeforeKey EOD reports failing due to resource constraints
- 23. Tried shell scripts to split reports
- 25. Reports still took 7 hours to run and taking longer to run each day
- 26. A single failure required a complete restart
- 27. A failure and restart would result in SLA failures to all downstream systems
- 28. Bank considers:Purchasing additional hardwareRe-assessing support requirementsProcess failures reduced significantly
- 29. On failure, reporting process can now resume from any step
- 30. Completed reports are now processed in 40 minutes
- 31. Fine grained audit trail of process to aid support staff
- 32. Hardware freed up for other projectsOther UsesMonte Carlo framework for pricing exotic structured credit instrumentsRisk Management ProcessingGeneral application processingProcess synchronisationLoading and parsing large datasets
- 33. SCALEFAST ArchitectureFlow stores, versions and schedules workflows which are predefined and synchronised grid jobs.Grid Clients are any processes able to submit grid jobs.Grid Distributor receives job requests and maps the reduced jobs as tasks across workers.Grid Workers request and process job tasksFlowGrid Client 0 Grid Client ...Grid Client QGrid DistributorWorker 0Worker ...Worker XServer 0RunnerRunnerRunnerServer ...Server NLocal diskShared Storage
- 34. Flow GUIA simple process example with 3 stepsProcesses can have multiple branch dependencies e.g. 1 to many and many to 1Processes can be build up from sub processesFlow highlights the status of the individual stepsBy clicking on a step, you are redirected to the Grid for further details.Processes can be paused and restarted.On a processes failure, it can be restarted at any step in the process.
- 35. Grid Job Details GUIParameters, status and details of a grid jobIndividual tasks can be drilled down intoStderr and Stdout out can be accessed and queried across all tasksInput parameters, context and output visible at job or task level
- 36. Grid Summary GUIHigh level view on the Grid status and activityView active worker nodesView job activity and history
Editor's Notes
- #4: Scheduling Tools: Active Batch
- #5: Issues in the top grouping can be addressed by tools like Active Batch
- #11: Support staff can now see in very granular process details where process failed.It is no easier to determine the cause of a failure, was it: Resource issues Bad static data Bugs in code
- #13: There is also an instance of flow that can be easily integrated into an Enterprise Workflow/Automation Application