(ATS3-PLAT08) Optimizing Protocol Performance
Download as PPTX, PDF2 likes532 views
This document discusses optimizing protocol performance in Pipeline Pilot. It recommends refactoring protocols to identify bottlenecks, profiling components to see where time is spent, optimizing data access by only retrieving necessary data and caching when possible. It also suggests parallelizing computationally intensive tasks, using job pooling to optimize server performance, and leveraging the scalability of the Accelrys Enterprise Platform. Applying these basic optimization principles can significantly improve protocol execution times.
(ATS3-PLAT08) Optimizing Protocol Performance
- 1. (ATS03-PLAT08) Optimizing Protocol Performance Andrew LeBeau Eddy Vande Water Advisory Product Manager Director, EMEA Field Application [email protected] [email protected]
- 2. The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision.
- 3. Agenda • Profiling and Refactoring • Data Access • Data Computing • Others key T&T • Server optimization • Summary
- 4. Protocol Refactoring • Consider the first version (V1) of a “complete” protocol…perhaps ~30 components • Protocol building is typically an incremental process with much iterative design. Therefore, completion of V1 represents the documentation of an intellectual process • However, very significant optimizations can be achieved by reviewing V1 and considering major (perhaps complete) refactoring of the protocol, using the knowledge developed from building V1
- 5. Component Profiling • Identify protocol bottlenecks • Ctrl+T to toggle between options – Absolute compute time (sec) – Compute time as percentage of total execution time
- 6. Demo: Protocol version 01 Protocol development flow • Get big file with activity on several targets and lot of other props. • Need to pivot data • Only interested by one target • Need structure • Join my activity • Compute new property • Need additional data from db • Only interested by a range of data • Create nice report
- 7. Demo: Protocol version 02 Why? Because I used some simple principles! 28 seconds instead of 6 minutes!
- 8. Data Access • Keep the records as small as possible to do what you need. Don’t read in things just because they are there in the file; only read it what you will use! Don’t pass anything further down the pipeline than it is needed. • If writing to disk to pass information between pipelines, caches are faster than delimited text (or any other file).
- 9. Merge / Join / Group / Sort / Cluster / etc. • All create implicit caches • Filter before merging/caching • Reduce the number of properties • Merge on a sub-stream then join back • Sort before join – on the primary key • Cache Writer: Use Pre-Index options if the cache will later be joined on
- 10. Database • Database access should be tuned: – See • (ATS3-PLAT04) Database Connectivity for Application Development • (ATS2-23) Managing Data Source Connections – PP should be located close to the database server – Join in the database if possible – Use batch inserts, etc. – Use batches with the SQL Select for Each Data
- 11. When and Where to Calculate Properties • Think about the order you need to do things • Compared with…
- 12. Parallel Processing in Subprotocols • Allows parallelization of computationally intensive tasks • Need to pay attention to batch size – don’t make it too small – Performance can be almost linear with number of cores (our numbers and customers’) • Can be problematic for subprotocols using R, or other external apps
- 13. Demo: Protocol version 2.0
- 14. Others key T&T • Prefer linear pipelines – Most efficient memory usage • Avoid excessive branching – Branching pipes causes data cloning. This can be expensive for large data records • Avoid hash tables as caches – Use a file cache • Reduce usage of caches and caching components – Merge, Group, Sort and Cluster create unseen caches – Be mindful of children nodes
- 15. Others key T&T ctd. • General relative speed of implementations: – Components >= Pilot Script >= Java • Protocol Function – Use AJAX to call a protocol within a page – Can provide better performance if only needs to update part of a report • Be careful! – Run To Completion (RTC) subprotocol can slow down protocol execution: Use sparingly… – Check point are very good to debug but should not be kept while protocol is finished.
- 16. __PoolID • PP Server uses daemons and job pooling to speed up executing jobs • Setting __PoolID sets which job pool your protocol is executed in • You CANNOT put the __PoolID parameter on the protocol itself => Admin discussion in (ATS3-PLAT11) Advanced Planning for AEP Deployments_Migrations
- 18. Using __PoolID • Built-in job pools (Some job pools are configured to run OOTB): – Warm-up pool – Keep-warm pool – Default pool • Job pools and impersonation
- 19. Server optimization See (ATS3-PLAT11) Advanced Planning for AEP Deployments_Migrations And also (ATS2-07) Solving Large Computing Challenges with Pipeline Pilot Cluster • Built into Pipeline Pilot • Leverages Existing Grid Engine • Sun GridEngine Grid • PBS Pro • LSF • Custom Scripts
- 20. Summary • Protocol Refactoring is a very critical step. • Application of basic principles can improve dramatically performances • Fine tuning needs good knowledge of the context • Use a specific job pool for your apps • Accelrys Enterprise Platform is very scalable.
- 21. The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision. For more information on the Accelrys Tech Summits and other IT & Developer information, please visit: https://community.accelrys.com/groups/it-dev