HyperBatch - Snowforce 2017
1 like•998 views
HyperBatch is a library that provides a faster alternative to the traditional Apex Database.Batchable interface in Salesforce. It was tested on three batch jobs - CreateContactsBatch, UpdateContactsBatch, and DeleteContactsBatch - involving 121k accounts and 363k contacts. Using HyperBatch, all three jobs completed in just 6 minutes, compared to 88 minutes using the traditional interface, resulting in a speed increase of over 80 minutes or 93%. HyperBatch achieves this speed by leveraging asynchronous processing, parallel execution of batch jobs, and automatic retry of transactions that encounter row locks.
HyperBatch - Snowforce 2017
- 3. Why HyperBatch? Speed! Case study: Account / Contact Batches • Prerequisite: 121K Accounts already in the system • CreateContactsBatch: Creates 3 Contacts for each Account with a random “probability” field for each. 363k Contacts total. • UpdateAccountsBatch: For each Account, update the highest and lowest probability on the Account by querying the child Contacts. Get the overall highest and lowest probability across all the Accounts. • DeleteContactsBatch: Delete all of the Contacts in the system. Keep a running total of how many get deleted.
- 9. Why HyperBatch? Speed! DeleteContactsHyperBatch • Using HyperBatch: • 3 mins 15 seconds • RowLock / retry • 9500 Batch Size
- 10. Why HyperBatch? Speed! Operation Database.Batchable HyperBatch Difference Percentage CreateContacts 45 1.4 43.6 3.2% UpdateContacts 10 0.9 9.1 9.9% DeleteContacts 33 3.25 29.8 10.9% Total 88 5.55 82.5 6.7%
- 11. Why HyperBatch? Speed! Summary • Running all 3 example batch jobs takes only 6 mins instead of 88 mins. • You save 82 mins. • It only takes 6.7% of the time!
- 13. Why HyperBatch? Concurrency • Asynchronous • Apex Batch • 5 running per org • HyperBatch • Not subject to 50 job per transaction limit (each request is a new transaction) • Synchronous • Apex Batch • N/A • HyperBatch • 10 long running per org, unlimited otherwise
- 16. How it Works - Summary • HyperBatch interface that mimics the Database.Batchableinterface. • Browser orchestration for selecting jobs and running them on-demand. • Lightning Design System, Visualforce(Lightning Components would be a data bottleneck). • AJAX toolkit for PK chunking the query locator. • Wrapping requests in unique identifiers for closed loop execution – JavaScript function binding. • Each execute can return some state of type Object, it can be anything. • Async • Parallel remote actions fire the qeueables for the batch executions methods. (Not serial!) • JavaScript polls for the status of the qeueables, waiting for them to complete. • Execute state is stored in a custom object, and a list of them is returned to the finish() method, then they are deleted. • Sync • Parallel remote actions fire the synchronous transactions. (Not serial!) • Execute state is returned synchronously and stored in JavaScript in the browser until they are all complete.
- 21. synchronous vs. asynchronous governor limits Synchronous Asynchronous SOQL Queries 100 200 Heap Size 6MB 12MB CPU Time 10 seconds 60 seconds
- 22. Demo
- 23. New Features • Synchronous mode • Batch Size Control • Test Methods
- 24. Roadmap • Throttle requests (x number of open connections) • Enhance the user interface • Support custom iterators instead of just query locator • Support simple data operations like update a field or delete records without having to write Apex • Chunk in 2 dimensions: (Parent Id, then Record Id) to avoid row lock errors
- 25. Resources • Salesforce Developer Blogs • “Data Chunking Techniques for Massive Orgs“ by Daniel Peter (https://developer.salesforce.com/blogs/developer-relations/2015/11/pk-chunking- techniques-massive-orgs.html) • “French Touch Dreamin’ – The first community-led event in France” by Philippe Ozil (https://developer.salesforce.com/blogs/developer-relations/2016/11/french-touch- dreamin-first-community-led-event-france.html) • Presentation from Forcelandia 2016: “PK Chunking – Divide and conquer massive objects in Salesforce” (http://www.slideshare.net/danieljpeter/forcelandia-2016-pk-chunking) • GitHub repo: HyperBatch (https://github.com/danieljpeter/HyperBatch)
- 26. Questions?
- 27. Thank you!