SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimization through customer engagements
1 like•284 views
The document discusses the spectrum of black-box optimization through customer engagements, highlighting challenges faced by material science researchers using Bayesian optimization. Key issues include budget constraints, exploration fears, and the necessity for tailored solutions to meet customer success criteria. It also mentions opportunities for professional services to improve black-box methods and customer satisfaction while introducing new features for neural network developers.
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimization through customer engagements
- 1. SigOpt. Confidential. Exploring the spectrum of black-box optimization through customer engagements Harvey Cheng 05/03
- 4. SigOpt. Confidential. Black-box optimization In the research literature
- 5. SigOpt. Confidential. Black-box optimization A spectrum from customer engagement Can you solve this problem for me? Glass box Set and forget; leave us alone. Black box
- 6. SigOpt. Confidential. Black-box optimization A spectrum from customer engagement Glass box Black box Academic collaboration
- 7. SigOpt. Confidential. Material science research Our Pitt collaborators wanted to use BO to accelerate their research. • Started using SigOpt as is and got some OK results. • Complicated experimental set-up and SigOpt was not a 1-to-1 perfect fit. • Were uninspired and had a lot of questions.
- 8. SigOpt. Confidential. Find the fabrication process for an optical device (e.g., glass) that exhibits desirable physical and optical properties. • High transmission • Low haze • High liquid contact angle Material science research
- 9. SigOpt. Confidential. Find the fabrication process for an optical device (e.g., glass) that exhibits desirable physical and optical properties. • High transmission • Low haze • High liquid contact angle Material science research 500 nm 100 nm 100 nm
- 10. SigOpt. Confidential. Bayesian optimization for material science What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears?
- 11. SigOpt. Confidential. Bayesian optimization for material science What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears? Extremely budget conscious.
- 12. SigOpt. Confidential. Bayesian optimization for material science What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears? Fear of exploration.
- 13. SigOpt. Confidential. Bayesian optimization for material science What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears? Expertise is a double-edged sword.
- 14. SigOpt. Confidential. Efficient search for the desirable material Modification and Adaptation of BO The researchers want: • To leverage their expertise. How we can help: • Careful consideration of the input parameter space.
- 15. SigOpt. Confidential. Efficient search for the desirable material Modification and Adaptation of BO The researchers want: • Lab equipment has limited precision. How we can help: • Understand how equipment precision demands a discrete domain.
- 16. SigOpt. Confidential. Efficient search for the desirable material Modification and Adaptation of BO The researchers want: • Multiobjective optimization, on a budget. How we can help: • Pose the problem as a constrained optimization problem to identify key points on the Pareto frontier.
- 17. SigOpt. Confidential. Efficient search for the desirable material
- 18. SigOpt. Confidential. Optimization platform Some of our customers cannot reveal anything about their problems. • Masking the input parameter names
- 19. SigOpt. Confidential. Optimization platform Some of our customers cannot reveal anything about their problems. • Masking the input parameter names • Misusage sometimes
- 20. SigOpt. Confidential. Optimization platform Some of our customers cannot reveal anything about their problems. • Masking the input parameter names • Misusage sometimes • Vastly different access patterns
- 21. SigOpt. Confidential. Black-box optimization A spectrum from customer engagement Academic collaboration Glass box Black box Optimization platform
- 22. SigOpt. Confidential. Optimization platform True black-box optimization What can we do in the absence of customer interaction? • Nonstandard benchmarking of our optimizer • Flexible design of the API • Scalable computation workflow
- 23. SigOpt. Confidential. Scalable computation workflow Online/Offline computation
- 24. SigOpt. Confidential. Scalable computation workflow Online/Offline computation
- 25. SigOpt. Confidential. Black-box optimization A spectrum from customer engagement Academic collaboration Glass box Black box Professional services Optimization platform
- 26. SigOpt. Confidential. Customer engagement at scale Lessons we learned from working with our friends in Academia. 1. Customers’ problems may not be addressed immediately by the existing service. 2. Customers have inherent preferences to what is considered as success and what is considered as failure. Can we apply the lessons we have learned and serve a broad array of customers?
- 27. SigOpt. Confidential. Professional services When a customer’s problem cannot be immediately solved by plugging in the SigOpt API, the PS team can help them by • Understanding the customer’s success criteria. • Building one-off projects to better interface SigOpt API with the customers.
- 28. SigOpt. Confidential. Adjusting to customer expectations Customers may judge their experience very differently from how we perceive it.
- 29. SigOpt. Confidential. Adjusting to customer expectations Customers may judge their experience very differently from how we perceive it. • “Want more exploiting, less exploring because there is one region where there is an optimal value. Not interested in exploring poor areas of performance.” - customer A • “Want to make sure the optimizer is effectively exploring the parameter space. Don’t mind if there’s a bit of extra work being done so long as it is sufficiently explored.” - customer B
- 30. SigOpt. Confidential. Production throttles Adjusting to customer biases Production throttles are hooks that we can build into the system to empower PS. • Adjust the optimization behavior to meet the customer’s demand. • No redeployment of code.
- 31. SigOpt. Confidential. Our spectrum thus far Collaboration - Services - Black box We have some customers with whom we have a collaborative relationship. For most of customers, we treat their problems as completely black box. We have identified opportunities to allow professional services to change the black box behavior without changing the product structure. Is this all?
- 32. SigOpt. Confidential. Beyond black box 2016 BayesOpt workshop: We should “Open the black-box” and go “Beyond” black-box optimization. What are some options? • Address problems with a different goal (e.g., balancing competing objectives). • Provide more information (level of noise for observations). • Operate in a different workflow. To do this, we must understand customer’s success criteria and failure modes.
- 33. SigOpt. Confidential. Special feature for neural networks Beyond black box We wanted to build a feature to effectively address neural network developers. We needed to identify the appropriate part of the black box spectrum on which this feature should lie. Academic collaboration Glass box Black box Professional services Optimization platform
- 34. SigOpt. Confidential. Special feature for neural networks Beyond black box Option 1: Hyperband • Some customers had mentioned Hyperband, which could be implemented using a Bayesian optimization strategy in the background.
- 35. SigOpt. Confidential. Special feature for neural networks Beyond black box Option 1: Hyperband • Some customers had mentioned Hyperband, which could be implemented using a Bayesian optimization strategy in the background. Complication: • When confronted with the required change in workflow (storing weights, work split across generations, idle machines), and the prospect of stopping SGD before convergence, customers balked. Conclusion: • Workflow change was too much towards glass box.
- 36. SigOpt. Confidential. Special feature for neural networks Beyond black box Option 2: Multi-task BO • We have a multi-task BO feature already in place; maybe neural network customers could be convinced to use that.
- 37. SigOpt. Confidential. Special feature for neural networks Beyond black box Option 2: Multi-task BO • We have a multi-task BO feature already in place; maybe neural network customers could be convinced to use that. Complication: • During customer interviews, we found customers confused by the definition of tasks and how to define them effectively. They also disliked stopping SGD runs before convergence. Conclusion: • Existing multi-task feature was too black box.
- 38. SigOpt. Confidential. Special feature for neural networks Beyond black box Resolution: Training Monitor • Respect the customer’s workflow (no change required). • Customers report progress during training. • Allow customers to monitor training and provide status updates regarding convergence. • Better internal models are built from all the progress information. Conclusion: • Does this fall in the best location of the spectrum?
- 39. SigOpt. Confidential. Black-box optimization A spectrum from customer engagement Academic collaboration Glass box Black box Professional services Optimization platform HPO for neural networks
- 40. SigOpt. Confidential. Thank You Paul Leu Sajad Haghanifar @University of Pittsburgh The entire SigOpt team. Special thanks to our gracious hosts, Jake, Matthias, and Uber. Hope to see you next month at ICML & CVPR! SigOpt events are being planned ...