Free Servers to Build Big Data System on: Bing’s Approach
Download as PPTX, PDF•0 likes•222 views
The document outlines a professional's experience with big data and AI platforms, highlighting a large-scale offline processing platform for Bing and a history of data set curation for Office 365. It discusses the Bing Magnetar platform's utilization of underutilized capacity from a million machines for deep learning and open source pipelines, addressing challenges like data availability and task allocation. Key technologies mentioned include Hadoop, Spark, and TensorFlow, alongside solutions for optimizing resource usage.
Free Servers to Build Big Data System on: Bing’s Approach
- 2. • 10-year experience in Big Data and AI platform • PMP, MBA, MCSE (Data Mgmt and Analytics) Now: • Large-scale (100K-server) offline processing platform for Bing • OSS stack evangelization and adoption Past: • Curated Data Sets for Office 365 • Compliant DL training platform Office 365 • Data-Driven Engineering Culture Building [email protected]
- 4. What is Bing MagneTar Platform
- 5. • Imagine you have 1 million machines • Not all of them are fully utilized • I can reuse underutilized capacity… • To host DL and Open Source pipelines Utilization curve 1 million machines 100% Big Data and Deep Learning (Hadoop, Spark, Kafka TensorFlow, ONNX, etc.)
- 7. Challenges and Solutions to use Free Servers
- 8. Yet-to-Retire Machines Maintenance Buffer Machines Online Serving Machines Key Characteristics Relatively stable, but subject to return any time; Large amount, but churning quickly Running production critical services May have spare cycles time to time Key Challenges Maintain data availability at bulk machine moves Predict machine return and smart task allocation Isolate data tasks from production services
- 9. PerfISO Advanced YARN NodeLabels HDFS Block Placement Policy
- 10. Primary SecondaryIdle Primary memory usage Secondary memory usage Total memory for primary + secondary Buffer
- 12. Local/Start Rack A Remote Rack
- 13. Thank You!