Azure: Lessons From The Field
3 likes•3,866 views
The document discusses the use of Azure cloud computing at Oak Ridge National Laboratory for scientific post-processing and data distribution, highlighting lessons learned in performance and scalability. It details various Azure services and storage options, emphasizing their application in managing large datasets and scientific computations. Key insights include the importance of handling data efficiently, the performance of Azure services, and the need for careful planning in cloud-based applications.
Azure: Lessons From The Field
- 1. Lessons from the Field:Azure for ScienceRob [email protected]@argodev
- 2. AgendaIntroductionsWhy is ORNL looking at Cloud Computing
- 3. Azure in 5 minutesPost-Processing and Data Distribution in the Cloud Using Cloud Computing for Post-Processing
- 4. Data hosting/distributionLessons (being) Learned General Lessons
- 5. PerformanceOak Ridge National Laboratory is DOE’s largest science and energy labWorld’s most powerful open scientific computing facility
- 6. Nation’s largest concentrationof open source materials research
- 7. $1.6B budget
- 10. $350 million investedin modernization
- 11. Nation’s most diverse energy portfolio
- 12. Operating the world’s most intense pulsed neutron source
- 13. Managing the billion-dollar U.S. ITER projectDelivering science and technologyUltrascale computingEnergy technologiesBioenergyITERNeutron sciencesClimateMaterials at the nanoscaleNational securityNuclear energy
- 14. UltrascaleScientific ComputingLeadership Computing Facility:
- 15. World’s most powerful open scientific computing facility
- 16. Peak speed of 2.33 petaflops (> two thousand trillion calculations/sec)
- 17. 18,688 nodes, 224,526 compute cores, 299 TB RAM, 10,000 TB Disk
- 18. 4,352 ft2 floor space
- 19. Exascale system by the end of the next decade
- 20. Focus on computationally intensive projects of large scale and high scientific impact
- 21. Addressing key science and technology issues
- 22. Climate
- 23. Fusion
- 24. Materials
- 25. Bioenergy
- 26. Home of the 1st and 3rd fastest super computers in the world.The world’s most powerful system for open science
- 27. Then Why Look at Cloud Computing???Science Takes Different FormsTight Simulations
- 29. Embarrassingly ParallelDearth of Mid-Range Assets256-1,000 cores
- 30. 1 of many possible solutionsScaling IssuesPower Consumption
- 33. Next-Generation ResearchersPrivate(On-Premise)Infrastructure(as a Service)Platform(as a Service)Types of CloudsYou manageApplicationsApplicationsApplicationsYou manageRuntimesRuntimesRuntimesSecurity & IntegrationSecurity & IntegrationSecurity & IntegrationManaged by vendorDatabasesDatabasesDatabasesYou manageServersServersServersManaged by vendorVirtualizationVirtualizationVirtualizationServer HWServer HWServer HWStorageStorageStorageNetworkingNetworkingNetworking
- 34. Private(On-Premise)Types of CloudsInfrastructure(as a Service)Platform(as a Service)
- 35. Application Services“Dublin”“Velocity”Frameworks“Geneva”SecurityAccess ControlProject “Sydney”ConnectivityService BusSQL Azure Data SyncDataComputeWindows Azure PlatformTable StorageBlob StorageQueueDriveContent Delivery NetworkStorage
- 36. Windows Azure ComputeDevelopment, service hosting, & management environment.NET, Java PHP, Python, Ruby, native code (C/C++, Win32, etc.)ASP.NET providers, FastCGI, memcached, MySQL, TomcatFull-trust – supports standard languages and APIsSecure certificate storeManagement API’s, and logging and diagnostics systemsMultiple roles – Web, Worker, Virtual Machine (VHD)Multiple VM sizes1.6 GHz CPU x64, 1.75GB RAM, 100Mbps network, 250GB volatile storageSmall (1X), Medium (2X), Large (4X), X-Large (8X)In-place rolling upgrades, organized by upgrade domainsWalk each upgrade domain one at a timeCompute
- 37. Windows Azure DiagnosticsConfigurable trace, performance counter, Windows event log, IIS log & file bufferingLocal data buffering quota managementQuery & modify from the cloud and from the desktop per role instanceTransfer to storage scheduled & on-demandFilter by data type, verbosity & time rangeCompute
- 38. Windows Azure StorageRich data abstractions – tables, blobs, queues, drives, CDNCapacity (100TB), throughput (100MB/sec), transactions (1K req/sec)High accessibilitySupports geo-locationLanguage & platform agnostic REST APIsURL: http://<account>.<store>.core.windows.netClient libraries for .NET, Java, PHP, etc.High durability – data is replicated 3 times within a cluster, and (Feb 2010) across datacentersHigh scalability – data is automatically partitioned and load balanced across serversStorageStorage
- 39. Windows Azure Table StorageDesigned for structured data, not relational dataData definition is part of the applicationA Table is a set of Entities (records)An Entity is a set of Properties (fields)No fixed schemaEach property is stored as a <name, typed value> pairTwo entities within the same table can have different propertiesNo schema is enforcedTable Storage
- 40. Windows Azure Blob StorageStorage for large, named files plus their metadataBlock Blob Targeted at streaming workloadsEach blob consists of a sequence of blocksEach block is identified by a Block IDSize limit 200GB per blobPage BlobTargeted at random read/write workloadsEach blob consists of an array of pagesEach page is identified by its offset from the start of the blobSize limit 1TB per blobBlob Storage
- 41. Windows Azure QueuePerformance efficient, highly available and provide reliable message deliveryAsynchronous work dispatchInter-role communication Polling based model; best-effort FIFO data structureQueue operationsCreate QueueDelete QueueList QueuesGet/Set Queue MetadataMessage operationsAdd MessageGet Message(s)Peek Message(s)Delete MessageQueue
- 42. Windows Azure DriveProvides a durable NTFS volume for Windows Azure applications to useUse existing NTFS APIs to access a durable driveDurability and survival of data on application failover Enables migrating existing NTFS applications to the cloudDrives can be up to 1TB; a VM can dynamically mount up to 8 drivesA Windows Azure Drive is a Page BlobExample, mount Page Blob as X:\http://<account>.blob.core.windows.net/<container>/<blob>All writes to drive are made durable to the Page BlobDrive made durable through standard Page Blob replicationDrive
- 43. Windows Azure Content Delivery NetworkProvides high-bandwidth global blob content delivery18 locations globally (US, Europe, Asia, Australia and South America), and growingBlob service URL vs. CDN URLBlob URL: http://<account>.blob.core.windows.net/CDN URL: http://<guid>.vo.msecnd.net/ Support for custom domain namesAccess detailsBlobs are cached in CDN until the TTL passesUse per-blob HTTP Cache-Control policy for TTL (new)CDN provides only anonymous HTTP accessContent Delivery Network
- 44. Tenants of Internet-Scale Application ArchitectureDesignHorizontal scaling
- 47. Fault tolerant (expect failures)SecurityClaims-based authentication & access control
- 49. Data encryption & key mgmt.ManagementPolicy-driven automation
- 50. Aware of application lifecycles
- 51. Handle dynamic data schema and configuration changesData & ContentDe-normalization
- 54. Diverse data storage options (persistent & transient, relational & unstructured, text & binary, read & write, etc.)ProcessesLoosely coupled components
- 55. Parallel & distributed processing
- 58. Isolation (separation of concerns)Application GoalsSimulate Post-Processing of Scientific DataGenerate Visualizations from “raw” data
- 59. Transform data to be consumable by general processes
- 60. Exercise various storage mechanismsFocus on MechanicsThe specific science problem being solved is secondary to the approach
- 61. Goal is to refine approach such that it can fade allowing the science to regain preeminencePutting Data Into the CloudSource DataNetCDF files – subset of US contribution to CMIP3 archiveVisualization SupportFlatten Source Files to CSV
- 62. Generate base “heat map”
- 63. Combine heat map and base map
- 64. Generate Video/AnimationGeneral Consumption/PublishingExpose data as a “service” (REST/XML/JSON, etc.)
- 65. Query-able
- 66. Azure Tables (OGDI) / Azure BlobApplication PatternsGrid / Parallel Computing ApplicationUserSilverlightApplicationWeb BrowserMobileBrowserWPFApplicationASP.NET(Web Role)Web Svc(Web Role)Jobs(Worker Role)ASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)Private CloudPublic ServicesASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)ASP.NET(Web Role)Enterprise ApplicationApplicationServiceEnterprise Web SvcDataServiceTable StorageServiceBlob StorageServiceQueueServiceEnterprise DataStorageServiceIdentityServiceEnterprise IdentityService BusAccess Control ServiceWorkflowServiceUserDataApplication DataReference Data
- 67. FlattenNetCDFGenerate ImageTable LoaderApplication FlowMessage From QMessage From QMessage From QDownload Binary FileDownload CSVDownload CSVFor each Time Period…Generate ImageRead In RowsFlatten to CSV (memory)Size ImageFor each Set of 100…Upload to Blob StorageUpload to Blob StorageSubmit Batch To TableQueue Table Load JobCombine with OverlayQueue Gen Image JobUpload to Blob StoragePeriod in Lookup Table
- 68. Context35 TB of numbers – How Much Data Is That?A single lat/lon map at typical climate model resolution represents ~40 KB
- 69. If you wanted to look at all 35 TB in the form of these lat/lon plots and if…
- 70. Every 10 seconds you displayed another map
- 71. You worked 24 hours/day, 365 days/year
- 72. You could complete the task in about 200 years.Dataset Used 1 NetCDF file, approximately 92 MB, located in blob storage
- 73. 1,825 CSV files generated.
- 74. 815.84 MB total
- 75. Average file size is around 457.76 KB
- 76. Each CSV represented 12,690 data points (lat/lon/temp)
- 78. 145.03 MB total
- 79. Heat Maps avg. 31.25 KB
- 80. Combined images avg. 49 KB
- 81. 23,652,000 entities added to azure tableLessonsPerformance CountersTake advantage of the new logging infrastructure within Azure to understand how your application is behaving.
- 82. However, like food at the dinner table, only take what you can eat.Flatten Operation – Proc utilization ~16% during active work
- 83. Image Generation – Proc utilization ~95% during active work
- 84. Table Load – Proc utilization ~57% during active work
- 85. Table Load – Proc utilization ~57% during active work
- 86. LessonsPerformance CountersTake advantage of the new logging infrastructure within Azure to understand how your application is behaving.
- 87. However, like food at the dinner table, only take what you can eat.Tracing InfrastructureHuge improvements from CTP to v1
- 88. Use categories to filter / limit what you transfer out
- 89. My eyes were bigger than my stomachTable Maintenance(nodes * counters) + (nodes * trace) == lots of data
- 90. Plan early for how you are going to maintain Wad* tables.
- 91. Remember… redundancy/availability has a cost. (Perf)Flatten: CSV Upload TimeOver 40,349 attempts, 249.99 ms (79.12ms) with a rate of 15.63 mb/s (4.74). Avg File size: 457.76 KB
- 92. Flatten: CSV Upload RateOver 40,349 attempts, 249.99 ms (79.12ms) with a rate of 15.63 mb/s (4.74). Avg File size: 457.76 KB
- 93. Flatten: Queue Insert DurationOver 40,345 attempts, given a msg size of 616b, insertion time averaged 254.96 ms (68.86)
- 94. Flatten: Single Table Entity InsertOver 40,353 attempts, average insertion time of 248.63 ms (108.16)
- 95. ImageGen: CSV File Download DurationOver 40,349 attempts, 249.99 ms (79.12ms) with a rate of 15.63 mb/s (4.74). Avg File size: 457.76 KB
- 96. ImageGen: CSV File Download RateOver 40,349 attempts, 249.99 ms (79.12ms) with a rate of 15.63 mb/s (4.74). Avg File size: 457.76 KB
- 97. ImageGen: Image Generation and ResizingOver 24,687 attempts, average generation time was 3.7s (0.283s)
- 98. ImageGen: Image File Upload DurationOver 24,688 attempts, 88.14ms (44.84ms) with a rate of 3.02 mb/s (0.614). Avg File size: 32 KB
- 99. ImageGen: Image File Upload RateOver 24,688 attempts, 88.14ms (44.84ms) with a rate of 3.02 mb/s (0.614). Avg File size: 32 KB
- 100. TableLoad: Batch Insert RateOver 89,202 batches (100 records each), average duration was 1.447s (0.316s)
- 101. LessonsDataGeneric formats tend to be large (92 MB NetCDF 816 MB CSV)
- 102. Data transfer within Azure datacenter is fast (from your computer is slow)
- 103. Think about transport overhead (ATOM/JSON/CSV/etc. – 9x larger)
- 104. Use Asynccalls for data uploads/downloads (use your CPU cycles wisely – you are paying for them)Azure Tables Inserts/Deletes are slow but relatively linear
- 105. Partition keys are not queryable… store them
- 106. Not well suited for “changing” data
- 107. If you are using the client library/ADO.NET Data Services, be careful of how you handle async calls – you can lose context
- 108. Use batch updates wherever possible (1 in 0.24863s or 100 in 1.447s) (6 individual updates take longer than 100 in a single batch.LessonsGeneralTimeouts happen – Expect/Plan for them (exponential back-off & retry policies)
- 110. Watch your compilation model (x86 vs. x64)
- 111. Data transfer within Azure datacenter is fast (from your computer is slow)
- 112. Don’t re-invent the wheel – use the available tools when practical
- 113. Powershell, PowerPivot, Logparser, and the NET Charting Libraries are your friend.Thank [email protected]@argodev
- 114. The Microsoft CloudData Center Infrastructure
- 115. The Microsoft CloudData Center Infrastructure
- 116. The Microsoft Cloud~100 Globally Distributed Data CentersQuincy, WAChicago, ILSan Antonio, TXDublin, IrelandGeneration 4 DCs