Should I move my database to the cloud?

Editor's Notes

• #2: So you have been running on-prem SQL Server for a while now.  Maybe you have taken the step to move it from bare metal to a VM, and have seen some nice benefits.  Ready to see a TON more benefits?  If you said “YES!”, then this is the session for you as I will go over the many benefits gained by moving your on-prem SQL Server to an Azure VM (IaaS).  Then I will really blow your mind by showing you even more benefits by moving to Azure SQL Database (PaaS/DBaaS).  And for those of you with a large data warehouse, I also got you covered with Azure SQL Data Warehouse.  Along the way I will talk about the many hybrid approaches so you can take a gradual approve to moving to the cloud.  If you are interested in cost savings, additional features, ease of use, quick scaling, improved reliability and ending the days of upgrading hardware, this is the session for you!
• #3: Fluff, but point is I bring real work experience to the session
• #6: Four Reasons to Migrate Your SQL Server Databases to the Cloud: Security, Agility, Availability, and Reliability Reasons not to move to the cloud: Security concerns (potential for compromised information, issues of privacy when data is stored on a public facility, might be more prone to outside security threats because its high-profile, some providers might not implement the same layers of protection you can achieve in-house) Lack of operational control: Lack of access to servers (i.e. say you are hacked and want to get to security and system log files; if something goes wrong you have no way of controlling how and when a response is carried out; the provider can update software, change configuration settings, and allocate resources without your input or your blessing; you must conform to the environment and standards implemented by the provider) Lack of ownership (an outside agency can get to data easier in the cloud data center that you don’t own vs getting to data in your onsite location that you own.  Or a concern that you share a cloud data center with other companies and someone from another company can be onsite near your servers) Compliance restrictions Regulations (health, financial) Legal restrictions (i.e. data can’t leave your country) Company policies You may be sharing resources on your server, as well as competing for system and network resources Data getting stolen in-flight (i.e. from the cloud data center to the on-prem user)
• #12: As you can see, SQL Server is not just a database. We have been adding capabilities across these three phases of the data lifecycle for years. Our engineering team continually aims to build new functionalities into the platform so customers don’t have to acquire and stitch solutions together. Let’s take in-memory technology as an example: We first introduced in-memory technology back in 2008 R2 and started improving analytics by building in-memory into PowerPivot to analyze millions of rows of data in Excel. Then in SQL Server 2012, we expanded our in-memory footprint by adding in-memory to Analysis Services so IT could build data models much faster, and introduced an in-memory column store that could improve query speeds. With SQL Server 2014, we introduced an in-memory OLTP solution to significantly speed transactional performance. If you’re running SQL Server 2005, you should start planning your upgrade before end of support hits next April. If you have an EA with Software Assurance, licensing costs for upgrade are included. After support ends: You will no longer receive security updates Maintenance costs will increase You may encounter compliance concerns
• #13: http://www.jamesserra.com/archive/2016/02/hp-superdome-x-for-high-end-oltpdw/ As you can see, SQL Server is not just a database. We have been adding capabilities across these three phases of the data lifecycle for years. Our engineering team continually aims to build new functionalities into the platform so customers don’t have to acquire and stitch solutions together. Let’s take in-memory technology as an example: We first introduced in-memory technology back in 2008 R2 and started improving analytics by building in-memory into PowerPivot to analyze millions of rows of data in Excel. Then in SQL Server 2012, we expanded our in-memory footprint by adding in-memory to Analysis Services so IT could build data models much faster, and introduced an in-memory column store that could improve query speeds. With SQL Server 2014, we introduced an in-memory OLTP solution to significantly speed transactional performance. If you’re running SQL Server 2005, you should start planning your upgrade before end of support hits next April. If you have an EA with Software Assurance, licensing costs for upgrade are included. After support ends: You will no longer receive security updates Maintenance costs will increase You may encounter compliance concerns
• #14: RAs Some assembly required OR No assembly required (through Partners) FT Training Understand File System layouts How you physically implement DW has to be like the guidelines Appliance MPP Training Queries are different Modeling is different Data structures are different Partitioning is different Key decision points: Data Volumes: If you get above 95 Terabytes Procurement and Operation Logistics: What is your HW SKU process? Can you put a brand new HW system onto the data center? Workload Characteristics: Highly concurrent data? DW Organizational Maturity: Do you already know MPP?
• #16: SMP is one server where each CPU in the server shares the same memory, disk, and network controllers (scale-up). MPP means data is distributed among many independent servers running in parallel and is a shared-nothing architecture, where each server operates self-sufficiently and controls its own memory and disk (scale-out).
• #23: In this session we will take a closer look at the third pillar or key investment area for SQL Server 2014, which is build a data platform for hybrid cloud. One of the key design points we have taken to approaching cloud computing is to drive towards a consistent data platform with common tools from on-premises to cloud. When it comes to Microsoft’s cloud, we have two offerings that can be used to run relational databases in the cloud depending on your application needs. Lets take a closer look at both of these Microsoft Azure options.
• #26: https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-sizes Note the IOPS for standard storage is the maximum rather than an expected number. For premium, IOPS are not just maximum but expected levels of performance. Standard Disk Storage (HDD) 1GB-1023GB, 500 IOPs, 60 MB/s throughput, $2/month per 100GB (locally redundant), $5/month per 100GB (geo-redundant), $6/month per 100GB (read-access geo-redundant)   Premium Disk Storage (SSD) P10, 128GB, 500 IOPs, 100 MB/s throughput, $20/month P20, 512GB, 2300 IOPs, 150 MB/s throughput, $73/month P30, 1024GB, 5000 IOPs, 200 MB/s throughput, $135/month
• #27: Azure web portal -> New -> Marketplace (see all)
• #28: https://azure.microsoft.com/en-us/documentation/templates/
• #29: https://blogs.msdn.microsoft.com/mast/2013/12/06/understanding-the-temporary-drive-on-windows-azure-virtual-machines/ SSD/HDD storage included in A-series, D-series, and Dv2-series VMs is local temporary storage. DS-series, G-series, GS-series SSD's have less local temporary storage due to storage used for caching purposes to ensure predictable levels of performance associated with premium storage. DS-series and GS-series support premium storage disks, which means you can attach SSD's to the VM (the other series support only standard storage disks). The pricing and billing meters for the DS sizes are the same as D-series and the GS sizes are the same as G-series. When you create an Azure virtual machine, it has a disk for the operating system mapped to drive C (size is 127GB) that is on Blob storage and a local temporary disk mapped to drive D. You can choose standard disk type or premium (if DS-series or GS-series) for your local temporary disk - the size of which is based on the series you choose (i.e. A0 is 20GB). You can also attach new disks - specify standard or premium, for standard: specify size (1GB-1023GB), for premium: specify P10, P20, or P30. the disks are .vhd files that reside in an Azure storage account
• #31: http://www.jamesserra.com/archive/2015/11/redundancy-options-in-azure-blob-storage/ Geo-replication in Azure disks does not support the data file and log file of the same database to be stored on separate disks. GRS replicates changes on each disk independently and asynchronously. This mechanism guarantees the write order within a single disk on the geo-replicated copy, but not across geo-replicated copies of multiple disks. If you configure a database to store its data file and its log file on separate disks, the recovered disks after a disaster may contain a more up-to-date copy of the data file than the log file, which breaks the write-ahead log in SQL Server and the ACID properties of transactions. If you do not have the option to disable geo-replication on the storage account, you should keep all data and log files for a given database on the same disk. If you must use more than one disk due to the size of the database, you need to deploy one of the disaster recovery solutions listed above to ensure data redundancy. https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sql-high-availability-dr/
• #32: In the past, after provisioning a SQL Server VM, you had to manually attach and configure the right number of data disks to provide the desired number of IOPs or throughput (MB/s). Then you need to stripe your SQL files across the disks or create a Storage Pool to divide the IOPs or throughput across them. Finally, you’d have to configure SQL Server according to the performance best practices for Azure VM. We’ve now made this part of the provisioning experience. You can easily configure the desired IOPs, throughput, and storage size within the limits of the selected VM size, as well as the target workload to optimize for (online transaction processing or data warehousing). As you change the IOPs, throughput, and storage size, we’ll automatically select the right number of disks to attach to the VM. During the VM provisioning, if more than one disk is required for your specified settings, we’ll automatically create one Windows storage space (virtual drive) across all disks.
• #33: https://azure.microsoft.com/en-us/regions/
• #34: https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-migrate-onpremises-database/ http://itproguru.com/expert/2015/03/how-to-move-or-migrate-sql-server-workload-to-azure-sql-database-cloud-services-or-azure-vm-all-version-of-sql-server-step-by-step/
• #35: http://www.sqlservercentral.com/blogs/sqlsailorcom/2015/09/24/azure-virtual-machine-blog-series-changing-the-size-of-a-vm/
• #36: https://azure.microsoft.com/en-us/blog/resize-virtual-machines https://buildwindows.wordpress.com/2015/10/11/azure-virtual-machine-resizing-consideration/ When a VM is running it is deployed to a physical server. The physical servers in Azure regions are grouped together in clusters of common physical hardware. A running VM can easily be resized to any VM size supported by the current cluster of hardware supporting the VM.
• #37: https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-classic-sql-dr/ It is up to you to ensure that your database system possesses the HADR capabilities that the service-level agreement (SLA) requires. The fact that Azure provides high availability mechanisms, such as service healing for cloud services and failure recovery detection for the Virtual Machines (https://azure.microsoft.com/en-us/blog/service-healing-auto-recovery-of-virtual-machines), does not itself guarantee you can meet the desired SLA. These mechanisms protect the high availability of the VMs but not the high availability of SQL Server running inside the VMs. It is possible for the SQL Server instance to fail while the VM is online and healthy. Moreover, even the high availability mechanisms provided by Azure allow for downtime of the VMs due to events such as recovery from software or hardware failures and operating system upgrades.
• #44: https://docs.microsoft.com/en-us/azure/sql-database/sql-database-service-tiers
• #45: Changing the service tier and/or performance level of a database creates a replica of the original database at the new performance level, and then switches connections over to the replica. No data is lost during this process but during the brief moment when we switch over to the replica, connections to the database are disabled, so some transactions in flight may be rolled back. This window varies, but is on average under 4 seconds, and in more than 99% of cases is less than 30 seconds. If there are large numbers of transactions in flight at the moment connections are disabled, this window may be longer. The duration of the entire scale-up process depends on both the size and service tier of the database before and after the change. For example, a 250 GB database that is changing to, from, or within a Standard service tier, should complete within 6 hours. For a database of the same size that is changing performance levels within the Premium service tier, it should complete within 3 hours.
• #47: By storing your data in Azure SQL Database, you take advantage of many fault tolerance and secure infrastructure capabilities that you would otherwise have to design, acquire, implement, and manage. Azure SQL Database has a built-in high availability subsystem that protects your database from failures of individual servers and devices in a datacenter. Azure SQL Database maintains multiple copies of all data in different physical nodes located across fully independent physical sub-systems to mitigate outages due to failures of individual server components, such as hard drives, network interface adapters, or even entire servers. At any one time, three database replicas are running—one primary and two or more replicas. Data is written to the primary and one secondary replica using a quorum based commit scheme before the transaction is considered committed. If the hardware fails on the primary replica, Azure SQL Database detects the failure and fails over to the secondary replica. In case of a physical loss of a replica, a new replica is automatically created. So there are always at minimum two physical, transactionally consistent copies of your data in the datacenter.
• #48: A self-service feature available for Basic, Standard, and Premium databases. Supports business continuity by recovering a database from a recent backup after accidental data corruption or deletion. Automatic backups Full backups weekly Differential backup daily Log backups every 5 minutes Daily and weekly backups automatically uploaded to geo-redundant Azure Storage Self-service restore Point-in-time up to a second granularity REST API, Windows PowerShell, or Portal Creates a new database in the same logical server Tiered retention policy Basic - 7 days Standard - 14 days Premium - 35 days No additional cost to retain backups
• #49: With all tiers of SQL Database now supporting active geo-replication, do you still talk to customers about geo-restore?  Or do you tell them to just use active geo-replication? Bill Gibson: Both are valid.  Geo replication still doubles the cost and for many customers is more than they want. A little more on this.  MYOB, our largest external user, relies on geo-restore exclusively at this point, as an example.   Regarding doubling the cost, we still have a now-deprecated option, Standard Geo-Replication, intended for DR only.  This option produces a single non-accessible secondary, which is only accessible after failover.  A Standard  geo-replication secondary is charged at 75% of the full database price.  This option is being discontinued and being replaced by active geo-replication, which as you observe is now available on all editions, which, because it results in a readable secondary, is charged at 100%.  
• #50: Enjoy business continuity and focus on building apps instead of keeping things running. Active geo-replication—available for Premium databases today—gives you the richest business continuity solution with the least risk of data loss and the most rapid recovery time.  Extends standard geo-replication with up to four geo-replicated secondaries online and readable at all times, and that can also be used for load balancing or to provide low-latency access to replicated data anywhere in the world.  Coming in 2016 for Basic and Standard databases. Coming in 2016 manual and automatic failover options
• #51: Based on customer feedback, Azure SQL Database is introducing new service tiers to help customers more easily innovate with cloud-designed database workloads. At the heart of this change, the new tiers deliver predictable performance across a spectrum of six performance levels for light- to heavy-weight transactional application demands. Additionally, the new tiers offer a spectrum of business-continuity features, a stronger uptime SLA, larger database sizes for less money, and an improved billing experience.
• #52: Migration tools Tools used include SQL Server Management Studio (SSMS), the SQL Server tooling in Visual Studio, and the SQL Azure Migration Wizard (SAMW), as well the preview of the new Azure management portal. Be sure to install the latest versions of the client tools as earlier versions are not compatible with the preview of the latest SQL Database Update. The role of each tool is summarized below together with a link for installing/accessing the latest version. SQL Server Management Studio (SSMS) SSMS can be used to deploy a compatible database directly to Azure SQL Database or to export a logical backup of the database as a BACPAC, which can then be imported, still using SSMS, to create a new Azure SQL Database. You cannot use the preview portal to import a BACPAC yet. You must use the latest version of SSMS to work with the preview of Azure SQL Database Latest Update which is available in CU5 of SQL Server 2014 or by downloading the latest version of the tools from http://msdn.microsoft.com/en-us/evalcenter/dn434042.aspx. SQL Azure Migration Wizard (SAMW) SAMW can be used to analyze the schema of an existing database for compatibility with Azure SQL Database, and in many cases can be used to generate and then deploy a T-SQL script containing schema and data. The wizard will report errors during the transformation if it encounters schema content that it cannot transform. If this occurs, the generated script will require further editing before it can be deployed successfully. SAMW will process the body of functions or stored procedures which is normally excluded from validation performed by the SQL Server tooling in Visual Studio (see below) so may find issues that might not otherwise be reported by validation in Visual Studio alone. Combining use of SAMW with the SQL Server tooling in Visual Studio can substantially reduce the amount of work required to migrate a complex schema. Be sure to use the latest version of the SQL Azure Migration Wizard from CodePlex at http://sqlazuremw.codeplex.com/. SQL Server tooling in Visual Studio The SQL Server tooling in Visual Studio can be used to create and manage a database project comprising a set of T-SQL files for each object in the schema. The project can be imported from a database or from a script file. Once created, the project can be targeted at the latest preview of Azure SQL Database; building the project then validates schema compatibility. Clicking on an error opens the corresponding T-SQL file allowing it to be edited and the error corrected. Once all the errors are fixed the project can be published, either directly to SQL Database to create an empty database or back to (a copy of) the original SQL Server database to update its schema, which allows the database to be deployed with its data using SSMS as above. You must install and use the preview of the SQL Server database tooling for Visual Studio with support for the preview of Azure SQL Database Latest Update V12. Make sure you have Visual Studio 2013 with Update 4 installed first. See this blog post for more details of this preview release and how to install it: http://blogs.msdn.com/b/ssdt/archive/2014/12/18/sql-server-database-tooling-preview-release-for-the-latest-azure-sql-database-update-v12-preview.aspx. You can keep track of updates to this software on the team blog at http://blogs.msdn.com/b/ssdt/.
• #59: 58
• #61: Enables query capabilities across common Hadoop distributions (HDP & Cloudera) and Hadoop file formats in Azure storage. Polybase for querying & managing non-relational Hadoop and relational data Allows leveraging existing SQL skills and BI tools Supports multiple non-relational file formats Improved time-to-insights & simplified ETL
• #62: When it comes to key BI investments we are making it much easier to manage relational and non-relational data with Polybase technology that allows you to query Hadoop data and SQL Server relational data through single T-SQL query. One of the challenges we see with Hadoop is there are not enough people out there with Hadoop and Map Reduce skillset and this technology simplifies the skillset needed to manage Hadoop data. This can also work across your on-premises environment or SQL Server running in Azure.
• #63: 62
• #64: Auto backups, every 4 hours, in Azure Storage and geo-replicated On-demand backups in Azure Storage, user can enable geo-replication REST API, PowerShell or Azure Portal Scheduled exports for long-term retention Near-Online backup/restore based on storage snapshots Backups retention policy: Auto backups, up to 35 days On-demand backups retained indefinitely Does SQL DW support data redundancy?  Yes.  As SQL Data Warehouse separates compute and storage, all your data is directly written to geo-redundant Azure Storage (RA-GRS).  Geo-redundant storage replicates your data to a secondary region that is hundreds of miles away from the primary region.  In both primary and secondary regions, your data is replicated three times each, across separate fault domains and upgrade domains.  This ensures that your data is durable even in the case of a complete regional outage or disaster that renders one of the regions unavailable.  To learn more about Read-Access Geo-Redundant Storage, read Azure Storage Redundancy Options. Is geo-restore used for disaster recovery? Geo-Restore is designed to recover your database in case it becomes unavailable due to a disruptive event.  You can restore a database from a geo-redundant backup to create a new database in any Azure region.  Because the backup is geo-redundant it can be used to recover a database even if the database is inaccessible due to an outage.  Geo-Restore feature comes with no additional charges.  To learn more about Geo-Restore, refer to Recover an Azure SQL Database from an outage. Does SQL DW support a point-in-time restore?  Are the databases automatically backed up? Yes to both.  A point-in-Time restore is designed to restore your database to an earlier point in time.  Azure SQL Data Warehouse service protects all databases with automatic backups every 4 hours and retains them for 35 days to provide you with a discrete set of restore points.  These backups are stored on RA-GRS Azure Storage and are therefore geo-redundant by default.  The automatic backup and point-in-time restore features come with no additional charges and provide a zero-cost and zero-admin way to protect databases from accidental corruption or deletion.  To learn more about Point In Time Restore, refer to Azure SQL Database Point in Time Restore. [‎4/‎15/‎2016 5:38 PM] Hey Matt, since SQL DW data is on RA-GRS, can I fire up another SQL DW to use that read-only copy? [‎4/‎15/‎2016 5:38 PM] Matt Usher: No [‎4/‎15/‎2016 5:39 PM] k...if the primary region goes out, how do I use that read-only copy? [‎4/‎15/‎2016 5:39 PM] Matt Usher: The data is for redundancy that can be recovered by engineering for brand protection. We snapshot every 4/8 hours and you can create a geo-restore from that to any other DC.  
• #65: 64
• #68: One of the key differentiators of Azure is its breadth of managed data services that you have at your disposal. You can run anything you want in a VM, but you’re responsible for a lot of the maintenance and management. You do have ultimate control over everything, but do you really need to control everything or are you ok giving up some of this sense of control for reduced cost, improved functionality, easier scaling, and reduced human overheard needs?
• #70: On-prem heavy
• #71: 1) Copy source data into the Azure Data Lake Store (twitter data example)2) Massage/filter the data using Hadoop (or skip using Hadoop and use stored procedures in SQL DW/DB to massage data after step #5)3) Pass data into Azure ML to build models using Hive query (or pass in directly from Azure Data Lake Store)4) Azure ML feeds prediction results into the data warehouse5) Non-relational data in Azure Data Lake Store copied to data warehouse in relational format (optionally use PolyBase with external tables to avoid copying data)6) Power BI pulls data from data warehouse to build dashboards and reports7) Azure Data Catalog captures metadata from Azure Data Lake Store and SQL DW/DB8) Power BI and Excel can pull data from the Azure Data Lake Store via HDInsight9) To support high concurrency if using SQL DW, or for easier end-user data layer, create an SSAS cube
• #72: Offer structures: A la carte, Data Intensive, Analytics Intensive, Stream Intensive, All-inclusive