Building Big data solutions in Azure
Download as PPTX, PDF•2 likes•936 views
The document outlines the capabilities and features of HDInsight, a Microsoft Azure implementation of the Apache Hadoop ecosystem, highlighting cluster types, custom configurations, and integrations with various business intelligence tools. It covers key components such as HBase, Hive, and Apache Storm, detailing their functionalities, data processing options, and usage scenarios within big data analytics. Additionally, it emphasizes the advantages of using HDInsight for maintaining and deploying Hadoop clusters without the need for hardware management.
Building Big data solutions in Azure
- 2. Session Objectives And Takeaways Understanding HDInsight cluster types & tiers in Azure HBase as a Hadoop NoSQL database Hive is a data warehouse software to manage large datasets using SQL Understanding data processing options in Hadoop ecosystem using Storm and Spark
- 3. • HDInsight is a cloud implementation on Microsoft Azure of the rapidly expanding Apache Hadoop technology stack that is the go-to solution for big data analysis. • It includes implementations of Apache Spark, HBase, Storm, Pig, Hive, Sqoop, Oozie, Ambari, and so on. • HDInsight also integrates with business intelligence (BI) tools such as Power BI, Excel, SQL Server Analysis Services, and SQL Server Reporting Services. • HDInsight is available on Windows and Linux • HDInsight on Linux: A Hadoop cluster on Ubuntu • HDInsight on Windows: A Hadoop cluster on Win Server 2012 R2 What is HDInsight
- 4. • HDInsight provides cluster Types & custom configurations for: • Hadoop (HDFS) • HBase • Storm • Spark • R Server (Preview) • Skip maintaining and purchasing hardware • HDInsight has powerful programming extensions for languages including C#, Java, and .NET. Use your programming language of choice on Hadoop to create, configure, submit, and monitor Hadoop jobs. HDInsight clusters on Azure
- 5. HDInsight clusters on Azure
- 6. • Apache HBase is an open-source, NoSQL database that is built on Hadoop and modeled after Google BigTable. • HBase provides random access and strong consistency for large amounts of unstructured and semistructured data in a schemaless database organized by column families • Data is stored in the rows of a table, and data within a row is grouped by column family. • The open-source code scales linearly to handle petabytes of data on thousands of nodes. It can rely on data redundancy, batch processing, and other features that are provided by distributed applications in the Hadoop ecosystem. What is HBase
- 7. Order No Customer Name Customer Phone Company Name Company Address 12012015 Mostafa 101-232-2345 Microsoft Redmond, WA Customer Company Order No Customer Name Customer Phone Company Name Company Address 12012015 Mostafa 101-232-2345 Microsoft Redmond, WA
- 8. • HBase Commands: • create Equivalent to Create table in T-SQL • get Equivalent to Select statements in T-SQL • put Equivalent to Update, Insert statement in T-SQL • scan Equivalent to Select (no where condition) in T-SQL • HBase shell is your query tool to execute in CRUD commands to a HBase cluster. • Data can also be managed using the HBase C# API, which provides a client library on top of the HBase REST API. • An HBase database can also be queried by using Hive using SQLHive. What is HBase
- 9. • Apache Hive is a data warehouse system for Hadoop, which enables data summarization, querying, and analysis of data by using HiveQL (a query language similar to SQL). • Hive understands how to work with structured and semi-structured data, such as text files where the fields are delimited by specific characters. • Hive also supports custom serializer/deserializers (SerDe) for complex or irregularly structured data. • Hive can also be extended through user-defined functions (UDF). • A UDF allows you to implement functionality or logic that isn't easily modeled in HiveQL. What is Hive
- 11. • Apache Storm is a distributed, fault-tolerant, open-source computation system that allows you to process data in real-time with Hadoop. • Apache Storm on HDInsight allows you to create distributed, real-time analytics solutions in the Azure environment by using Apache Hadoop. • Storm solutions can also provide guaranteed processing of data, with the ability to replay data that was not successfully processed the first time. • Ability to write Storm components in C#, JAVA and Python. • Azure Scale up or Scale down without an impact for running Storm topologies. • Ease of provision and use in Azure portal. • Visual Studio project templates for Storm apps What is Apache Storm
- 12. • Apache Storm apps are submitted as Topologies. • A topology is a graph of computation that processes streams • Stream: An unbound collection of tuples. Streams are produced by spouts and bolts, and they are consumed by bolts. • Tuple: A named list of dynamically typed values. • Spout: Consumes data from a data source and emits one or more streams. • Bolt: Consumes streams, performs processing on tuples, and may emit streams. Bolts are also responsible for writing data to external storage, such as a queue, HDInsight, HBase, a blob, or other data store. • Nimbus: JobTracker in Hadoop that distribute jobs, monitoring failures. Apache Storm Components
- 14. • Apache Spark™ is a fast and general engine for large-scale data processing. • Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. • Write applications quickly in Java, Scala, Python, R. • Combine SQL, streaming, and complex analytics. • Spark's in-memory computation capabilities make it a good choice for iterative algorithms in ML and graph computations. • Spark is also compatible with Azure Blob storage (WASB) so your existing data stored in Azure can easily be processed via Spark. • Support for R Server & Azure Data Lake. What is Apache Spark
- 16. Session Objectives And Takeaways Understanding HDInsight cluster types & tiers in Azure HBase as a Hadoop NoSQL database Hive is a data warehouse software to manage large datasets using SQL Understanding data processing options in Hadoop ecosystem using Storm and Spark
Editor's Notes
- #2: The session covers how to get started to build big data solutions in Azure. Azure provides different Hadoop clusters for Hadoop ecosystem. The session covers the basic understanding of HDInsight clusters including: Apache Hadoop, HBase, Storm and Spark. The session covers how to integrate with HDInsight in .NET using different Hadoop integration frameworks and libraries. The session is a jump start for engineers and DBAs with RDBMS experience who are looking for a jump start working and developing Hadoop solutions. The session is a demo driven and will cover the basics of Hadoop open source products.
- #3: The session covers how to get started to build big data solutions in Azure. Azure provides different Hadoop clusters for Hadoop ecosystem. The session covers the basic understanding of HDInsight clusters including: Apache Hadoop, HBase, Storm and Spark. The session covers how to integrate with HDInsight in .NET using different Hadoop integration frameworks and libraries. The session is a jump start for engineers and DBAs with RDBMS experience who are looking for a jump start working and developing Hadoop solutions. The session is a demo driven and will cover the basics of Hadoop open source products.
- #4: Ref: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-introduction/
- #9: Ref: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-overview/
- #10: Ref: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-overview/
- #11: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-tutorial-get-started/ A) Working with hbase shell: Create a table. Insert a record. Update a record. Delete a record. Create a hive table that maps to hbase table we just created. B) Working with Hive: use the dashboard to create database and tables.
- #12: Apache Storm in HDInsight https://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-overview/
- #13: Apache Storm in HDInsight https://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-overview/ Tips: The Nimbus node provides similar functionality to the Hadoop JobTracker, and it assigns tasks to other nodes in the cluster through Zookeeper.
- #14: Demo: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-develop-csharp-visual-studio-topology/ Overview in HDInsight project templates in Visual Studio 2015: Create storm application Create Hive Application
- #15: Ref: http://spark.apache.org/ https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-overview/
- #16: Demo: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-ipython-notebook-machine-learning/ Apache Spark notepads https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-jupyter-spark-sql/
- #18: HD Insight main documentation: https://azure.microsoft.com/en-us/documentation/services/hdinsight/