How to Operationalise Real-Time Hadoop in the Cloud
Download as PPTX, PDF•0 likes•755 views
The webinar discusses operationalizing real-time Hadoop in the cloud, featuring speakers from Attunity, Cloudera, and Microsoft. It emphasizes the importance of data ingestion, scalability, and compliance within cloud environments, while showcasing real-life use cases demonstrating Hadoop's impact on various industries. Attendees are guided on leveraging technologies such as Attunity Replicate and Azure for seamless data management and analytics.
How to Operationalise Real-Time Hadoop in the Cloud
- 1. June 15, 2016 ) How to Operationalise Real-Time Hadoop in the Cloud WEBINAR:
- 2. Welcome! Meet Today’s Speakers • Ted Orme • VP Technology EMEA - Attunity • Ian Archibald • Pre Sales Director EMEA - Attunity Reminder • Attendees are on mute • Q & A at the end via chat • Webinar is recorded for on-demand review • David Tishgart • Product Marketing Director, Cloud Solutions - Cloudera • Hans Wieser • Advanced Analytics Partner Lead - Microsoft
- 3. • Introductions • Attunity - How to ingest the most valuable enterprise data into Hadoop • Cloudera - Real-life use cases • Microsoft - The scalable flexibility of Azure • Attunity Replicate Demo • Live Q&A session Agenda
- 4. • Accelerate data delivery across enterprise and cloud • Empower rapid utilisation of data by the business • Continually optimise with intelligent insight Attunity Corporate Overview Over 2000 Customers in 65 Countries Financial Services Manufacturing / Industrials GovernmentHealth Care Technology / Telecommunications Other Industries Enterprise Data Management On Premise | Cloud | Across Platforms Global Organisation USA EMEA APAC
- 5. Attunity Platform for Enterprise Data Management • Accelerate data delivery • Empower rapid utilisation of data • Continuously improve the management of data Attunity Replicate Attunity Compose Attunity Visibility Universal Data Availability Data Warehouse Automation Metrics Driven Data Management Integrate new platforms Automate ETL/EDW Optimise performance and cost On Premises / Cloud Hadoop FilesRDBMS EDW SAP Mainframe
- 6. Attunity Replicate Move the data that moves your business
- 7. Demand • Hadoop has moved out of test • Enterprise Use Case • Closer to Production • Business impact Enterprise + Real-time VS Sqoop + Batch Attunity - Replicate • High performance connectivity to Hadoop though native APIs for data ingest and publication • Automated schema generation in Hcatalog • Drag & drop configuration with Click-2-Replicate design • High-speed data load options: • Full reload with overwrite • Insert only appends • Change Data Capture(CDC) • In-memory data filtering and transformation • Monitoring dashboard with web-based metrics, alerts and logfile management Operationalising Hadoop Hadoop
- 8. Demand • Easy Ingest + CDC • Real-time processing • Real-time monitoring • Real-time Hadoop • Scalable to 1000’s Applications • One Publisher – Multiple Consumers Attunity - Replicate • Direct integration using Kafka APIs • In-memory optimised data streaming • Support for multi-topic and multi-partitioned data publication • Full Load and CDC • Integrated management and monitoring via GUI Kafka and Real-time Streaming
- 9. T1/P0 T2/P1 T3/P0 Broker 1 Attunity Replicate for Kafka - Architecture M0 M1 M2 M3 M4 M5 M6 M7 M8 M0 M1 M2 M3 M4 M5 M0 M1 M2 M3 M4 M5 M6 M7 T1/P1 T2/P0 Broker 2 M0 M1 M2 M3 M4 M0 M1 M2 M3 M4 M5 M6
- 10. Facilitate data availability in the Cloud for BI applications BI & Analytics in Cloud Database, Data Warehouse Ingest data for Big Data Analytics using Hadoop in the CloudHadoop/Big Data Hybrid, All in Cloud Simplify and accelerate database & application migration (“Lift & Shift”) Cloud Data Migration Prioritise EDW archiving, off-load cold/hot data to the CloudEDW Optimisation and Migration Hadoop in the Cloud
- 11. Attunity Replicate for Enterprise and Cloud • Centralised control. On Prem and Cloud. • On-Prem to On-Prem • On-Prem to Cloud • Cloud to On-Prem • Cloud to Cloud • Wider breadth of sources and targets • Optimized, Hi-Speed data transfer • Data Compression • Parallel/Concurrent data transfer • Configurable batch sizes TargetsSources On Premises Cloud Platform HadoopRDBMS Data Warehouse Hadoop RDBMS Data Warehouse
- 12. Cloudera - Real life use cases
- 13. Hadoop is driving board level initiatives DRIVE CUSTOMER INSIGHTS IMPROVE PRODUCTS & SERVICES EFFICIENCY LOWER BUSINESS RISKS Modernize Data Architecture
- 14. Drivers for Hadoop in the Cloud Where data lives Flexible resourcing Enterprise Acceptance Key requirements: • Portability • Security without compromise • Ecosystem support
- 15. The new analytics paradigm Understand why it happened Change what happens next Determine what happened Make it happen consistently
- 16. Common Use Cases for Cloudera on Azure Perpetually “on” clusters in the cloud Common lift-and-shift cluster requirements: • High availability and disaster recovery • Cluster operational management • Cluster auto-scaling • Resource management • Security Examples of lift-and-shift-use cases in the cloud: • HBase clusters • Kafka clusters • BI analytics • Large, multi-user clusters
- 17. TRAVEL » CUSTOMER EXPERIENCE » INTERNET OF THINGS » ADVANCED ANALYTICS Preventative Maintenance • To improve traveler satisfaction and safety, a European needed to reduce downtime for critical operational machines • Cloudera Enterprise on Azure captures and correlates sensor data with transactional data to proactively assess the health of its machines and deliver necessary fixes to prevent failure
- 18. HUMAN RESOURCES » IMPROVED PRODUCTIVITY » COST REDUCTION » MACHINE LEARNING Recruiting & Job Matching • In a competitive market, Adecco S.A. wanted to improve the accuracy of its job placement technology • Cloudera Enterprise on Azure powers unrivaled search and match solution to more quickly connect qualified candidates to job vacancies • 30% reduction in time to fill vacancies • 20% reduction in job board spend within first three months of go-live date
- 19. Microsoft - The flexibility & security of Azure
- 20. Cloud is becoming integral to business transformation 22 “71% of strategic buyers cite scalability, cost, and business agility as the most important drivers for using cloud services.“ – Gigaom Research Leverage economies of scale and expertise Reshape how you engage with customers Drive new and more rapid sources of innovation
- 21. 23 Privacy & Control You control use and access to your data. Security The confidentiality, integrity, and availability of your data is protected. Compliance Designed to help meet your compliance needs. Transparency You have visibility into where your data is located and how it is managed. Our trusted cloud principles Commitment to principles worthy of your organization’s trust
- 22. Continual evaluation, benchmarking, adoption, test & audit Compliance strategy helps customers address business objectives, industry standards, and regulations, including ongoing evaluation and adoption of emerging standards and practices. Independent verification Regular verification on a regular basis by third- party audit firms. Access to audit reports Microsoft shares audit report findings and compliance packages with customers to help them assess Microsoft services against their own legal and regulatory requirements. Best practices Prescriptive guidance on securing data, apps, and infrastructure in Azure makes it easier for customers to achieve compliance. Compliance certifications Microsoft maintains a team of experts focused on ensuring Microsoft meets its own compliance obligations, which helps customers meet their own compliance requirements. The Microsoft approach to compliance To help you assess Microsoft services against your own legal and regulatory requirements 24
- 23. Private Cloud Consolidate datacenter operations MICROSOFT SOLUTIONS Windows Server System Center Windows Azure Pack Migrate to the cloud at your own pace Microsoft Confidential 25 Consistent platform and tools | Single management console Public Cloud “Europe” Achieve scale, agility and lower cost MICROSOFT SOLUTIONS Microsoft Azure Office 365 Microsoft Dynamics CRM Online Public Cloud “Germany” Achieve scale, agility and lower cost with German Data Trustee Model MICROSOFT SOLUTIONS Microsoft Azure Germany Office 365 Germany Microsoft Dynamics CRM Online Germany Hybrid Cloud Migrate less sensitive data MICROSOFT SOLUTIONS Risk Assessment and Data Governance services
- 24. Why a German cloud? 26 1Gartner, Market Trends: Cloud-Based Security Services Market, Worldwide, 2014, October 2013 Data privacy regulations in the European Union (EU) are among the strictest and strongest in the world. Regional regulation German privacy regulations are outlined and enforced through Federal Act and individual state laws. Local regulation Customers want to know where their customer data resides, who has access to it, and which country's laws govern that access. Data residency According to Gartner1, privacy requirements have severely impacted the deployment of all forms of cloud-based services in the region
- 25. 27 Complete cloud Hybrid options Commitment to compliance Commitment to innovation Microsoft Cloud differentiators
- 26. Replicate Product Architecture and Demo
- 27. Attunity Replicate Architecture Transfer TransformFilter Batch CDC Incremental In-Memory File Channel Batch Hadoop Files RDBMS Data Warehouse Mainframe Cloud On-prem Cloud On-prem Hadoop Files RDBMS Data Warehouse Kafka Persistent Store
- 28. Heterogeneous – Broad support for sources and targets RDBMS Oracle SQL Server DB2 LUW DB2 iSeries DB2 z/OS MySQL Sybase ASE Informix Data Warehouse Exadata Teradata Netezza Vertica Actian Vector Actian Matrix Hortonworks Cloudera MapR Pivotal Hadoop IMS/DB SQL M/P Enscribe RMS VSAM Legacy AWS RDS Salesforce Cloud RDBMS Oracle SQL Server DB2 LUW MySQL PostgreSQL Sybase ASE Informix Data Warehouse Exadata Teradata Netezza Vertica Pivotal DB (Greenplum) Pivotal HAWQ Actian Vector Actian Matrix Sybase IQ Hortonworks Cloudera MapR Pivotal Hadoop MongoDB NoSQL AWS RDS/Redshift/EC2 Google Cloud SQL Google Cloud Dataproc Azure SQL Data Warehouse Azure SQL Database Cloud Effective: 12/10/2015 Kafka Message Broker targets sources