Log Analytics Optimization
Download as PPTX, PDF0 likes811 views
The document discusses Hortonworks' log analytics optimization through its data platform, emphasizing the importance of efficient data ingestion and the integration of various analytics tools. It highlights features like content-based routing and the ability to expand log collection and analytics capabilities using their platforms. The partnership with Splunk is presented as a key component for enhancing data integration and analytics within modern data architectures.
Log Analytics Optimization
- 1. Optimizing Log Analytics from the Edge April 2016 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Hortonworks Customer Momentum ~800 customers (as of Feb 10, 2016) Publicly traded on NASDAQ: HDP Hortonworks Data Platform Completely open multi-tenant platform for any app and any data Consistent enterprise services for security, operations, and governance Partner for Customer Success Leader in open-source community, focused on innovation to meet enterprise needs Unrivaled Hadoop support subscriptions Founded in 2011 Original 24 architects, developers, operators of Hadoop from Yahoo! 800+ E M P L O Y E E S 1500+ E C O S Y S T E M PA R T N E R S
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved EMBRACE AN OPEN APPROACH MASTER THE VALUE OF DATA EVERY BUSINESS IS A DATA BUSINESS
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DATA AT REST DATA IN MOTION ACTIONABLE INTELLIGENCE MODERN DATA APPLICATIONS Actionable Intelligence from Connected Data Platforms Capturing perishable insights from data in motion Ensuring rich, historical insights on data at rest Necessary for modern data applications Hortonworks DataFlow Hortonworks Data Platform
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Optimizing Log Ingest with Hortonworks DataFlow
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why Hortonworks DataFlow? Because even the best data scientists and most powerful platforms need the right data to analyze
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Store Data Process and Analyze Data Acquire Data Perception of DataFlows: Easy, Definitive Dataflow
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Reality of Dataflows: Complex, Convoluted Store Data Process and Analyze Data Acquire Data Store DataStore Data Store Data Store Data Acquire Data Acquire Data Acquire Data Dataflow
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDF has 130+ Processors - Multiple for Log Analytics HTTP Syslog Email HTML Image Hash Encrypt Extract TailMerge Evaluate Duplicate Execute Scan GeoEnrich Replace ConvertSplit Translate HL7 FTP UDP XML SFTP Route Content Route Context Route Text Control Rate Distribute Load AMQP
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Log Analytics Systems Today LOG ANALYTICS PLATFORMNetwork Device Logs • Not all data can be captured • Not all captured data is valuable • Transport all data
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Cost Effectively Expand Storage Options of Log Data LOG ANALYTICS PLATFORM Network Device Logs HDP HDF 3. Cost effectively expand collection and grow timescale of logs collected 2. Content-based routing based on dynamic evaluation of content, attributes, priority 1. Integrate and enrich logs across data centers and security zones
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Efficiently Expand Log Ingestion from the Edge LOG ANALYTICS PLATFORM Network Device Logs HDF HDF HDF HDPHDF • Expand collection to new sources of machine data • Edge analytics to transform, enrich and prioritize content based routing • Capture and transport only valuable data
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Expand Analytics and Reporting Options with HDP LOG ANALYTICS PLATFORM Network Device Logs HDF HDF HDF HDPHDF ODBC interface traditional BI tools Easy access to log analytics data through traditional BI tools Give data scientists better tooling – Spark, Storm etc
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Expand to small scale, remote systems LOG ANALYTICS PLATFORM Network Device Logs HDF HDF HDF HDPHDF
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Optimize Log Analytics with Content Based Routing LOG ANALYTICS PLATFORM Edge analytics for cost-effective and efficient movement of machine data HDF Intelligent, content based routing, transformation and enrichment Send data to alternative systems based on value, content, priority HDP HDF HDF HDF
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk Optimization: Using HDP as Data Refinery
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk Hadoop Connect 17 Reliable bi-directional integration Import Browse Export Splunk Hadoop Connect >2000 downloads HA Indexes and Storage Commodity Servers Hadoop (MapReduce & HDFS) Report & analyze Custom dashboards Monitor and alert Ad hoc search
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk, Hunk & Hortonworks YARN Ready Partner Certified on Hortonworks Data Platform Existing Sandbox tutorial
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk, Part of the Modern Data Architecture • Bi-directional data integration between Splunk & HDP • Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics • Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization • Archiving from Splunk Enterprise to Hadoop
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk, Part of the Modern Data Architecture • Bi-directional data integration between Splunk & HDP • Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics • Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization • Archiving from Splunk Enterprise to Hadoop
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hunk + Hortonworks 21 Explore, analyze and visualize data in HDP from one integrated platform Simply point Hunk at your HDP cluster(s) and start exploring data immediately Search data, change perspectives and preview results as MapReduce jobs run INTERACTIVE EXPLORATION RICH DEVELOPER ENVIRONMENT Build big data apps on data in HDP using standard web languages and frameworks FULL-FEATURED ANALYTICS FAST TO DEPLOY AND DRIVE VALUE
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Augment Splunk Deployment with Hortonworks Data Platform Heavy Indexer Universal Forwarders HDP Enables Splunk Storage • Expansion to more data than previously feasible • Archive data from Splunk into Hadoop • Query archived Splunk data in Hadoop • Focus Splunk infrastructure on what really matters
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Find out how you much can optimize your log analytics infrastructure today. Contact [email protected]
Editor's Notes
- #9: In reality, dataflows move all over. Data is moved and stored in multiple places – sometimes interim, sometimes longterm. Data is procesed in different places, and then moved again. Complicated, convoluted, messy.
- #20: Interactively search without fixed schemas or moving data. Preview results and accelerate reports for fast search and improved cluster performance. Provide self-service analytics for business and IT stakeholders with data models and pivot. Rapidly build big data apps with a rich developer environment.
- #21: Interactively search without fixed schemas or moving data. Preview results and accelerate reports for fast search and improved cluster performance. Provide self-service analytics for business and IT stakeholders with data models and pivot. Rapidly build big data apps with a rich developer environment.