Analyzing the World's Largest Security Data Lake!
1 like1,982 views
The document discusses Symantec's CloudFire Analytics platform for analyzing security data at scale. It describes how CloudFire provides Hadoop ecosystem tools on OpenStack virtual machines across 50+ data centers to support security product analytics. Key points covered include analytics services and data, administration and monitoring using tools like Ambari and OpsView, and plans for self-service analytics using dynamic clusters provisioned through CloudBreak integration.
Analyzing the World's Largest Security Data Lake!
- 1. Hadoop Summit 2015 CloudFire Analytics: Transforming Security Analyzing Symantec’s security data lake Stephen Brodsky and Darrell Kienzle
- 2. Hadoop Summit 2015 Outline CPE CloudFire: Analytics and Products1 Analytics Services and Data2 Analytics Administration and Monitoring3 Self-Service Analytics and Dynamic Clusters4 Symantec CloudFire Analytics 2
- 3. Hadoop Summit 2015 CPE CloudFire: Analytics and Products • CPE – Cloud Platform Engineering – Symantec’s Private Cloud for security products and analytics – Spans 50+ data centers around the world • CloudFire – CPE’s scalable cloud platform – New data centers, new hardware, scalable build-out – All open source – OpenStack for virtualization – Analytics for big data analysis • Cloud for bringing together and integrating Symantec’s: – Products – Big Data, Analytic applications, and Services – Compute, Network Symantec CloudFire Analytics 3
- 4. Hadoop Summit 2015 Analytics for supporting Security Products Goals of the Security Teams 1. Do we have the data? 2. Can we analyze the data? 3. Can we provide timely Insights? CloudFire Analytics for supporting Security 1. Data available • All frequently used Security Data available • Data at scale: Data available for parallel analysis (PB scale) • Leveraging CPE Data Center Availability, Compute, Net 2. Analysis engines • Hadoop ecosystem engines (MR, Hive, Kafka, Spark, HBase, Storm, Phoenix, ++) • Analytics Pipeline 3. Analysis in near real-time • Analytics timescale days/hours -> seconds • Batch -> Streaming 4Symantec CloudFire Analytics
- 5. CPE Analytics Architecture Overview Inbound Messaging (Data import, Kafka) Products Distributed Storage (HDFS), Metal or Virtual (OpenStack) Servers Analytic Applications, Workload Management (YARN) Stream Processing (Storm, Spark) Real-time Results (HBase, ElasticSearch) Query (Hive, Spark SQL) Device Agents Telemetry, Data Threats: Top Web-based Attacks 192 web-based attacks recorded last month 36% 30% 14% 10% 7%3% Malvertisement Exploit Kit Suspicious Download Analytics Clusters • All major open source engines • PB-scale Data Store • Key Telemetry • Multi-Data Center • Administration • Monitoring • Prod, Dev/Test • Application Deployments CloudFire Analytics Data Transfer 5
- 6. Hadoop Summit 2015 A key security Product Symantec CloudFire Analytics 6 • Live with hundreds of external customers • Improved time to analyzed results from hours to seconds (5000x) at production scale • Leverage Streaming Analytics • Move analytics to more powerful, high performance open source technologies – Kafka for queuing – Storm for analytics • Improve analytics – Server reputation – URL reputation • Moving forward – Leverage Analytics Pipeline – NoSQL DBs – Graph DBs HDFSMonitors
- 7. Hadoop Summit 2015 Analytics Services and Data Symantec CloudFire Analytics 7
- 8. Hadoop Summit 2015 Product Journey with CloudFire Analytics Symantec CloudFire Analytics 8 SDAP current • Scale? • Availability? • Usage? • Hours/Days timeframe Batch Analytics •Data Scale: Multi-PB HDFS •Analytics scale: Hadoop MR, Hive •Availability – high •Usage prioritization •Minutes/Hours timeframe Streaming Analytics •Analytics Scale: Kafka, Storm, Spark, ElasticSearch, HBase, ++ •Availability – high •Usage prioritization •Seconds timeframe Application style • Bring in data • Analyze • Architecture: Serial->Parallel • Workflow oriented Application style • Bring in data in zip files • Unzip, load, SQL • Architecture: Serial • Application oriented Application style • Stream in data directly • Analytics pipeline • Architecture: Streaming Parallel • Streaming oriented
- 9. CloudFire Analytics Pipeline Data Sources Streaming Analysis Batch Analysis Actively Available Results Online Access Research Telemetry Clients World Cloud Data Assemble Analytic Applications Analytic Pipeline Message Queue 9 Storm, Kafka, Spark Hive, Spark HBase, HDFS, Kafka, Cassandra HBase, Index, Cache
- 10. Hadoop Summit 2015 CloudFire Analytics Services Symantec CloudFire Analytics 10 Analytic Services and Data Transfer • Hadoop, YARN - Compute • HDFS/Knox/HttpFS – File Storage • Hive, Pig – SQL Batch • Oozie – Workflow and Scheduling • Service Endpoints – Admin / BDSE • Storm – Streaming • Kafka - Messaging • HBase – BigTable column store • Spark (SQL,ML) – Stream analytics • Research and admin CLI VMs • MTS Parallel file Import / Export • Falcon data management • ElasticSearch indexing • Cassandra, Graph, Drill, Solr Web UIs Admin Tools • Ambari - Admin • OpsView (Zabbix) – Uptime monitor • YARN and HDFS UIs – Perf & Logs • Ganglia & Nagios – Performance • Storm & Kafka lag – Perf/Problems • LMM – Problem determination & metrics • Resource Manager, Name Node UIs • Network data transfer monitoring • Puma performance usage analysis • Continuous Validation • Ranger user management Data Science Tools • HUE – SQL, Jobs, Files, Workflow • Ambari Views
- 11. Hadoop Summit 2015 Analytics Administration and Monitoring Symantec CloudFire Analytics 11
- 12. Hadoop Summit 2015 Ambari Analytics Administration 12CPE CloudFire Analytics Extensible administration platform for automated deployment, monitoring and alerting, configuration management, rolling upgrades, and rolling restarts.
- 13. Hadoop Summit 2015 Ambari + ElasticSearch – Ambari Stack extensibility Symantec CloudFire Analytics 13 Ambari custom service for ElasticSearch, a new service to enable Ambari to manage. Ambari deploy, start/stop, config, monitor Uses the new Ambari Views, integrated via iFrame Elastic HQ admin UI as our use case. Keeps the dashboard clean, yet extensible. On github to share…
- 14. Hadoop Summit 2015 Ambari Views with iFrame – Ambari extensibility Symantec CloudFire Analytics 14
- 15. Hadoop Summit 2015 Cluster Usage Analysis using Analytics Symantec CloudFire Analytics 15
- 16. Hadoop Summit 2015 Ambari Metrics Rapid Metrics Dashboards Construction - Reliability Symantec CloudFire Analytics 16 Prototype for building out flexible dashboards of most interesting application and platform metrics. Alerting available via the LMM service.
- 17. Hadoop Summit 2015 Ambari Metrics Rapid Metrics Dashboards Construction - Availability Symantec CloudFire Analytics 17
- 18. Hadoop Summit 2015 OpsView / Nagios monitor and alert service Symantec CloudFire Analytics 18
- 19. Hadoop Summit 2015 Data Transfer Network Monitor Symantec CloudFire Analytics 19
- 20. Hadoop Summit 2015 Data Transfer Rate Monitor Symantec CloudFire Analytics 20
- 21. Hadoop Summit 2015 Self-Service Analytics and Dynamic Clusters CloudBreak integration and Development In Progress Symantec CloudFire Analytics 21
- 22. Hadoop Summit 2015 Self-Service Analytics and Dynamic Clusters CloudBreak integration and Development In Progress Symantec CloudFire Analytics 22 Purpose ● Create and connect Analytics Clusters: ● Authoritative Data Lake, Regional Hubs ● Development, Test, Integration, Staging, and Production ● Ability for CloudFire users to spin up new clusters to develop their applications. Feature ● One-click spin up of Analytics clusters using OpenStack VMs ● VMs that developers spin up are part of their own OpenStack quota ● Developers will have admin access to the cluster they spin up, so they can install more services if needed. Automation ● We control the version of the software by using the same automation code that we use for deploying our production clusters.
- 23. Hadoop Summit 2015 Creating and managing clusters Symantec CloudFire Analytics 23 Cluster Directory
- 24. Hadoop Summit 2015 Self-Service Analytics Architecture Symantec CloudFire Analytics 24 Managed Dynamic Clusters Cross-Cluster Capabilities Cluster Management Cluster 1 Cluster Directory Continuous Validation & Monitoring Cluster 2 Cluster ... Cluster N Data Catalog Analytics Tools Each Cluster: • Analytic Services • Deployment • Networking • Monitoring integration • OpenStack VMs + Containers • Deployed Applications Assemble Analytic Applications Analytic Pipeline Message Queue