Workshop: Big Data Visualization for Security
29 likes22,200 views
The document discusses big data visualization for security, outlining a comprehensive agenda that covers data sources, processing techniques, and various tools such as Elasticsearch, Logstash, and Kibana. It emphasizes the importance of context in visualization, showcases the capabilities of the ELK stack for managing and visualizing security logs, and highlights the Moloch system for packet capturing. Additionally, it addresses data visualization principles and tools necessary for effectively analyzing and presenting security-related data.
![18 Secur i ty. Analyt ics . Ins ight .
Storing and Indexing Logs
Raw log:
Aug 2 13:29:58 pixl-ram sshd[1631]: Accepted publickey for ram from 192.168.30.1 port 49864 ssh2
Non parsed:
{“text“: “Aug 2 13:29:58 pixl-ram sshd[1631]: Accepted publickey for ram from 192.168.30.1 port 49864 ssh2”}
Parsed (through grok in LogStash):
{“text“: “Aug 2 13:29:58 pixl-ram sshd[1631]: Accepted publickey for ram from 192.168.30.1 port 49864 ssh2”,
“time”: “Aug 2 13:29:58”, “host”: “pixl-ram”, ”process”: “sshd”, “pid”: 1631}
-> structured search: time > “Aug 1 2014”](https://image.slidesharecdn.com/ue14bigdatasecviz-140915022839-phpapp01/85/Workshop-Big-Data-Visualization-for-Security-15-320.jpg)
![19 Secur i ty. Analyt ics . Ins ight .
Grok
• Instead of re-writing regexes
• Ships with about 100 patterns
• Patterns you don't have to write yourself
• It is easy to add new patterns
HOSTNAME b(?:[0-9A-Za-z].......!
IP (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]…!
IPORHOST (?:%{HOSTNAME}|%{IP})!](https://image.slidesharecdn.com/ue14bigdatasecviz-140915022839-phpapp01/85/Workshop-Big-Data-Visualization-for-Security-16-320.jpg)
![• Automatic schema inference
• Assigns analyzers (prefix indexing, etc.)
• Field properties:
• “store” [field and document level]
• “index”:
• “analyzed”: tokenized, analyzed
• “not_analyzed”: indexed as is
• “no”: no indexing
20 Secur i ty. Analyt ics . Ins ight .
ElasticSearch on Grokked Data](https://image.slidesharecdn.com/ue14bigdatasecviz-140915022839-phpapp01/85/Workshop-Big-Data-Visualization-for-Security-17-320.jpg)
![Running LogStash
For debugging:
logstash -e ‘input { … } … output { … }’ !
24 Secur i ty. Analyt ics . Ins ight .
!
Other Command line parameters:
-w <number of cores>!
--debug!
!
!
input {
stdin {
type => "stdin-type"
}
!
file {
type => "syslog-ng"
path => [ "/var/log/*.log", “/var/log/messages" ]
}
}
!
output {
stdout { }
elasticsearch{
embedded => false
host => "192.168.0.23"
cluster => "logstash-cluster"
node_name => “logstash"
protocol => “node”
}
}
Act as an ES node,
not as an unknown client](https://image.slidesharecdn.com/ue14bigdatasecviz-140915022839-phpapp01/85/Workshop-Big-Data-Visualization-for-Security-21-320.jpg)
![34 Secur i ty. Analyt ics . Ins ight .
Moloch - Couple Additions
• Web API’s
• Access meta information
• Grab PCAPs
!
• Indexing PCAP files:
! ${moloch_dir}/bin/moloch-capture -c [config_file] -r [pcap_file]](https://image.slidesharecdn.com/ue14bigdatasecviz-140915022839-phpapp01/85/Workshop-Big-Data-Visualization-for-Security-26-320.jpg)
![AfterGlow - Creating DOT/GDF Files From CSV
Parser Grapher
CSV File Graph
LanguageFile
digraph structs {
graph [label="AfterGlow 1.5.8", fontsize=8];
node [shape=ellipse, style=filled,
fontsize=10, width=1, height=1,
fixedsize=true];
edge [len=1.6];
!
"aaelenes" -> "Printing Resume" ;
"abbe" -> "Information Encryption" ;
"aanna" -> "Patent Access" ;
"aatharuv" -> "Ping" ;
}
65 Secur i ty. Analyt ics . Ins ight .
aaelenes,Printing Resume
abbe,Information Encrytion
aanna,Patent Access
aatharuy,Ping
cat file | ./afterglow –c simple.properties –t | neato –Tgif –o test.gif](https://image.slidesharecdn.com/ue14bigdatasecviz-140915022839-phpapp01/85/Workshop-Big-Data-Visualization-for-Security-53-320.jpg)
![68 Secur i ty. Analyt ics . Ins ight .
LogStash Setup - Exercise
1. Check out /home/davix/ue14
logstash-syslog.conf [read, understand!]
2. Run logstash and index data:
! sudo /opt/logstash/bin/logstash -f logstash-syslog.conf!
! head -10 firewall | nc localhost 5000!! # send data
3. Check what’s in LogStash:
sudo /etc/init.d/logstash-web start!
! open http://localhost:9292 !# kibana
4. Use script to extract data
read_es.py [check out the script]
update the script to output a (src_ip, dst_ip, dst_port) tuple
5. Convert the CSV output to a GDF file to then load into Gephi
OR create a TM3 file for the treemap tool
curl 'http://localhost:9200/_all/_search?q=ACCEPTED'
curl ‘http://localhost:9200/twitter/_search?q=user:kimchy'](https://image.slidesharecdn.com/ue14bigdatasecviz-140915022839-phpapp01/85/Workshop-Big-Data-Visualization-for-Security-56-320.jpg)
Workshop: Big Data Visualization for Security
- 1. Big Data Visualization for Security UE14 - Romania September 2014 Raffael Marty, CEO
- 2. 3 Secur i ty. Analyt ics . Ins ight . I am Raffy - I do Viz! IBM Research
- 3. 4 Secur i ty. Analyt ics . Ins ight . Agenda Introduction Data Sources DAVIX Log Data Processing • Big Data Ecosystem • Security Big Data Tools • Managing Security Data • Visualizing Big Data
- 5. 8 Secur i ty. Analyt ics . Ins ight . Big Data - The Three V’s Velocity Volume Variety
- 6. The Big Data Ecosystem 9
- 7. 10 Secur i ty. Analyt ics . Ins ight . Hadoop Ecosystem Mahout machine learning Hive data warehouse HiveQL query lang Pig programming language (pig latin) HBase big data store rndm read and write auto sharding Map Reduce Impala interactive SQL queries distributed file system data redundancy fault-tolerance HDFS random, real-time read/write access append only namenode / datanode architecture Zookeeper centralized “brain” Sentry Storm
- 8. Berkeley Data Analysis Stack (BDAS) 11 Secur i ty. Analyt ics . Ins ight . https://amplab.cs.berkeley.edu/software/ SparkSQL
- 9. http://elasticsearch.org 12 Secur i ty. Analyt ics . Ins ight . Elastic Search • Schema free & document oriented • Simple HTTP interface • indexes JSON documents • Queries, aggregations, highlighting, etc. • Distributed - super easy to add nodes • Real-time indexing • Based on Lucene • Replication • Partitioning / sharding • how an index is assigned to nodes • Snapshots Up and running in 10 minutes!!
- 10. 13 Secur i ty. Analyt ics . Ins ight . Elastic Search - Admin Interface
- 11. Big Data Security Tools 14
- 12. 15 Secur i ty. Analyt ics . Ins ight . ELK Stack • Elastic Search • LogStash • Kibana
- 13. LogStash http://logstash.net/ input filter output http://www.elasticsearch.org/overview/logstash 16 Secur i ty. Analyt ics . Ins ight .
- 14. logstash http://logstash.net/ input files syslog email tcp socket Flume 17 Secur i ty. Analyt ics . Ins ight . ! AMQP STOMP Beanstalk redis ! twitter HTTP filter timestamp parsing anonymize drop events parse fields (grok) multiline joins output ElasticSearch Graylog2/GELF MongoDB Nagios TCP syslog WebSockets ! AMQP STOMP beanstalk redis messaging formats avro msgpack thrift xml protobuf csv
- 15. 18 Secur i ty. Analyt ics . Ins ight . Storing and Indexing Logs Raw log: Aug 2 13:29:58 pixl-ram sshd[1631]: Accepted publickey for ram from 192.168.30.1 port 49864 ssh2 Non parsed: {“text“: “Aug 2 13:29:58 pixl-ram sshd[1631]: Accepted publickey for ram from 192.168.30.1 port 49864 ssh2”} Parsed (through grok in LogStash): {“text“: “Aug 2 13:29:58 pixl-ram sshd[1631]: Accepted publickey for ram from 192.168.30.1 port 49864 ssh2”, “time”: “Aug 2 13:29:58”, “host”: “pixl-ram”, ”process”: “sshd”, “pid”: 1631} -> structured search: time > “Aug 1 2014”
- 16. 19 Secur i ty. Analyt ics . Ins ight . Grok • Instead of re-writing regexes • Ships with about 100 patterns • Patterns you don't have to write yourself • It is easy to add new patterns HOSTNAME b(?:[0-9A-Za-z].......! IP (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]…! IPORHOST (?:%{HOSTNAME}|%{IP})!
- 17. • Automatic schema inference • Assigns analyzers (prefix indexing, etc.) • Field properties: • “store” [field and document level] • “index”: • “analyzed”: tokenized, analyzed • “not_analyzed”: indexed as is • “no”: no indexing 20 Secur i ty. Analyt ics . Ins ight . ElasticSearch on Grokked Data
- 18. 21 Secur i ty. Analyt ics . Ins ight . Grok Patterns Pattern database located in: /opt/logstash/patterns ! Debug Grok rules: http://grokdebug.herokuapp.com/
- 19. 22 Secur i ty. Analyt ics . Ins ight . LogStash UI - Kibana
- 20. • Block POST / PUT / DELETE to ES instance • Older versions: script.disable_dynamic: true! ! action.destructive_requires_name: true! • Use aliases to allow only certain users access to certain indexes • Use iptables to block ports (9200, 9300, …) • Performance tuning: • https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/ 23 Secur i ty. Analyt ics . Ins ight . Running ElasticSearch
- 21. Running LogStash For debugging: logstash -e ‘input { … } … output { … }’ ! 24 Secur i ty. Analyt ics . Ins ight . ! Other Command line parameters: -w <number of cores>! --debug! ! ! input { stdin { type => "stdin-type" } ! file { type => "syslog-ng" path => [ "/var/log/*.log", “/var/log/messages" ] } } ! output { stdout { } elasticsearch{ embedded => false host => "192.168.0.23" cluster => "logstash-cluster" node_name => “logstash" protocol => “node” } } Act as an ES node, not as an unknown client
- 22. 25 Secur i ty. Analyt ics . Ins ight . Running Kibana Authentication not built in Use nginx as a proxy For example: https://github.com/elasticsearch/kibana/blob/master/sample/nginx.conf
- 23. 26 Secur i ty. Analyt ics . Ins ight . Moloch Open source, large scale IPv4 packet capturing, indexing and database system powered by elastic search. Web interface for PCAP browsing, searching, reporting, and exporting PCAPs https://github.com/aol/moloch
- 24. 27 Secur i ty. Analyt ics . Ins ight . Moloch – Components • Capture • Sniffs the network interface, • Parses the traffic and creates the Session Profile Information (aka SPI-Data) • Writes the packets to disk ! • Database • Elasticsearch is used for storing and searching through the SPI-Data ! • Viewer • A web interface that allows for GUI and API access from remote hosts
- 25. Moloch – Capture – SPI-Data Types • Moloch parses various protocols to create SPI-Data: • IP • HTTP • DNS • IP Address • Hostname • IRC • Channel Names • SSH • Client Name • Public Key • SSL/TLS • Certificate elements of various types (common names, serial, etc) ! 28 Secur i ty. Analyt ics . Ins ight . • This is not an all inclusive list
- 26. 34 Secur i ty. Analyt ics . Ins ight . Moloch - Couple Additions • Web API’s • Access meta information • Grab PCAPs ! • Indexing PCAP files: ! ${moloch_dir}/bin/moloch-capture -c [config_file] -r [pcap_file]
- 27. 35 Secur i ty. Analyt ics . Ins ight . PacketPig • Analyze PCAP files using Apache Pig • Number of scripts made available • e.g., running SNORT on the PCAPs ! https://github.com/bigsnarfdude/packetpig pig -x local ! -f pig/examples/binning.pig ! -param pcap=data/web.pcap
- 28. Security Onion •Bro IDS, your choice of Snort or Suricata, Sguil analyst console, ELSA, Squert, Snorby and capME web interfaces •All setup to work with each other out of the box http://securityonion.blogspot.com/ pixlcloud | turning data into actionable insights copyright (c) 2014
- 29. Storing Security Data 37
- 30. PCAP in HDFS or HBase Row or columnar, fixed schema? Unstructured in ElasticSearch, enrich on ingestion? ES or relational 38 Secur i ty. Analyt ics . Ins ight . Data Type and Use • What data do you have? • PCAP • Flows • Context, (e.g., threat feeds) • “Text” logs • What’s your use-case? • Search • Analytics • Forensics on PCAP Index -> Elastic Search Columnar, SQL enabled Moloch? Or extract meta data and store PCAP in HDFS/HBase
- 31. 39 Secur i ty. Analyt ics . Ins ight . OpenSOC
- 32. Raffael . Marty @ pixlcloud . com 40 Visualization
- 33. Visualization To … Present / Communicate Discover / Explore 41 Secur i ty. Analyt ics . Ins ight .
- 34. 42 Secur i ty. Analyt ics . Ins ight . Show Context 42
- 35. 43 Secur i ty. Analyt ics . Ins ight . Show Context 42 is just a number and means nothing without context
- 37. Use Numbers To Highlight Most Important Parts of Data 45 Secur i ty. Analyt ics . Ins ight . Numbers Summaries
- 38. Visualization Creates Context Visualization Puts Numbers (Data) in Context! 46 Secur i ty. Analyt ics . Ins ight .
- 39. Principals of Analytic Design • Show comparisons, contrasts, differences • Show causality, mechanism, explanation, systematic structure. • Show multivariate data; that is, show more than 1 or 2 variables. 47 Secur i ty. Analyt ics . Ins ight . ! by Edward Tufte
- 40. 48 Secur i ty. Analyt ics . Ins ight . Add Context Additional information about objects, such as: • machine • roles • criticality • location • owner • … • user • roles • office location • … source destination machine and user context machine role
- 41. Traffic Flow Analysis With Context 49 Secur i ty. Analyt ics . Ins ight .
- 42. 50 Secur i ty. Analyt ics . Ins ight . Visualize Me Lots (>1TB) of Data ! ! SecViz is Hard!
- 43. 51 Secur i ty. Analyt ics . Ins ight . Data Visualization Workflow Overview Zoom / Filter Details on Demand Principle by Ben Shneiderman
- 44. 52 Secur i ty. Analyt ics . Ins ight . Backend Support This visualization process requires: • Low latency, scalable backend (columnar, distributed data store) • Efficient client-server communications and caching • Assistance of data mining to • Reduce overall data to look at • Highlight relationships, patterns, and outliers • Assist analyst in focussing on ‘important’ areas
- 46. 54 Secur i ty. Analyt ics . Ins ight . Mondrian • Graphs: • Histogram • Box plots • Scatterplot • Mosaicplots • Parallel Coordinates • Boxplots • ... • Linking, brushing, … • Reads CSV files http://www.theusrus.de/Mondrian/
- 47. TM3 Input files: Source Port Destination Action STRING INTEGER STRING STRING 10.0.0.2 80 23.2.1.2 failed 55 Secur i ty. Analyt ics . Ins ight . Treemap 4.1 www.cs.umd.edu/hcil/treemap
- 48. Gephi http://gephi.org •Gephi UI • interactive link graphs • multiple layout algorithms • reads: CSV, DOT, GDF, etc. • graph metrics •Gephi Toolkit • APIs • Gephi Plugins • Gephi ‘Platform’ • adding JavaFX components 56 Secur i ty. Analyt ics . Ins ight .
- 49. 57 Secur i ty. Analyt ics . Ins ight . Visually Finding Insight in Gephi 1. Loading Data
- 50. 58 Secur i ty. Analyt ics . Ins ight . Visually Finding Insight in Gephi 2. Run Layout Algorithm (Force Atlas 2)
- 51. 60 Secur i ty. Analyt ics . Ins ight . Visually Finding Insight in Gephi 3. Use Degree as color and size of nodes
- 52. 63 Secur i ty. Analyt ics . Ins ight . Visually Finding Insight in Gephi 6. Use Preview and export Graph
- 53. AfterGlow - Creating DOT/GDF Files From CSV Parser Grapher CSV File Graph LanguageFile digraph structs { graph [label="AfterGlow 1.5.8", fontsize=8]; node [shape=ellipse, style=filled, fontsize=10, width=1, height=1, fixedsize=true]; edge [len=1.6]; ! "aaelenes" -> "Printing Resume" ; "abbe" -> "Information Encryption" ; "aanna" -> "Patent Access" ; "aatharuv" -> "Ping" ; } 65 Secur i ty. Analyt ics . Ins ight . aaelenes,Printing Resume abbe,Information Encrytion aanna,Patent Access aatharuy,Ping cat file | ./afterglow –c simple.properties –t | neato –Tgif –o test.gif
- 54. Hands On 66
- 55. 67 Secur i ty. Analyt ics . Ins ight . Processing Pipeline 1. Get data into ElasticSearch Parse data first, then store in ES 2. Get data out of ES (query) Get into data format for visualization tool (e.g., CSV) 3. Visualize in the visualization tool Potentially translate CSV into other format (e.g., DOT, GDF) Process the data (aggregation, enhancement, etc)
- 56. 68 Secur i ty. Analyt ics . Ins ight . LogStash Setup - Exercise 1. Check out /home/davix/ue14 logstash-syslog.conf [read, understand!] 2. Run logstash and index data: ! sudo /opt/logstash/bin/logstash -f logstash-syslog.conf! ! head -10 firewall | nc localhost 5000!! # send data 3. Check what’s in LogStash: sudo /etc/init.d/logstash-web start! ! open http://localhost:9292 !# kibana 4. Use script to extract data read_es.py [check out the script] update the script to output a (src_ip, dst_ip, dst_port) tuple 5. Convert the CSV output to a GDF file to then load into Gephi OR create a TM3 file for the treemap tool curl 'http://localhost:9200/_all/_search?q=ACCEPTED' curl ‘http://localhost:9200/twitter/_search?q=user:kimchy'
- 57. 69 Secur i ty. Analyt ics . Ins ight . BlackHat Europe - Workshop VISUAL ANALYTICS DELIVERING ACTIONABLE SECURITY INTELLIGENCE October 14, 15 - Amsterdam
- 58. Security Visualization Community Share, discuss, challenge, and learn about security visualization. •http://secviz.org •List: secviz.org/mailinglist •Twitter: @secviz pixlcloud | turning data into actionable insights copyright (c) 2013