Log Analysis
Download as PPT, PDF0 likes1,337 views
This document provides information about different types of log formats and log analysis. It discusses common log formats like the Common Log Format, Extended W3C Log Format, and Squid Log Format. It also covers multi-line logs, Iptables logs, and tools for log analysis like Splunk and OSSEC. The key details provided include sample log entries for each format and basic configuration steps for Splunk after installation.
![Common Log Format
Host Ident Authuser [Date] Request Status Bytes
127.0.0.1 - - [14/Oct/2010:15:41:45 +0530] "GET /announce?
info_hash=%9E%F7E%21m0%C3%BB%C8%17%AC%CF%C7K
%CCO%85L%B7S&peer_id=-AZ4510-
jsyzmsekckgz&supportcrypto=1&port=14523&azudp=14523&uploaded
=0&downloaded=0&left=144067607&corrupt=0&event=started&numwa
nt=34&no_peer_id=1&compact=1&key=R34XtNXz&azver=3 HTTP/1.1"
404 300 "-" "Azureus 4.5.1.0;Linux;Java 1.6.0_18"
(Last two fields makes the format – Combined Log Format)](https://image.slidesharecdn.com/20101016loganalysispune-101026081827-phpapp02/85/Log-Analysis-8-320.jpg)
![Squid Log Format
Native access.log
Time Duration ClientIp ResultCodes RequestMethod URL Ident Hierarchy Type
1286536314.464 475 192.168.0.188 TCP_MISS/200 627 GET
http://api.bing.com/qsml.aspx? - DIRECT/122.160.242.136 text/xml
1286536314.489 780 192.168.0.68 TCP_MISS/200 507 POST http://rcv-
srv37.inplay.tubemogul.com/streamreceiver/services - DIRECT/174.129.41.128
application/xml
Custom access.log
Dec 15 06:44:23 box squid[2011]: 192.9.101.130 TCP_MISS/200 389 POST
http://65.54.49.123/gateway/gateway.dll? - DIRECT/65.54.49.123 application/x-
msn-messenger
Dec 15 06:45:24 box squid[2011]: 192.9.101.130 TCP_MISS/200 389 POST
http://65.54.49.123/gateway/gateway.dll? - DIRECT/65.54.49.123 application/x-
msn-messenger
Dec 15 06:47:25 box last message repeated 2 times
Dec 15 06:47:32 box squid[2011]: 127.0.0.1 TCP_MISS/200 68777105 GET
http://update.nai.com/products/commonupdater/dat-5832.zip -
DIRECT/122.166.109.18 application/zip](https://image.slidesharecdn.com/20101016loganalysispune-101026081827-phpapp02/85/Log-Analysis-10-320.jpg)
Log Analysis
- 1. Thank You
- 2. Topic - Log Analysis * Not a log analysis of a hacked server
- 3. References Security Metrics – Replacing FUD *Diversion ahead
- 4. Good Metrics - - Consistently Measured - Cheap to Gather - Expressed as a number / percentage - Expressed using one unit of measure - Contextually specific – avoid So What ?
- 5. Bad Metrics - - Inconsistently Measured / Varies from person to person - Cannot be Gathered cheaply - Does not express results with numbers e.g. - ratings and grades
- 6. Log – Record of all the activities in an Application / Server or a Process Log Analysis – Extracting information from the logs
- 7. Pre-Requisites of Log Analysis - Logging should be enabled - Correct Time to be recorded in the logs - Data should not be corrupted - Known / Intuitive Log format - Patience - Caution
- 8. Common Log Format Host Ident Authuser [Date] Request Status Bytes 127.0.0.1 - - [14/Oct/2010:15:41:45 +0530] "GET /announce? info_hash=%9E%F7E%21m0%C3%BB%C8%17%AC%CF%C7K %CCO%85L%B7S&peer_id=-AZ4510- jsyzmsekckgz&supportcrypto=1&port=14523&azudp=14523&uploaded =0&downloaded=0&left=144067607&corrupt=0&event=started&numwa nt=34&no_peer_id=1&compact=1&key=R34XtNXz&azver=3 HTTP/1.1" 404 300 "-" "Azureus 4.5.1.0;Linux;Java 1.6.0_18" (Last two fields makes the format – Combined Log Format)
- 9. Extended W3C Log Format #Software: Microsoft Internet Security and Acceleration Server 2004 #Version: 2.0 #Date: 2009-10-28 00:00:01 #Fields: computer date time IP protocol source destination original client IP source network destination network action status rule application protocol bytes sent bytes sent intermediate bytes received bytes received intermediate connection time connection time intermediate username agent session ID connection ID FW1 2009-10-28 00:00:01 TCP 192.9.133.33:2179 124.153.12.25:443 192.9.133.33 Internal External Establish 0x0 LAN to Internet HTTPS 0 0 0 0 - - - - 248445 7348626
- 10. Squid Log Format Native access.log Time Duration ClientIp ResultCodes RequestMethod URL Ident Hierarchy Type 1286536314.464 475 192.168.0.188 TCP_MISS/200 627 GET http://api.bing.com/qsml.aspx? - DIRECT/122.160.242.136 text/xml 1286536314.489 780 192.168.0.68 TCP_MISS/200 507 POST http://rcv- srv37.inplay.tubemogul.com/streamreceiver/services - DIRECT/174.129.41.128 application/xml Custom access.log Dec 15 06:44:23 box squid[2011]: 192.9.101.130 TCP_MISS/200 389 POST http://65.54.49.123/gateway/gateway.dll? - DIRECT/65.54.49.123 application/x- msn-messenger Dec 15 06:45:24 box squid[2011]: 192.9.101.130 TCP_MISS/200 389 POST http://65.54.49.123/gateway/gateway.dll? - DIRECT/65.54.49.123 application/x- msn-messenger Dec 15 06:47:25 box last message repeated 2 times Dec 15 06:47:32 box squid[2011]: 127.0.0.1 TCP_MISS/200 68777105 GET http://update.nai.com/products/commonupdater/dat-5832.zip - DIRECT/122.166.109.18 application/zip
- 11. Multi-line Log Format Generated by Applications which runs multiple processes internally. Such logs are created when a single activity as seen by the End User internally translates to several different tasks in the Application. LogEntry1 of Task1 LogEntry2 of Task2 LogEntry3 of Task3 Almost all Mail Server Logs are Multi line logs. Example – Postfix and IronPort (Cisco) Email Server
- 12. Iptables Log Format Dec 5 00:17:38 box Shorewall:nic012FW:ACCEPT: IN=eth1 OUT= MAC=00:1f:e2:6c:cb:6d:00:1e:58:22:6b:30:08:00 SRC=124.153.10.16 DST=192.168.1.4 LEN=44 TOS=00 PREC=0x00 TTL=55 ID=39105 CE PROTO=TCP SPT=36597 DPT=5666 SEQ=3522285426 ACK=0 WINDOW=5840 SYN URGP=0 Dec 5 00:17:40 box Shorewall:nic012FW:DROP: IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:e0:e4:a5:46:f1:08:00 SRC=192.168.1.87 DST=192.168.1.255 LEN=229 TOS=00 PREC=0x00 TTL=128 ID=3758 PROTO=UDP SPT=138 DPT=138 LEN=209 Dec 5 00:17:47 box Shorewall:nic012nic01:DROP: IN=eth1 OUT=eth1 MAC=00:1f:e2:6c:cb:6d:00:1b:b9:63:a3:19:08:00 SRC=192.168.1.120 DST=202.54.157.139 LEN=48 T OS=00 PREC=0x00 TTL=127 ID=3754 DF PROTO=TCP SPT=1414 DPT=80 SEQ=2498894647 ACK=0 WINDOW=65535 SYN URGP=0
- 13. Splunk – Monitor, Report and Analyze live streaming / historical IT data
- 14. Basic Configuration after installation cd /opt/splunk/bin export SPLUNK_IGNORE_SELINUX=1 ./splunk start Use your browser to login to http://localhost:8000
- 15. New Apps goes to /opt/splunk/etc/apps For custom log format - update the following configuration file /opt/splunk/etc/system/local/props.conf and /opt/splunk/etc/system/local/transforms.conf with entries of new log format
- 16. OSSEC – This is an Open Source Host Based Intrusion Detection System which can work in a client – server mode.