VMware Log Insight

Editor's Notes

• #5: More materials and steps by steps guides are available at our blogs.
• #8: It can digest any type of log data. The users don’t need to think about it, they can just send their data to Log Insight. Log Insight automatically identified structures in the data and create a high performance index for performing analytics Unlike databases there is no need to engage db admins to Extract Transform and Load (ETL) the data. Just send it over It scales up and down very well. It can ingest TBs of data per node per day. It is much faster and much more efficient than the competition It has a configurable retention policy, i.e. you attach 500GB storage and if it runs out of space it rotates the old data out. Optionally they can be written to an archive, making it virtually maintenance free It ships with out of the box knowledge of vSphere logs in the form of a Content Pack, a collection of queries, alerts, dashboards and fields. We are also working with partners to include more Content Packs (e.g. for storage, applications, network devices) as well as more Vmware producst (Nicira/Networking, View, etc.)
• #10: A regional Bank performed an internal audit on its VMware environment. One area of improvement is the speed to produce report on “Who does what action to what object when”. Object here can mean any vSphere objects, such as VM, ESXi Host, Cluster and vCenter. The action here includes commands issued at the ESXi Console via SSH session. Audit data to be stored for >1 year. Reason is compliance with regulation. A regional bank engineering evaluated VMware Log Insight 2.5. While there are 1-2 minor limitations, the product helps the bank in audit compliance. Tips: do not get bogged down on the minor limitations. 80/20 can be good enough
• #11: We can go back in time for months. This is showing almost 2 months of data. To see all the vCenter Tasks, all we need is to select a built-in variable called VC Task Type. A log entry that has this field exist will appear. We are showing the data by day. You will see later we can change this to second when we zoom. Other than the chart, you can also see it in table format.
• #12: As there are a lot of results, let’s see if there are 1-2 tasks that are dominating the result. We can group the result by Task Type. To make it easier to see, I’ve removed the time line. As we can see here, there is 1 tasks that happens >20000x! That’s worth investigating on why it’s happening so often. As the topic today is for audit, let’s remove the top 3 tasks as they are not relevant.
• #13: We’ve removed the top 3 tasks. We simply say the task does not contain these 3. As you can see, the bar chart is easier to see now. The scale has changed from 20K to 200. Since the use case is audit, let’s drill down to the task “Reconfigure Virtual Machine”. We want to know who makes the changes.
• #14: Drilling down to that single task. As you can see, it’s all just that 1 task. Since it’s only 1 task, we are showing the time line again so we can see when the changes are made. Let’s now group the event by VM, so we can see the VM names.
• #15: Same chart, but showing the break down of the VMs. We can see the list of VMs in the legend. We can see that 1 VM has a lot of changes. This is a regular changes, expected by VDPA product, so I’m gonna exclude it. The changes were all made by its service account. So let’s exclude that account.
• #16: Excluding the service account. All we have to do is say VC User Name not matching that user name. There is a lot less data now. The bar chart now showing maximum 12 changes on any given day, less than half of the 30 we saw earlier. The bar chart is also easier to see. Let’s drill down further to a single VM. I’ll take one of these VRNI VM.
• #17: Drilling down to a single VM. We can see it’s all done by my brother Hicham Mourad, architect from our technical marketing team. We know what date he changes. Let’s see what time.
• #18: Drilling down the time. The scale is now down to 1 hour. We can see he made 3 changes only 9 July at 3 different hours, but 3 changes within the hour on 13 July.
• #22: We are interested in events impacting our consumer (VMs), so let’s filter it out.
• #25: What does this screen tell us?
• #29: Now… can you guess the snapshot name? It’s in the log above. Hints: I was singing an old songby The Beatles. Ok, it wasn’t technically singing, it was a bad attempt at singing
• #33: http://virtual-red-dot.info/what-are-the-esxi-hosts-that-a-vm-has-ever-run-on/
• #38: Storage latency at vmkernel level by Device There is an outlier at 6000 ms on a magnetic disk.
• #39: In this example, we should check the latency at the Magnetic Disk on 22 - 23 May. However, due to time limit in this presentation, we will move on.
• #43: This is our present architecture. It has evolved throughout the year.
• #48: What I faced back in May 2015 may not be applicable. Certain products limitations have been addressed. So I am sharing the relevant ones. I’m also adding things that I think you might encounter, that luckily I did not have to.
• #50: Most VMware admin do not even look at the logs. Most do not even understand the various logs well.
• #56: Issue: Sending syslog over the WAN can consume a large amount of bandwidth. A bug in product could trigger excessive logs. If the link is down or busy, logs are lost. There is no buffering at the sender site (e.g. ESXi does not buffer) Possible solution A Log Insight server (or cluster thereof) can receive logs and then forward those logs on to another Log Insight server (or cluster thereof) via CFAPI. The chart below compares native syslog vs Log Insight Ingestion API. It shows around 30 Mb vs 1 Mb due to compression Arcsight is using the syslog protocol Reference: http://sflanders.net/2015/03/04/12-reasons-why-you-should-use-the-log-insight-forwarder Ingestion API (default protocol) should be used if since the remote destination is another Log Insight instance Forwarder adds complexity Need to be setup & managed. No license required. Forwarders using the ingestion API support sending events via encrypted channel (SSL). Forwarders minimize the attack vector of the primary Log Insight instance / remote syslog destination. In short, if the only devices allowed to send events to the primary Log Insight are the Forwarders then security becomes easier. It means simpler inter-segment routing/firewalling. You aggregate logs within local LI forwarder & allow it to connect to the main LI server via single connection and single firewall rule (else think about 100s of devices connecting to the remote LI server) This is a compelling reason to deploy forwarders in every datacenter. In a regional bank, this is also a reason why we choose Forwarder. The architecture diagram distinguish between secured and unsecured channel Forwarders are complete Log Insight instances and offer the same options as regular Log Insight instances. vSphere integration can be configured on forwarders keeping traffic local to vSphere / forwarder. Forwarders using the ingestion API can add metadata to events add additional information to an event to make it easier to query for or correlate over. While many syslog agents provide a solution to this problem (see this post for more information), they do so by manipulating the actual event instead of providing supplemental or metadata information. The Log Insight forwarder offers a tags option and supports JSON key/value pairs for specifying tags. This makes it possible to add information such as which datacenter the event came from. Forwarders support filtering of events:  Filtering makes it possible to ensure “noise” is not ingested by the central Log Insight instance. However, need to define them manually one by one. Forwarders using the ingestion API can respond to throttle requests (i.e. back pressure) and can ensure no message loss during intermittent connection issues back pressure is handled by sending a 503 HTTP status code while connection issues can be detected and prevented against through REST calls. The Log Insight forwarder provides a configurable disk-backed queue used to save events until the server has acknowledged ingestion. This disk-backed queue prevents drops of forwarded events during intermittent connection issues where syslog events already sent, but not acknowledged by the server, can be lost after the TCP ACK window expires. Free versions of syslog agents like Rsyslog and Syslog-NG do not provide back pressure functionality nor do they provide a disk-based queue (some do provide a configurable in-memory queue). In addition, these agents rely only on the retry abilities of the TCP protocol (e.g. TCP ACK window) if the TCP protocol is being used to send the traffic–none of this is available for UDP. Forwarders can server as a backup: events that are forwarded are also stored on the forwarder. In addition, forwarders are complete Log Insight instances and offer the same UI as regular Log Insight instances. This means if the central Log Insight instance becomes temporarily unavailable the events still exist on the forwarder, users can be given direct access to forwarders, alerts can be configured on forwarders, etc. The forwarder local capacity can be configured to store events for as long as desired. We can keep the data small, as we are not going to query the Forwarder. 1 day of data is sufficient. This keeps them light Forwarders make it easier to support disaster recovery (DR): The forwarder is a Log Insight instance; it contains the events it forwarded and supports querying. In addition, forwarders can send events to up to 10 different destinations. This make it easy to forward events to 2 different remote destinations (e.g. Log Insight instances) to handle complete DR. Forwarders make it possible to support variable retention periods:  Since forwarded events can be filtered, it is possible to send security events to one Log Insight instance where events are kept for 90 days on the remote Log Insight instance and all other events to a separate Log Insight instance where events are kept for 30 days. In ank, we will take advantage of this. 1 central instance for Security purpose (longer retention), and 2 other instance for general purpose (60 days)