Kemp LB for Log Insight Mgr

Just recently I was playing around with log insight or as they now like to call it vRealize Log Insight. One of the new features with 2.5 is the ability to have to have an integrated load balancer. In previous versions VMware allowed for log insight worker nodes to scale out but this introduced an issue with evenly distributing the load. With 2.5, the claim is an external load balancer is no longer needed.

KEMP has been in the industry for some time and offers load balancer for all kinds of solutions. In fact it was one of the first vendors to ever make a virtual load balancers for VMware ESXi back when it was just ESX as well as other hypervisors. Some of their VMware specific load balancers can be found here. The one we are interested in is called the LoadMaster for VMware vCenter Log Insight Manager. I know this can be a mouthful. But its functionality is pretty straightforward with simple deployment and maintenance.

I don’t want to go into the details of how to deploy KEMP’s LB. We will be referring to it as the VLM (Virtual LoadMaster). It is an OVF that can be downloaded from the KEMP Website. Once the OVF is deployed it takes minutes for this bad boy to start working. Their deployment guide can be found here which covers their entire process of deployment . Hence, there is no point in me repeating the same information. However, I will point out a couple of things which might make your deployment a bit easier specially if load balancing is not something you don’t work with on the regular basis or if you are new to log insight:

  1. You will need at least 2 Log Insight nodes deployed (you can work with 1 but then what’s the value of a LB?)
  2. Do not enable the ILB (internal load balancer if you are using log insight 2.5)
  3. Do not forget to install the Log Insight Add On pack once you have deployed the VLM (section 2)

NOTE: The LoadMaster build that will be posted to in early February will include the Add On Pack by default.

  1. A virtual address is the IP of the service often referred to as a VIP. Basically this will be the address your clients will connect to.
  2. The real servers in this case will be your Log Insight nodes. Once the client connects to the virtual IP, the VLM will forward them to one of the real servers (Log Insight nodes) based on configured scheduling methods and health checks.

The image below that I borrowed from KEMP does a pretty good job in giving you an over of what the VLM does.


I don’t generally deal with load balancers in general so I felt it was important for me to clarify some of the above information. The good news is that I was able to deploy and make this work within minuets and if I can, anyone can. I deployed 2 Log Insight nodes and configured my virtual services in VLM as well as added the two nodes as the real servers.. After pointing my ESXi servers to the virtual IP addresses, VLM was put to work.  Read the rest of this entry »

Shell.log Holmes – CSI ESXi

I think most folks in IT are investigative / analytical to a certain degree. After all, it is kind of important to have those traits in order to identify and fix issues right? So I was going through the log files in /var/log/ on an ESXi 5 host and I noticed a file that I have somehow ignored until now. As you may already know, in ESXi 5 there is a nice breakdown of log files that are separated by the kind of information they collect.  This KB article has some very good information on what these log files record and where are they located.

I am interested in the shell.log file. Apparently this file is a spy on your hosts. This file logs everything that you have attempted to do on your host. By everything, I mean EVERYTHING. That means every time you SSHed into the host, it will remember that. Every time you typed in a command, it will be there. Obviously every time you mistyped a command it would be there too. But I think this information can be very handy when doing some forensics to figure out why your hosts isn’t doing what it was supposed to. You can tell from the screenshot I issued esxtop, then looked at the shell.log file, then did a ls -l and switched directories and so on. Such an invasion of privacy right 😀

So how do you know what user issued the command is one of the obvious questions. For that we need the help of another file in the same /var/log location. This one is auth.log file. Using this file you can figure out what user got authenticated using the same world ID. This kb article describes the process as well.

To put it simply, take a look at the output on the right in white. Notice the [2521971], that is basically the world ID associated with these commands and events that occurred on this host. Now we will go to /var/log/auth.log file and find out who caused those logs in the shell.log file using that World ID.

On the right in blue is a screenshot from the auth.log file. See what it says at the very bottom next to the World ID we are interested in. Yep! Thats Mr. Root right there. Now of course this wouldn’t really help much in finding out “who” issued the command unless only one person knew the root password. Which is also not good.

So I wanted to see how this would behave if I login using a non-root local account and simply did a su -. I created a local user called “bilal” and  SSHed into the host. On the right in white is the output from the shell.log file of that event. Notice it shows my ‘su -‘ command and world ID [2523034] is associated with it.

Upon looking at the auth.log file you can easily tell who opened the session that has World ID [2523034] associated with it. So there, if you do follow best practice and don’t allow root for ssh connection, you can tell what other local user account was used to do the su and then track back the user that was used. Now if you are using a generic local account then we are back to the square one.

This made me want to test the obvious. How many do actually create multiple local accounts on the hosts really. So I added my host to the domain, and created a ESXitemp account on a domain called “cloud”. Gave this account admin rights on the host and SSHed to host. I then went and did the “su -” to see how this played. The su in this case is associated with World ID [2525316].

Now the only thing left to do was to go to the auth.log file and confirm the cloudESXitemp account was responsible for the events. As you can tell in the auth.log file, World ID [2525316] confirms the account we are interested in. Although it has an extra backslash between the domain and the username which I am not sure why. But in any case, you can still cross reference when using a domain account and figure out what happened, who did it etc.

I wanted to see how commands issued from VMA would get logged but I am in the middle of getting some new hardware and can’t afford to spin up a VMA appliance in my lab. I know VMA is tiny, but trust me when I say this.

So what’s the moral of the story? Follow best practice and disable root from SSHing into your hosts. Try and use unique accounts (preferably AD as it already exists in most shops) to login to the host and su when needed. This way when you are asked to find out what happend and who did it, you know where to go and what to look for instead of a generic account that can SSH and whose password is shared by many. Last but not the least, if you dont have a remote syslog configured, make sure your ESXi logs are persistent. This is how you can create a persistent scratch location for your ESXi host. Good luck and happy investigating.