When SSO goes bye bye, things get interesting

January 29th, 2013 Posted in Virtualization
5 Comments »

So until today I was under the impression that SSO only effects the web-client in 5.1. The way I understood was the vSphere client still behaves the way it did before and SSO is not engaged unless the web-client is used to login. This also brought me to the conclusion that if SSO goes down, one cannot login via the web-client but the vSphere client can still be used. Wrong!!

A colleague of mine pointed me to a this page that clearly states the following:

How does SSO integrate with the vSphere Client?

SSO does not integrate with the vSphere Client. However, when you log in through the vSphere Client, vCenter Server sends the authentication request to SSO.

Once I read that I started doubting my thought process and the importance of SSO in 5.1 Apparently all access to vCenter must be down once SSO is down (both via web and vSphere client).

After doing a lot testing this is what I found (vcenter 5.1 build 799731). When SSO is down,

  • access via web-client is down as expected 
  • access via vSphere client is flakey

What does flakey access mean? Well, I got mixed results and was finally able to see some pattern. When SSO service is down, I was able to login with the account that has had a successful login while SSO was running. The important thing here was, “use windows session credentials” had to be checked for which I had to be logged in with the account that had successfully logged in when SSO was up. If I didnt check the box and entered the credentials myself, it told me the username and pwd were incorrect. I know I can fat finger keys at times but I tested this over and over to come to this conclusion. It wasnt me. Access was only allowed when the check box was checked.

This also meant any new account that was created or granted access couldn’t login using the vSphere client. Rememeber we only had luck with accounts that were able to login succssfuly prior to SSO service going down. And that too required the checkbox to be checked. If the account was just created or granted access after SSO went down, the screen showed the beautiful message on the right. The same message was received if the account didn’t successfully login while SSO was up. Why cant this message say the SSO cannot be reached is beyond me. By the way the web-client will tell you “Failed to communicate with the vCenter Single Sign On server” when SSO is down. So thank you VMware for doing that. 

Another thing to keep in mind. When SSO service is down, your vCenter service continues to run. However, if you attempt to restart your vCenter service you will find yourself in trouble. I was unable to get the vCenter service to start with SSO being offline. Which makes SSO even more important. Yes even with vCenter down your VMs continue to work but there are other vCenter specific features that will not function like DRS, sDRS for example. And if this vCenter is connected to a vCloud instance thats another can of worms.

So the bottom line is, SSO is very very important. It has two parts, the application and the DB part. VMware has done a great job in giving the option to install SSO as single, clustered or even multi-site type deployments. The high availability in the application side is thought out there. However, the problem is DB. VMware does not fully support SSO DB on a SQL cluster. As a matter of fact, there have been known issues that have come about when trying to deploy SSO using a SQL cluster. So the real option with full support is a stand alone SQL node. But that also creates a single point of failure. When the DB goes down, you are unable to login using the web-client, you maybe able to login using the vSphere client and all other things we discussed above.

So building redundancy is extremely important. VMware’s recommended solution is to use vCenter Heartbeat. We all know that can be a pricey solution. However, if full support along with redundancy is importnat to you, that is the way to go. I hope VMware extends their full support to at least allow running DB on a SQL cluster for all their products including vCenter (which is still a grey area). That would be the right thing to do. Heartbeat provides added functionality and there will always be a market for that as well. I hope full support on DB residing on SQL clusters is not further delayed in the interest of the vCenter Heartbeat product.

In the end I will borrow Tom Petty’s words to tell VMware “Don’t do me like that”…

, ,
Read More

Multiple domains, vCenter and SSO

January 16th, 2013 Posted in Virtualization
No Comments

Not having blogged for sometime due to everything else that I have been involved in lately, I figured the new years will help me make time for myself. Well, its been 16 days since the new years so I guess a late happy new year to you. I have been playing around with some of the new features in 5.1. Recently somebody asked me about ways to handle access to a vCenter via different domains. Few months ago I would have pointed them to an AD guy and said build some kind of a trust.

A lot has changed in a few months obviously. SSO has come around, and more importantly it works (finally :D ). I know low blow, oh well. Using SSO one can add multiple identity sources (AD, Open LDAP etc). You can have multiple AD identity sources added to the same SSO. So what does that mean?

When you install SSO on a windows server, the domain this server is on automatically gets added as an identity source in SSO. You can then pick users and group from here and assign them privileges. In addition to that, you can also add additional domains to the identity sources and assign entities in this domain access to vCenter as well. And its so simple that even I can do it. But I figured I will still write up a few steps and summarize the process as I can very easily forget what I did to make this work.

  1. In order to add users/groups from a domain, the domain needs to be added to the SSO identity sources
  2. The domain the SSO server is on gets added to the identity sources (in my case, the SSO server is on a domain called cloud)
  3. Login to SSO with Admin permission (default user admin@system-domain works but you can use a different account with same privileges as well)
  4. Click on Administration
  5. Click on “Configuration” under “Sign-on and Discovery”

  6. Click the plus sign on the top left 


  7. Add the appropriate info (In my case, I am adding a domain called “sunny.bilalhashmi.com” with an alias “SUNNY”) – I am using IP for the server URL as I dont have DNS resolution between the SSO server and the SUNNY domain. Ideally you would want to use the DNS name.



  8. Once you have entered the info, test the connection and hit OK, you should be able to add users/groups from the new domain (SUNNY domain in my case)
  9. When adding new users/groups, click on the drop down and you should be able to see the new domain. Select the new domain add the new permissions and you should be all set. Why do I have “Cloud” along with other things in the list? As I mentioned in Step 2, “Cloud” is the domain my SSO server is on, so it gets added automatically. 



  10. Again, no trust requirements between the domain. This is all happening due to our new friend SSO.
Obviously you will have to make sure connectivity between the the new domain and the SSO etc is working. I am unsure about the exact permissions that is needed in AD for making the connection to SSO. Remember in step 7, we add the username and pwd for the account in the domain to make the connection. I used the Administrator account that had all kinds of access. I believe an account that has access to query the accounts should be sufficient, but I havent tested that aspect. 


Why do you have folks from multiple domains coming into the same vCenter? Good question. One use case would be a customer who is supported by another company for their VMware infrastructure. In this case, both the customer and the supporting team can use their own accounts to carry on business. I personally hate having multiple accounts. Thats a recipe for being the account  lockout champion.
, , ,
Read More

Restricting Domain Admins from vCenter

February 21st, 2012 Posted in Virtualization
5 Comments »

This morning I was talking to a friend who asked me how can he restrict the domain admins in his domain from having Admin access in vCenter. This is a very typical concern usually in bigger environments where roles are distributed. This might be common knowledge for some of us but for those who don’t know this yet, read on.

When you setup vCenter on windows (notice I said windows because we also have vCSA now), by default the local administrators group is given Admin access in vCenter. You can see this in the “Permissions” tab in vCenter.

Obviously this also means that the “Domain Admins” also have Administrator access in vCenter. Why? Because Domain Admins are also part of the local administrators group. So what can you do?

  1. Create a security group in AD and call it something that makes sense
  2. Add the appropriate parties to this group (at least add yourself)
  3. Log into the vCenter server as an Administrator and create a local group on this server with an appropriate name
  4. Add the group we created in step 1 to the local group we just created
  5. Also add the local Administrator account to this local group (why?, in case your AD goes kaboooom!!)
  6. Log into the vCenter using the vi client
  7. Go to the very top of the tree and click on the permissions tab
  8. Add the local group we just created
    • Right click on the empty space
    • Click “Add Permissions”
    • Click “Add” on the next screen
    • Make sure that you have selected “server” from the drop down menu in the domain section (not your domain)

    • Find the local group we just created and highlight it. Click “Add”
    • Click “Ok”
    • On the next screen, make sure you select “Administrator” from the drop down and ensure that “Propagate to Child Objects” is checked

    • Hit “Ok”

Now all users that are part of the local group (step 3) and the AD group (step 1), should have admin access to the vCenter.

But wait, your domain admins still have access. Right click on the “Administrators” group under the permissions tab and click “Delete”.

But before you do this last step, please please please make sure that access from the other groups we created is working as expected and you do have Admin rights on the vCenter via that group, create a test account or something but make sure that works. Once you have confirmed that, remove the Administrators group and you should be all set. Now your Domain Admins are still administrators on the vCenter server but they are not vCenter Admins, of course they can change this if they want to, but hey we can only do so much. It may not be a bad idea to give your Domain Admin’s read only rights, this usually keeps them from getting annoyed and who knows they may never realize that they only have read only access.

Again, make sure your access works before you remove the Administrators group from vCenter, or else you may have an extremely secure environment where no one is an Admin :D

, ,
Read More

vSphere Web Client and OS X

February 7th, 2012 Posted in Virtualization
5 Comments »

So I finally decided to install the Web Client server in my lab. Everything was pretty straight forward as far as installation goes. Ones you get through the wizard, you will basically have to register the Web Client with the vCenter server. This happens on the server where the web client server is installed via the browser. You will go to https://web-client-server:9443/admin-app/ where server is the name/IP of the web client server. Keep in mind this requires flash to be installed on the server. I am not a big fan of installing flash unless absolutely needed, so here is the workaround thats mentioned in this kb article.

If you don’t want to install flash on the web-client server, simply type the following command on your web-client server:

admin-cmd.bat register https://web-client-server:9443/vsphere-client https://vCenter-server username password

Be sure to replace the portions in blue above with your info (web-client server, vCenter server, username, password).

Once the registration is complete, you can simply head over to https://web-client-server:9443/vsphere-client and login. However, be warned, though this is a great alternative to the vSphere client, it does not have all the functionalities and even less if you are an unfortunate Mac user.

 At the bottom left of the screen, you will notice where it says “Download Client Integration Plug-in”, and if you are a Mac user you will be disappointed. Why? Because this will not install, you will be prompted with with a message as displayed below:

 

So whats the big deal?

To start off, client integration does a couple of important thing that I am aware of:

  1. Allows you to access a virtual machine’s console in the vSphere Web Client
  2. Allows you to connect virtual devices that reside on a client computer to a virtual machine (think USB devices on a client machine)

So if your an OS X user, you will not be able to perform those tasks using your web-client. Will this kill you? Probably not but it’s disappointing to see that OS X has been left out for now. I have heard that support for OS X is in the roadmap for the future. When exactly? No clue. I guess after crying for years, VMware listened to the Linux users and released the web-client and I think its only a matter of time before the Mac world starts to see all the features.

Don’t get me wrong. The web-client doesn’t cover all the features a traditional vSphere client has but it’s ver 1 and I am sure it will get there soon. If you are a Mac user, keep in mind the functionality of the web-client on OS X is even more limited.Will I still use it? Absolutely, this will replace the traditional client we have known for years one day.

 

, , , ,
Read More

vCenter service fails to start

January 12th, 2012 Posted in Virtualization
No Comments

Ever had that happen? The first instinct after a connection failure is to ping the vCenter box followed by seeing if the vCenter service is running. If the service is stopped, you would perhaps try to restart it (a good admin would probably check  the logs to see why it failed) and what if it fails to start? Next thing you know you are trying to verify that everything on the DB world is happy and ones that is also confirmed, you come back to the logs which is what should have been checking to begin with.

The vpxd logs would definitely come in handy this time, these are basically your vCenter logs. They are located on your vCenter server (vCenter 4.x) @ %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs (In Windows 2008, the vpxd.log file is located at C:\ProgramData\VMware\VMware VirtualCenter\Logs)

Based on what you find here would really help out in finding the root cause. I was seeing  the following in the log file:

[VpxdReverseProxy] Failed to create http proxy: An attempt was made to access a socket in a way forbidden by its access permissions.
[Vpxd::ServerApp::Init] Init failed: VpxdMoReverseProxy::Init()
Failed to intialize VMware VirtualCenter. Shutting down…
Forcing shutdown of VMware VirtualCenter now

Based on what I was seeing in my case, it could have been a bunch of things that could have been wrong according to this kb article. Obviously the thing that jumps out is that perhaps the vCenter ports are already being used somewhere. vCenter needs port 80, 443 and 902 at least unless these were changed to something else during install.

What ports are engaged:

One quick way to find out if these ports are already in use is by using the netstat command.

  • On you vCenter server go to run and type cmd
  • At the command prompt type “netstat -ao” and hit Enter. You should now see a big list of all engaged ports along with their PID
  • If you see any of the ports mentioned above (80, 443, 902) already being used while your vCenter is dead, its possible that is whats stopping vCenter from running
  • Note the PID of the port in question
  • Open Task Manager and click view –>Select columns
  • Check “PID (process identifier) ” and press ok.
  • Find the PID you noted above and that should point you in the direction of the application that has been putting your vCenter to sleep

Keep it simple:

In my case it was the infamous love between IIS and port 80, killing that fixed the deal. Why is IIS running on this server? Good question, I don’t know the answer. Ideally it should have no business there.

It’s important to note that by simply restarting the server its possible that your vCenter would have come back to life as long as the other application doesn’t engage that port first. However, you will find yourself in a vicious cycle of rebooting every time it goes down. Why do that when you can find the root cause and fix it. Keep your vCenter server clean and simple, it does a lot of complex task with simple clicks.

, , ,
Read More

Duplicate MACs in vCenter

May 2nd, 2011 Posted in Virtualization
No Comments

I don’t have a lot of experience with Hyper-V, but I have worked with people who have. After hearing their horror stories, I don’t envy acquiring that sort of experience. Speaking of horror stories, my favorite one is, when one of my co-workers told me about a Hyper-V environment they had setup which was generating duplicate MAC addresses. I was amused but in the back of my mind I started thinking if this was possible in a vSphere setup. Yes it is.

UID

vCenter assigns MAC addresses using the unique ID that’s assigned to it under Administration > vCenter Server Settings > Runtime Settings

This unique ID can be set to a value between 0-63. If you have two or more vCenter’s running the same instance ID, its only a matter of time before you start seeing mac conflicts in your environments.

vCenter assigns MAC addresses using a simple formula (00:50:56:80HEX+UID:00:00). So if your vCenter ID is 45, your VMs mac should be 00:50:56:ad:XX;XX. As you can tell the fourth byte is what can help you identify which vCenter was used to create the VM. The fifth and sixth bytes are the ones that are usually edited if you have a need for assigning a MAC instead of vCenter doing it for you.

Another interesting thing I noticed, I saw 3 different types of MAC addresses in my VMs. I saw,

VM1 00:50:56:ad:c2:3F

VM2 00:0C:29:73:B1:2F

VM3 00:5056:a5:d2:6F

It turns out, that VM1 was created on my new vCenter with a UID 45 (ad=80HEX + 45), VM2 was created directly on a ESXi host and VM3 was created on a different vCenter with a UID of 37(a5=80HEX + 37).

Though these little things don’t matter as much, but its important to know how all this comes together. It will be very helpful when you find yourself in a situation where your VMs have identical MACs. Someone forgot to set a unique ID for their vCenter.

 

, , , , ,
Read More