I’m in the middle of a Network Segmentation project at the moment, which involves splitting a previously flat network into various subnets/vlans for security and management reasons. End of the day it involves changing IP addresses on servers.
I recently completed the segmentation of the VMWare environment, which includes around 50 hosts and a VirtualCentre server. I ran into some issues which took a bloody long time to fix, so I thought I’d share my experiences.
First off you will need the following:
- Out of band management (such as iLO or Console Access) to your ESX hosts. SSH is NOT out of band management. You WILL lose IP connectivity to your host during these changes.
- root logins for your esx hosts.
- Knowledge of your network, specifically what VLAN your new ESX host Service Console is going to be on and what IPs/Gateways to use. For this guide I am assuming that the Service Console will be on a tagged VLAN presented to the ESX host. If you don’t know what that means then ask your network person.
- An outage window for the VirtualCentre move. You will have to disconnect and reconnect ALL your ESX hosts after this change, which will mean they won’t be able to be managed for a while.
- If you do not have licenses for VMotion this is going to be a lot harder and more disruptive. For the purposes of this guide I will assume you DO have VMotion available.
Before we start, a word about name resolution. VMWare is very reliant of DNS for operation of its various services. It is necessary for you to know how your name resolution works. In my case, each host used a HOSTS file (/etc/hosts) to resolve names. You may use this method, or a DNS server. Find this out now.
Also before we start, disable HA on your clusters if you can, this will stop any migrations while you are trying to disconnect/reconnect hosts. You can do this by right clicking your cluster and clicking Edit Settings, then removing the checkbox from VMWare HA.
Section 1: Change VirtualCentre IP
- Log on to Server hosting VirtualCentre. Stop VirtualCentre service.
- Change IP Address/Gateway
- Change port VLAN assignment if required
- Reconnect to Server.
- Run ipconfig /flushdns, ipconfig /registerdns and nbtstat -RR
- Start VirtualCentre Service.
- Test connectivity from your VirtualCentre server to your hosts (ping).
- Connect to VirtualCentre using Virtual Infrastructure client.
- Confirm it loads cluster information successfully (lists Clusters/Hosts etc, disregard their status for now, if they show as offline that’s OK. Don’t worry your VMs are still running).
At this point you will need to update either your /etc/hosts file on each server or make sure that your dns server returns the correct address when you lookup the name of your VirtualCentre server. If not, update the A record.
Your hosts will probably all be showing up as offline at this stage. Don’t worry. We’ll soon fix that.
- In the Virtual Infrastructure client, right click the first host in your cluster and select disconnect. Wait for this to complete, then right click again and select connect. Once this is complete the host will show up online and be able to be managed again.
- Repeat for each host in the Cluster. Again, this will NOT pause or stop any VMs running on the affected host.
Once you’ve disconnected and reconnected all your hosts your VI Client should look all happy campers again, with all your hosts online. If you are not going to change the ESX Host IPs you’re done. Re-enable HA and test VMotion, Cloning etc.
Section 2: Change ESX Host IP
- Open your VI Client and connect to your VirtualCentre Server.
- Right click on your first host and select “Enter Maintenance Mode”. At this point all the running VMs should be VMotioned off to another server automatically. If this doesn’t occur you will need to manually migrate them by clicking each VM and selecting Migrate, then selecting a destination. Don’t move them all to one host or you will probably hit major performance issues.
- Once the host is in Maintenance Mode, right click it and select Remove. Once this is complete the ESX host should no longer be in the console.
- Connect to the Host using iLO or a Keyboard and Mouse.
- Push Alt-F1 to get a console, you should be presented with a username prompt.
- Login as root.
- Run the following command, where XX is the VLAN that your new Service Console IP address is on: ‘esxcfg-vswitch -p “Service Console” -v XX vSwitch0′. This will change the VLAN that your Service Console listens on.
- Run the following command, where X.X.X.X is the new IP address of your Service Console: ‘esxcfg-vswif -i X.X.X.X vswif0′. This will change the IP address of the Service Console.
- Add a new default route using the following command, where X.X.X.X is the default gateway for your new service console: ‘route add default gw X.X.X.X’.
- Test connectivity to the VirtualCentre server using ping.
- You can now log out of the console using Ctrl-D and close your iLO.
- At this stage you need to update any DNS/Hosts files to make sure the Host’s name resolves correctly to the new IP address (especially on the VirtualCentre server).
- Go back to your VI client, right-click your Cluster and select “Add Host”.
- Put in the hostname, username and password for the host you just changed the IP address on.
- The host should be discovered and added to the cluster, and should still be in maintenance mode.
- Click on the host, and in the right pane select Configuration then Networking. Click Properties on vSwitch0.
- Select your Service Console and click Edit. When prompted select “Continue modifying this connection”
- VLAN ID, IP address and Subnet mask should all be populated, but Default Gateway will be blank, Click Edit.
- Fill out the Default Gateway field under Service Console, then click OK and exit the configuration.
- Right click your host and select “Exit Maintenance Mode”. Once this is complete your host is ready to go.
- Repeat this process for all your other hosts, making sure to update either your DNS server or hosts file each time you change an IP.
- Once all your hosts are done, you will need to test that your Cluster is still functioning.
If you do not remove and re-add the host, the VI client will still show it and it will appear healthy. However some automated operations, such as Cloning or VMotion will fail with cryptic, unhelpful messages. Removing and re-adding the hosts is the ONLY way to successfully resolve this.
So there it is. In my case I didn’t discover the need to remove and re-add hosts until after I had made all my changes (only disconnecting and reconnecting rather than removing). This meant that I had to do all my changes 2 or 3 times.
Save yourself some time and follow the process.