Domain Failed State – LAB2PROD

Upgrading VMware Cloud Foundation (VCF) 4.5.1 to VCF 5 results in a host failure or host error state, which effectively roadblocks completion of the upgrade, during the host upgrade phase. If you are concerned that you may be facing this issue, refer to the image below, if you see this message then continue reading.

The environment was in a healthy state, however, once the upgrade phase entered host remediation during the upgrade, a race condition occurred, where the host was upgraded, but not before the SDDC Manager workflow task timeout. As far as SDDC Manager was concerned, the host was now in an errored state.

The video below is a demonstration of the upgrade process.

If you are certain that the host has not errored in anyway and you have hit this race condition, follow the steps below.

The Fix?

Notice, in the below image, the host is reporting the correct version (8.0.1).

1. SSH onto SDDC Manager, as the vcf account.
2. Issue the command in the code block below as the VCF user, replace <hostID> with the ID of your host. You can retrieve the host ID in the host tab, simply, select the host and it will be visible in the URL.

When upgrading a VMware Cloud Foundation Domain, the domain that is being upraded may move into an errored state. This could be for multiple reasons, however, in this case it was due to a host moving into an errored state. As long as the host is not in any actual errored state, a simple curl command can clear this issue and allow you to complete the upgrade.

Notice that host1.region2.shank.com has a configuration status of Error. The error message you saw earlier is a result of this host being in an errored state. One thing to keep in mind is, when starting the upgrade, you are required to complete and pass the environment precheck, so you may be asking yourself how the upgrade proceeded.