August 27, 2021

VCF SDDC Manager Bundle Checksum Issue

LcmException: bundle checksum does NOT match the manifest value

The Issue? SDDC Manager Bundle Downloads Fail with the above issue

When attempting to upgrade from VCF 4.2.1 to VCF 4.3, all the bundles required for the download were set to download. However, when checking later, 2 were successful and the rest had failed. When looking in Bundle Management in SDDC Manager, error message in the image below presented itself.

LcmException: bundle checksum does NOT match the manifest value

To properly clean up the SDDC Manager database and repo, the bundles and their associations must be removed completely.

Troubleshooting The Issue

For more VCF Troubleshooting tips check out my Unofficial VCF Troubleshooting Guide

  1. SSH onto SDDC Manager, su to root and restart the lcm service systemctl restart lcm. The service can take up to 15 minutes to come back and initialize the depot. This did not resolve the issue for me.
  2. Reboot SDDC Manager. This also did not resolve the problem for me.
  3. After looking in /nfs/vmware/vcf/nfs-mount/bundle, I noticed the failed bundles had folders created with some content. At this point, it would seem the bundles need to be cleaned up and re-downloaded. There is a utility on SDDC Manager that can assist administrators in achieving this.

The Fix

Whilst the underlying issue is not known, the resolution is straightforward and quick. If you run into this issue, and this does not resolve it, leave a comment and we’ll see what we can do!

  1. SSH onto SDDC Manager and su to root.
  2. Navigate to the directory with the tool cd /opt/vmware/vcf/lcm/lcm-app/bin
  3. Identify bundle ID’s for the failed bundles in SDDC Manager, this can be done by navigating to Bundle Management and clicking more details on one of the failed bundles.
  4. For each failed bundle, issue the following command followed by the bundle ID, ./bundle_cleanup.py <bundleID>. The command is shown in the image below, and the output from the command is also displayed.
Command being run with bundle ID
The easiest options to troubleshoot this issue are.

Bundle download available again

Conclusion

LOG FILE : /var/log/vmware/vcf/lcm/bundle_cleanup.log
-----------------------------------------------------
2021-08-27 04:13:48,922 [INFO] root: Performing cleanup for bundle with IDs : ['ed86b356-e99c-45c7-add8-92791cb28757']
2021-08-27 04:13:48,922 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from upgrade where upgrade_status in ('INPROGRESS','CANCELLING');"
2021-08-27 04:13:48,949 [INFO] root: b' 0n'
2021-08-27 04:13:48,950 [INFO] root: b'n'
2021-08-27 04:13:48,950 [INFO] root: RC: 0
2021-08-27 04:13:48,950 [INFO] root: Out: 0

2021-08-27 04:13:48,950 [INFO] root: Stopping LCM service.
2021-08-27 04:13:48,951 [INFO] root: Execute cmd: systemctl stop lcm
2021-08-27 04:13:49,087 [INFO] root: RC: 0
2021-08-27 04:13:49,087 [INFO] root: Out:
2021-08-27 04:13:49,088 [INFO] root: Removing LCM NFS mount.
2021-08-27 04:13:49,088 [INFO] root: Execute cmd: rm -rf /nfs/vmware/vcf/nfs-mount/bundle/ed86b356-e99c-45c7-add8-92791cb28757
2021-08-27 04:13:49,127 [INFO] root: RC: 0
2021-08-27 04:13:49,128 [INFO] root: Out:
2021-08-27 04:13:49,128 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upload_id from bundle_upload where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,145 [INFO] root: b'n'
2021-08-27 04:13:49,146 [INFO] root: RC: 0
2021-08-27 04:13:49,146 [INFO] root: Out:

2021-08-27 04:13:49,146 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in bundle upload table
2021-08-27 04:13:49,146 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select download_id from bundledownload_by_id where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,163 [INFO] root: b' fb61d3c8-c791-4ace-9f09-18d57e3ba48cn'
2021-08-27 04:13:49,164 [INFO] root: b'n'
2021-08-27 04:13:49,164 [INFO] root: RC: 0
2021-08-27 04:13:49,164 [INFO] root: Out: fb61d3c8-c791-4ace-9f09-18d57e3ba48c

2021-08-27 04:13:49,164 [INFO] root: Execute cmd: curl -s -X DELETE localhost/tasks/registrations/fb61d3c8-c791-4ace-9f09-18d57e3ba48c
2021-08-27 04:13:49,211 [INFO] root: RC: 0
2021-08-27 04:13:49,211 [INFO] root: Out:
2021-08-27 04:13:49,211 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upgrade_id from upgrade where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,230 [INFO] root: b'n'
2021-08-27 04:13:49,230 [INFO] root: RC: 0
2021-08-27 04:13:49,230 [INFO] root: Out:

2021-08-27 04:13:49,231 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in upgrade table
2021-08-27 04:13:49,231 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upgrade_id from upgrade where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,249 [INFO] root: b'n'
2021-08-27 04:13:49,249 [INFO] root: RC: 0
2021-08-27 04:13:49,249 [INFO] root: Out:

2021-08-27 04:13:49,249 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in upgrade table
2021-08-27 04:13:49,250 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from bundle where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,267 [INFO] root: b' 1n'
2021-08-27 04:13:49,268 [INFO] root: b'n'
2021-08-27 04:13:49,268 [INFO] root: RC: 0
2021-08-27 04:13:49,268 [INFO] root: Out: 1

2021-08-27 04:13:49,268 [INFO] root: Deleting bundle & upgrade info for bundle ID : ed86b356-e99c-45c7-add8-92791cb28757
2021-08-27 04:13:49,268 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from bundle where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,290 [INFO] root: b'DELETE 1n'
2021-08-27 04:13:49,290 [INFO] root: RC: 0
2021-08-27 04:13:49,290 [INFO] root: Out: DELETE 1

2021-08-27 04:13:49,290 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from image where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,311 [INFO] root: b' 1n'
2021-08-27 04:13:49,311 [INFO] root: b'n'
2021-08-27 04:13:49,311 [INFO] root: RC: 0
2021-08-27 04:13:49,311 [INFO] root: Out: 1

2021-08-27 04:13:49,311 [INFO] root: Deleting bundle ed86b356-e99c-45c7-add8-92791cb28757 in image table
2021-08-27 04:13:49,311 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from image where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,330 [INFO] root: b'DELETE 1n'
2021-08-27 04:13:49,331 [INFO] root: RC: 0
2021-08-27 04:13:49,331 [INFO] root: Out: DELETE 1

2021-08-27 04:13:49,331 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from partner_bundle_metadata where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,350 [INFO] root: b' 0n'
2021-08-27 04:13:49,350 [INFO] root: b'n'
2021-08-27 04:13:49,350 [INFO] root: RC: 0
2021-08-27 04:13:49,350 [INFO] root: Out: 0

2021-08-27 04:13:49,351 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in partner_bundle_metadata table
2021-08-27 04:13:49,351 [INFO] root: Starting LCM service.
2021-08-27 04:13:49,351 [INFO] root: Execute cmd: systemctl start lcm
2021-08-27 04:13:49,466 [INFO] root: RC: 0
2021-08-27 04:13:49,466 [INFO] root: Out:

All my other VCF related articles can be found here

When this occurs, the bundle cannot be retried and all options are greyed out. This may not occur all too often, but without knowing what to do, administrators are generally stuck.

Similar Posts