Scenario
The important decision is not whether the team wants to continue.
It should control the upgrade.
Before the runbook enters the execution window, the team should complete the following checks.
Break it into explicit work packages:
The upgrade is not finished when the first window closes.
Confirm business impact is acceptable:
The upgrade path touches VCF Operations, SDDC Manager, VCF Management Services, licensing, NSX, vCenter, ESXi, NSX Edge, workload domains, and Day-N operational services.
Before core components move forward, the runbook should verify:
If these answers show up in the planning meeting, the runbook is not ready.
Why This Matters Operationally
“Do we have enough evidence to proceed to the next phase?”
Do not rely on memory for version-sensitive claims. VCF 9.1 sequencing, Aria Operations patch-level behavior, licensing, management services, and supported upgrade paths can change.
The exact sequence must follow the supported Broadcom guidance for the environment’s starting version. The runbook should not flatten this into “upgrade the core stack.”
- A phase can be technically complete but operationally incomplete.
- A component can upgrade successfully while observability, licensing, identity, or application acceptance remains unresolved.
- A failed phase may not have a clean “undo” path unless the fallback decision was defined before the work began.
Every gate needs an owner, evidence, and a stop/go decision.
Symptoms of a Weak VCF 9.1 Upgrade Runbook
Each gate answers a simple question:
The runbook should also define stop conditions.
| Symptom | Why It Is a Problem |
|---|---|
| “We will validate after the upgrade.” | Validation must be designed before execution, not improvised after change. |
| “Backups are handled by the backup team.” | The upgrade team still needs backup evidence and recovery ownership. |
| “Rollback is restore from backup.” | That may be true for one component and wrong for another. |
| “Monitoring will be checked at the end.” | VCF Operations is part of the sequencing model, not a final dashboard check. |
| “The platform team owns it.” | Platform teams often do not own DNS, IPAM, PKI, identity, network policy, or application testing. |
| “Workload domains can be done later.” | Later still needs an owner, sequence, acceptance model, and Day-N backlog. |
The team cannot hide missing DNS records inside a task list. It cannot bury unclear rollback language under a generic change record. It cannot treat application testing as a courtesy check after the platform team leaves.
The Runbook Model: Gates Before Tasks
That creates three practical risks:
The inventory stage answers the question:
A version report is useful. A validated dependency map is better.
A component upgrade is not complete until its control plane, data plane, and operational visibility are validated.
This is the stage where many upgrade windows get derailed.
A real runbook is what gets the platform back into a supportable operating state.
| Component Area | What to Capture |
|---|---|
| VCF baseline | Current VCF version, SDDC Manager version, workload domain list. |
| vSphere | vCenter versions, ESXi versions, cluster lifecycle model, vSAN state, distributed switch versions. |
| NSX | NSX Manager version, transport nodes, TEP health, edge clusters, routing, federation if applicable. |
| Operations | Aria Operations version, VCF Operations state, policies, dashboards, collectors, adapters, alerts. |
| Management services | Existing fleet management state, identity services, licensing services, log management services. |
| External dependencies | DNS, IPAM, NTP, PKI, identity provider, backup platform, syslog, monitoring integrations. |
| Workload dependencies | Critical workloads, application owners, smoke tests, maintenance constraints. |
Examples:
This inventory should be reviewed by the owners, not only exported by the platform team.
Runbook Stage 2: VCF Operations Readiness
This stage needs more than SDDC Manager prechecks.
They mention backups, but not restore confidence.
- whether Aria Operations is already deployed,
- the exact Aria Operations version and patch level,
- whether the environment must first move to Aria Operations 8.18,
- whether the exact 8.18.x patch level has known upgrade-path caveats,
- whether VCF Operations 9.1 will be upgraded in place or deployed,
- whether Cloud Proxy placement is required,
- whether collectors and integrations are healthy,
- whether dashboards, alerts, policies, and custom content are documented,
- whether post-upgrade observability acceptance criteria are defined.
That record should not simply say “monitoring healthy.”
It should validate:
Define fallback by phase.
Runbook Stage 3: SDDC Manager and Management Services Readiness
Conclusion: A Good Runbook Controls the Upgrade, Not Just the Tasks
The Day-N handoff should include:
| Dependency | Readiness Check |
|---|---|
| SDDC Manager | Backup evidence, precheck output, upgrade bundle availability, administrative access, certificate/password health. |
| VCF Management Services | IP range plan, management network reachability, DNS requirements, internal network overlap review. |
| License services | License server deployment plan, A/PTR records, connectivity, post-upgrade license validation. |
| DNS/IPAM | Reserved addresses, FQDNs, reverse lookup, change timing, owner availability. |
| Security | Firewall paths, certificate trust, privileged access, audit requirements. |
| Support | Support entitlement, case process, log bundle procedure, escalation owner. |
The following items should be explicit in the runbook, not left to memory.
After VCF Operations readiness is complete, the runbook moves into SDDC Manager and VCF Management Services.
Runbook Stage 4: Core Platform Upgrade
They describe rollback, but not what rollback means for each phase.
The rollback plan should not pretend the whole platform has one undo button.
This is the difference between technical completion and operational acceptance.
| Work Package | Required Validation |
|---|---|
| NSX upgrade | NSX backup, manager health, edge health, transport node health, routing validation, federation checks if applicable. |
| vCenter upgrade | Backup evidence, temporary IP requirements, RDU or in-place decision, service health, plugin compatibility. |
| ESXi remediation | Cluster lifecycle model, image readiness, evacuation capacity, vSAN health, host maintenance behavior. |
| NSX Edge finalization | Edge version alignment, routing checks, north/south connectivity, service impact validation. |
| Management-domain acceptance | SDDC Manager health, VCF Operations visibility, license state, monitoring state, backup resumption. |
A VCF 9.1 upgrade runbook needs to prove more than task order. It needs to prove that the organization can execute the upgrade, validate the result, stop safely when needed, and hand the platform back to operations without ambiguity.
“What are we actually upgrading?”
Runbook Stage 5: Validation
It should say what was checked, who checked it, what changed, what remains at risk, and who accepts the operational state.
Common warning signs include:
Platform Validation
VCF Operations is the first major readiness gate.
- SDDC Manager reports expected versions and health.
- VCF Operations is reachable and collecting expected data.
- VCF Management Services are healthy.
- NSX managers are healthy.
- NSX Edge nodes are healthy.
- vCenter services are running.
- ESXi hosts are connected and remediated.
- vSAN and storage health are clean.
- Backup jobs resume successfully.
- License assignment is valid.
Operations Validation
The point is to avoid discovering a blocking dependency while the environment is already in motion.
- Alerts are firing where expected.
- Dashboards and policies are intact or intentionally replaced.
- Logs are flowing to the expected destination.
- Admin access works through the expected identity path.
- Support bundle generation is understood.
- Certificate and password workflows are documented.
- Lifecycle views show the expected state.
- Service desk routing is ready for post-upgrade noise.
Workload Validation
The first stage is source validation.
- Representative VMs remain available.
- Critical applications pass smoke tests.
- North/south connectivity works.
- East/west connectivity works.
- Backup and restore confidence checks complete where applicable.
- Monitoring accurately reflects workload state.
- Application owners provide acceptance.
Confirm the infrastructure state:
Runbook Stage 6: Rollback and Fallback
VCF 9.1 deserves a runbook that treats testing, fallback, and readiness as first-class workstreams.
A VCF 9.1 runbook should be organized around gates.
A patch-window checklist may get the team into the upgrade.
VCF 9.1 makes this especially important because VCF Operations, management services, licensing, NSX, vCenter, ESXi, and Day-N workload-domain planning are all connected.
| Phase | Fallback Question |
|---|---|
| Before VCF Operations upgrade | Can we restore Aria Operations state, content, and integrations if the upgrade fails? |
| Before SDDC Manager upgrade | Do we have a valid SDDC Manager backup and documented recovery procedure? |
| Before VCF Management Services deployment | Can DNS, IPAM, firewall, or deployment changes be backed out cleanly if deployment fails? |
| Before NSX upgrade | Do we have NSX backups, edge validation, routing state, and a supported recovery plan? |
| Before vCenter upgrade | Do we have vCenter backup evidence, temporary IP planning, and a supported RDU or upgrade recovery path? |
| Before ESXi remediation | Can workloads evacuate safely, and are cluster capacity/admission constraints understood? |
| Before workload-domain upgrades | Are application owners available for validation and incident response? |
A weak runbook usually sounds confident until you ask who owns the evidence.
Confirm the platform can still be operated:
- VCF Operations upgrade fails and observability cannot be trusted.
- SDDC Manager precheck fails with unresolved blockers.
- VCF Management Services deployment cannot complete due to DNS/IP/network issues.
- NSX manager health is degraded before upgrade.
- NSX Edge routing validation fails.
- vCenter upgrade fails or services do not stabilize.
- ESXi remediation creates cluster capacity or vSAN health risk.
- Application validation fails for a critical service.
The output of this stage should be an operations readiness record.
In some phases, fallback may mean restoring a specific appliance. In other phases, it may mean stopping the sequence, stabilizing the environment, collecting logs, opening a support case, and waiting for a supported recovery path.
Runbook Stage 7: Day-N Handoff
The runbook should make those risks visible before the window starts.
This is where the team confirms the runbook is based on current information, not stale assumptions.
| Handoff Item | Owner |
|---|---|
| Remaining workload-domain upgrades | VCF platform owner |
| VCF Operations cleanup | Operations / monitoring owner |
| License validation and cleanup | Licensing / platform owner |
| Identity transition tasks | Security / identity owner |
| Log management plan | Operations / logging owner |
| Documentation updates | Platform architecture owner |
| Application validation follow-up | Application owners |
| Known issues and support cases | Change owner |
| Next maintenance windows | Platform owner and change manager |
The key principle is simple:
VCF 9.1 Readiness Scorecard
The VCF 9.1 upgrade path has been selected. The planning tool has been reviewed. The target version is understood. The team has a maintenance window on the calendar.
