Lifecycle & Updates: The Offline Depot

Draft — VCF 9.1 lab series. New page for the 9.1 series. Builds on the Deployment (Bring-Up) page (the depot must exist before bring-up) and the Salt automation guide.

📘 Official documentation. Every process here is already documented by Broadcom and covered in public sources like William Lam’s blog. This article simply combines those sources into the workflow that supports my lab — it’s a walkthrough, not a replacement for the vendor docs. The recommendation is always to follow the official documentation: Broadcom’s Download Bundles to an Offline Depot (VCF 9.1). Where this guide and the official docs differ, the official docs win.

Why Offline

I rebuild this lab a lot. Keeping the binaries in a separate, standing depot means a fresh VCF Installer just points at one URL and pulls everything it needs — no re-downloading from Broadcom on every rebuild, and the lab can stay disconnected. (Pointing the installer at the depot is covered in the Deployment guide.)

The depot outlives the fleet: I populate it on demand, and each rebuild syncs against whatever’s there.

How the Depot Fits

VCF 9.0 introduced a single unified VCF Software Depot. In 9.1 that same depot is what every lifecycle consumer reads from. It can point at Broadcom’s online repository or — as here — an offline host you populate yourself and serve over HTTPS. One offline depot feeds both lifecycle planes: Fleet LCM (fleet-wide bundles) and instance / SDDC LCM (vCenter, NSX, ESX, and the UMDS-served ESX patch store, which in 9.1 is pre-wired to the Fleet LCM URL).

Terminology note. I used to think of this as “uploading to the fleet manager with a support token.” In 9.1 it’s the other way round: the fleet pulls from the depot URL, and the Broadcom-side credential is now a per-depot activation code (below), not a reusable token.

What the Depot Holds

There are four kinds of content in the depot, obtained two different ways. The first two are pulled with the VCF Download Tool (VCFDT); the last two are content libraries you publish by hand (VCFDT does not fetch them in 9.1):

#	Content	How it’s obtained	Lands under	Consumed by
1	Install & upgrade binaries	VCFDT binaries download	PROD/COMP/…	Installer / Fleet LCM
2	ESX metadata & binaries	VCFDT esx download	PROD/COMP/ESX_HOST/…	vLCM (cluster images)
3	Supervisor content library	Manual zip from the Broadcom portal	PROD/COMP/SUPERVISOR/	vCenter subscription → Supervisor
4	VKS releases (VKR) library	Mirror of a public VMware CDN	PROD/COMP/VKR/	vCenter subscription → VKS guest clusters

Everything is served from the same depot host (pgdepot, nginx config below) at https://vcfdepot.pgnet.io and lands under one active PROD tree.

Publishing to the Depot

This part is just my lab housekeeping, in short: download or mirror into a side staging directory, take a ZFS snapshot of the depot dataset as a rollback point, then rsync add-only (no --delete) into the live PROD tree — and for content libraries, re-check that the items.json manifest still matches what’s on disk (the gotcha below). The per-content commands follow; the snapshot/validate wrapping is the same each time.

Common paths:

• Depot root: /mnt/nas/vcfdepot/depot (NFS share, mounted on the bastion host pgext2)
• Active served tree: /mnt/nas/vcfdepot/depot/PROD (what nginx serves)
• Fleet base URL: https://vcfdepot.pgnet.io (HTTPS, basic auth)

---

1. Install & Upgrade Binaries (VCFDT)

VCFDT runs on the lab’s bastion host (pgext2) — it has the outbound access to reach Broadcom while the rest of the lab stays disconnected, and it mounts the NFS depot share. Install it once — this also covers section 2.

mkdir -p /opt/vmdt
tar -xvf vcf-download-tool-*.tar.gz -C /opt/vmdt --strip-components=1   # this lab: 9.1.0.0100
cd /opt/vmdt/bin

## generate a software depot ID, then redeem it for an activation code
## in the VCF business console (https://vcf.broadcom.com); save the code
./vcf-download-tool configuration generate --software-depot-id
echo "your-broadcom-activation-code" > /opt/vmdt/token

9.1 change — activation codes. A reusable download token no longer works against any depot. You register a software depot ID and redeem a per-depot activation code; rebuild the depot and you generate a fresh one.

Download the install and upgrade images into the vmdt staging dir, then publish:

cd /opt/vmdt/bin
STAGE=/mnt/nas/vcfdepot/depot/vmdt
SERVE=/mnt/nas/vcfdepot/depot/PROD
TOKEN=/opt/vmdt/token

## install images
./vcf-download-tool binaries download --sku=VCF --vcf-version=9.1 \
  --type=INSTALL --depot-store="$STAGE" --depot-download-activation-code-file="$TOKEN"

## upgrade images
./vcf-download-tool binaries download --sku=VCF --vcf-version=9.1 \
  --type=UPGRADE --depot-store="$STAGE" --depot-download-activation-code-file="$TOKEN"

## publish (snapshot first), add-only
rsync -a "$STAGE/PROD/" "$SERVE/" && chmod -R a+rX "$SERVE"

2. ESX Metadata & Binaries (VCFDT)

The ESX component repository feeds vLCM. Pull the whole thing into staging, then publish the same way:

cd /opt/vmdt/bin
STAGE=/mnt/nas/vcfdepot/depot/vmdt
SERVE=/mnt/nas/vcfdepot/depot/PROD
TOKEN=/opt/vmdt/token

./vcf-download-tool esx download --depot-store="$STAGE" \
  --depot-download-activation-code-file="$TOKEN"

rsync -a "$STAGE/PROD/" "$SERVE/" && chmod -R a+rX "$SERVE"

Pull the whole ESX directory. esx download fetches the vLCM compatibility data and the full set of ESX_HOST component depots (vmtools, iovp, addon, main — ~5500+ VIBs). This complete repository is what lets vLCM compose cluster images — image-based lifecycle may have issues without it, so I download it all to be sure. It’s also the slow part: ~1 hour in this lab. A successful run ends with “Depots downloaded successfully” and logs to /opt/vmdt/log/vdt.log.

3. Supervisor Content Library

The vSphere Supervisor images aren’t pulled by VCFDT — they ship as a content library you download and serve. From the Broadcom portal (VCF 9.1.0.0 → Primary Downloads → vSphere Supervisor), grab VMware-vSphere-Supervisor-9.1.0.0100-25417926.zip (~3.9 GB). The zip is already a vcsp v2 content-library layout — unzip and serve it as-is:

SUPERVISOR/
  lib.json                          # library manifest (relative hrefs)
  items.json                        # per-item file list + sizes + etags
  supervisor-9.1.0.0100-25417926/   # Photon OVA (ovf/vmdk/mf/cert) + item.json
  spherelet-v1.30/ … v1.32/         # spherelet depots + solution.json

Because lib.json/items.json use relative hrefs, serving the folder over HTTPS makes it a valid subscribed content library — no Fleet depot-service republish needed. Publish it with the standard rule:

## stage the zip, then publish to PROD/COMP/SUPERVISOR (snapshot + validate as above)
unzip -o VMware-vSphere-Supervisor-9.1.0.0100-*.zip -d /mnt/nas/vcfdepot/depot/_stage/supervisor
rsync -rltu /mnt/nas/vcfdepot/depot/_stage/supervisor/SUPERVISOR/ \
            /mnt/nas/vcfdepot/depot/PROD/COMP/SUPERVISOR/

Gotcha: manifest ↔ disk mismatch

A subscribed content library fails wholesale if any items.json href 404s. The lab hit this twice — once on Supervisor (a half-upgraded depot whose items.json still listed a removed pre-release build) and once on VKR (below). The fix is cheap (swap in the manifest that matches what’s on disk — a 4 KB change, no 4 GB re-sync), but the lesson is the rule above: the served manifest is load-bearing, so publish add-only and validate that every entry exists. Don’t hand-merge entries for builds you’ve removed.

4. VKS Kubernetes Releases (VKR)

VKS (vSphere Kubernetes Service) provisions guest clusters; each cluster’s node image comes from a VKR content library. In 9.1 vks-download-tool does not fetch it (expected to change in 9.1.1), so it’s a manual mirror. The good news: the source is a public, no-auth VMware CDN that is itself a valid content library:

• https://wp-content.vmware.com/v2/latest/ — lib.json + items.json, currently 123 releases (v1.17.x → v1.34.x). Each item is an ob-<build>-<os>-<arch>-v<ver>…/ directory with the OVA + item.json.

Supervisor 9.1 only deploys a recent K8s window, so mirror a slice rather than all 123. Filtering to the v1.34 line is ~9 items / ~39 GB — a clean upgrade-demo path. The key trick: write a trimmed items.json containing only the items you mirrored, so the published library is self-consistent.

## mirror a version slice (writes a trimmed items.json), then publish to PROD/COMP/VKR
vkr-mirror --filter 'v1\.34\.' --dest /mnt/nas/vcfdepot/depot/_stage/vkr   # resumable
rsync -rltu /mnt/nas/vcfdepot/depot/_stage/vkr/ \
            /mnt/nas/vcfdepot/depot/PROD/COMP/VKR/

VKR’s version of the gotcha. The depot once had 14 GB of GA v1.34.2 content on disk but 0-byte lib.json/items.json — advertised but non-functional. Same lesson: content without a valid manifest = nothing works. A subscribed library also fails on a single missing artefact, which is exactly why the mirror writes a trimmed manifest matching only what it pulled. Extend the slice later by widening the filter (e.g. add v1\.35\.).

---

Pointing VCF at the Depot

Binaries (Installer / VCF Operations)

For a new install, trust the depot CA chain on the installer/SDDC Manager appliance, then in the installer UI open Depot Settings → Binary Management → Offline Depot, enter https://vcfdepot.pgnet.io with the htpasswd credentials, and Configure. Once active, Download the 9.1 binaries and continue the bring-up.

For a deployed fleet, set it once under Build → Software Depot → Edit → Offline Depot (same URL + credentials). That single setting auto-propagates to the VCF instance and SDDC Manager. Then run the lifecycle metadata sync + ESX component sync from Build → Lifecycle → VCF Instances and confirm both finish under Build → Tasks.

Supervisor & VKR (vCenter content-library subscriptions)

The two content libraries are consumed by subscribing to them in vCenter (not via the binary depot config). In vSphere Client → Content Libraries → Create → Subscribed, supply the basic-auth credentials and the library URL:

Library	Subscription URL
Supervisor	https://vcfdepot.pgnet.io/PROD/COMP/SUPERVISOR/lib.json
VKR	https://vcfdepot.pgnet.io/PROD/COMP/VKR/lib.json

Download content immediately to a shared datastore. Assign the Supervisor library to the cluster before enabling Supervisor (it exposes the 9.1 OVF + spherelet artifacts); the VKR library then feeds VKS guest-cluster provisioning. Test by enabling Supervisor and provisioning a VKS cluster on a v1.34.x release.

Refreshing & Salt

A refresh is the same per-content workflow: re-pull/mirror into staging, snapshot, publish add-only into PROD, then trigger the fleet’s metadata/ESX sync (binaries/ESX) or let the subscribed libraries re-sync (Supervisor/VKR).

The bastion host (pgext2) is a Salt minion, so each publish path is wrapped in a small state and run on demand — the binaries/ESX refresh, plus a supervisor_depot and vkr_depot state that each codify the check → stage → validate → snapshot → add-only sync → verify pipeline (so a single stale manifest can’t silently break a library):

salt-call --local state.apply depot.refresh                                   # binaries + ESX
salt-call --local state.apply supervisor_depot pillar='{"supervisor_depot": {"zip": "VMware-vSphere-Supervisor-9.1.0.0100-25417926.zip"}}'
salt-call --local state.apply vkr_depot pillar='{"vkr_depot": {"filter": "v1\\.34\\."}}'

The fleet-side metadata sync still runs from VCF Operations. See the Salt guide for the minion setup pattern.

Lab Notes & Gotchas

• Stage, snapshot, add-only. Never write into live PROD directly; snapshot the depot dataset first; publish with rsync and no --delete.
• Manifest ↔ disk mismatch (see §3) — the single biggest content-library trap. The served items.json must list exactly what’s on disk; one missing artefact fails the whole library sync.
• Cert SANs must include the IP on the vcfdepot.pgnet.io cert — otherwise the fleet’s import into the Fleet Depot Service fails after bring-up (repeated “trust this certificate” prompts that never stick).
• Full-file pulls only. The vcfdepot vhost sets max_ranges 0, so clients can’t resume partial downloads — keep the depot ↔ fleet path fast and stable.
• Activation code is per-depot. Rebuild the depot → new software depot ID + fresh activation code.
• VKS needs VKR in 9.1. vks-download-tool doesn’t mirror the VKR library yet; symptom if missing is repeated “cannot find the 9.1 VKR artefacts”. Expected to be automated in 9.1.1.

Depot Host (nginx)

For reference, the depot is a self-contained nginx:alpine container (pgdepot) deployed with Docker Compose — nginx terminates TLS itself, no upstream proxy. William Lam’s blog is the best background on the offline depot and how to serve it (see Further Reading). A couple of notes on the setup:

• The NFS depot share is bind-mounted to /var/www/vcfdepot/source; the vcfdepot vhost serves source/depot (which holds the active PROD tree).
• Certificates are mounted read-only at /etc/nginx/certs/, named per vhost. The vcfdepot.pgnet.io cert must list the FQDN and the depot IP in its SANs.
• Basic auth uses an .htpasswd generated at container start from HTPASSWD_USER / HTPASSWD_PASS by the mounted entrypoint (hashes with openssl passwd -apr1; only the hash is written, to a tmpfs).
• Two vhosts: depot.pgnet.io is a public autoindex (handy for browsing); vcfdepot.pgnet.io is the authenticated base URL the fleet and content-library subscriptions use.

compose.yml:

services:
  pgdepot:
    image: nginx:alpine
    container_name: pgdepot
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - /mnt/nas/vcfdepot:/var/www/vcfdepot/source:rw   # NFS depot share from the NAS
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - ./entrypoint.sh:/docker-entrypoint.d/10-htpasswd.sh:ro
      - ./certs:/etc/nginx/certs:ro
    tmpfs:
      - /run/htpasswd:mode=0700
    environment:
      HTPASSWD_USER: "vcf"
      HTPASSWD_PASS: "<from-secrets-store>"

nginx.conf:

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1d;

# HTTP :80 — redirect to HTTPS, answer health checks
server {
    listen 80 default_server;
    server_name _;
    location = /health { access_log off; default_type text/plain; return 200 "ok\n"; }
    location / { return 301 https://$host$request_uri; }
}

# HTTPS catch-all — refuse unknown hosts
server {
    listen 443 ssl default_server;
    server_name _;
    ssl_certificate     /etc/nginx/certs/depot.pgnet.io.crt;
    ssl_certificate_key /etc/nginx/certs/depot.pgnet.io.key;
    location / { return 444; }
}

# depot.pgnet.io — public autoindex of the whole content tree, no auth
server {
    listen 443 ssl;
    server_name depot.pgnet.io;
    ssl_certificate     /etc/nginx/certs/depot.pgnet.io.crt;
    ssl_certificate_key /etc/nginx/certs/depot.pgnet.io.key;
    root /var/www/vcfdepot/source;
    location / { autoindex on; autoindex_localtime on; }
}

# vcfdepot.pgnet.io — VCF depot, root maps to source/depot, basic auth
server {
    listen 443 ssl;
    server_name vcfdepot.pgnet.io;
    ssl_certificate     /etc/nginx/certs/vcfdepot.pgnet.io.crt;
    ssl_certificate_key /etc/nginx/certs/vcfdepot.pgnet.io.key;
    max_ranges 0;                       # force full-file pulls (no resume)
    location / {
        root /var/www/vcfdepot/source/depot;
        autoindex on;
        add_header Accept-Ranges "none" always;
        auth_basic "VCF Depot";
        auth_basic_user_file /run/htpasswd/.htpasswd;
        client_max_body_size 0;
        try_files $uri $uri/ =404;
    }
    # NoCloud datasource kept at a stable, unauthenticated path
    location /source/cloud-init/ { alias /var/www/vcfdepot/source/cloud-init/; autoindex on; }
}

Further Reading

William Lam’s blog is an excellent source on the VCF offline depot and how to host it — these were invaluable putting this together:

• VCF 9.1 — New HTTP Offline Depot Support for VCF Installer & Fleet Depot Service — the 9.1 depot service, HTTP/HTTPS and auth options, and the cert SANs gotcha (in the comments).
• VCF Software Depot Structure Deep Dive for Install & Upgrade — how the depot tree is laid out, which explains what VCFDT writes and what nginx serves.
• Quick Tip: host a VCF offline depot with Python SimpleHTTPServer + auth — a minimal alternative to the nginx host above.
• Amayacitta Gill’s How to build a VCF 9.1 Offline Depot and VKS & Supervisor Content Libraries with an Offline Depot — full build + the Supervisor/VKR library walkthrough this section adapts.

Official Documentation

• Download Bundles to an Offline Depot — the authoritative VCF 9.1 procedure for the VCF Download Tool, depot ID, and activation codes.
• VMware Cloud Foundation 9.1 documentation — full 9.1 doc set (planning & prep, administration, lifecycle).

This page records a specific disconnected lab build and intentionally takes shortcuts the official docs do not. Always check the docs above for current requirements before applying any of this to a production fleet.