VCF 9.1: A Documentation Assistant That Cites Its Sources

A large language model cannot look anything up. Ask one about a VCF 9.1 default and you get an answer assembled from every version that was ever on the public internet, delivered with total confidence and no page reference to check it against — which is exactly the wrong shape for the only question I ever actually have, which is what does the design guide say I should do about this.

So I built the thing that fixes it, on the lab I already had. It reads the VCF documentation set and answers like this:

$ rag_query.py "what does a stretched cluster need for witness bandwidth?"

 1. [decision] VSAN-STRETCH-CFG-004 (Design Decision)
    VCF 9.1 Architectural Guide · p. 812 · vSAN Stretched Cluster
    "Provision a minimum of 2 Mbps of bandwidth per 1,000 components..."

Document, version, page. That is the whole point — a number you cannot trace to a page is a number you cannot put in a design document or repeat to a customer.

The full build is a separate guide: eleven steps, three parts, about three hours, most of it waiting. This post is why it is worth the afternoon, and the three things I did not expect.

What it actually is

Two VMs. A GeForce card passed through to a VKS worker with a custom VM class, the NVIDIA GPU Operator handling everything driver-side inside Kubernetes, Ollama serving a 24-billion-parameter model on the card, and a retrieval pipeline that turns product PDFs into 15,770 searchable passages carrying their own citations.

Retrieval is the part that matters. The model never has to know anything about VCF — it is handed the six relevant paragraphs at question time and asked to summarise them. Change the documents and the answers change, with no retraining and no fine-tuning. That is also why a card costing A$1,000 is enough: you are not asking the model to have memorised VMware’s documentation, you are asking it to read.

Three things I did not expect

Keyword search beat semantic search, and it was not close. Measured across five real questions with a known-correct passage, plain BM25 put the right passage at mean rank 2.4; vector similarity managed 5.0. Fusing them weighted three-to-one toward keyword got to 2.0. Product documentation has precise, stable vocabulary — “stretched cluster”, “Tier-0 gateway”, “witness appliance” — and the person asking reuses it, so exact matching is the stronger signal and embeddings are the paraphrase backstop. If someone proposes a pure-vector RAG over technical documentation, that table is the counter-argument.

The design-decision tables are the valuable part, and PDF extraction ruins them. Pulled out as a table, a decision record becomes semantic mush — fragments like iSCSI / • / Principal* / • that embed to nothing. Flattened into one sentence per row, it embeds like a sentence and retrieves like one. Those flattened records are 1,629 of the 15,770 passages, and they are the ones that answer “what does the guide say I should do about X”.

There is no vector database, on purpose. 15,770 passages at 768 dimensions is a 48 MB matrix, and a full cosine scan takes about 1.5 ms in numpy. A vector store would add a component to operate, back up and upgrade for no measurable gain at this scale. Reach for one when you outgrow a flat scan, not before.

It cost tokens, and it earned some

Three of those findings only exist because the thing got built and run rather than described. The keyword-versus-vector result came out of an experiment I set up expecting the opposite answer — four retrieval methods over five questions with a known-correct passage each, mean rank recorded — and the shipped weighting is the one that won, not the one I assumed. The residency settings came from measuring three Ollama configurations one variable at a time and watching a 14 GB model become 17 GB and fall off the card. Two models I wanted to use turned out not to fit, and that is in the guide too.

Ten things broke along the way: an undersized root disk, a rollout that could never place a second GPU worker, image pulls wedged behind a slow registry, CDI injection VKS containerd does not support, a certificate rejected for a missing CN, an ingress that never got an address, PodSecurity refusing stock images, a vSphere API key silently dropped for being lowercase, an Ollama registry bug I had to confirm against a plain registry before believing, and every VIP in the lab disappearing when a router lost its BGP neighbours. Each fix sits inline in the guide at the step where you would hit it — the file you copy already has the fix in it.

The guide is equally explicit about what was not tested: it is n=1, the retrieval comparison is five questions rather than a benchmark, there is no concurrency testing, and the NSX load-balancer path is cited from the documentation rather than measured. That list matters as much as the numbers do.

The honest limits

The GPU is a single point of failure and HA cannot rescue it. DirectPath passthrough already rules out vMotion, but High Availability can only restart the worker on a host advertising a matching PCI device — if that card is the only one of its kind in the cluster, an HA event does not relocate the workload, it fails to place it. Every host patch is a declared outage of the endpoint and everything downstream of it.

And this is deliberately the hand-built path. It uses neither VMware Private AI Foundation with NVIDIA nor VCF Automation, and that omission is the point of the exercise rather than a recommendation. What I built is one service, provisioned by an administrator with kubectl, for a handful of people who know it exists. What a business needs is a self-service catalogue, tenancy and quotas, model governance, GPU sharing across teams and a supported lifecycle — which is precisely what those two products are. Build this to learn the mechanics and to prove the platform. Recommend the product path for anything anyone depends on.

The guide is explicit about where the line sits, with the reference-design decisions that route through VCF Automation by default and a straight list of what hand-building costs you.

Where to go next

The build guide has every step: the VM class JSON the REST API is picky about, the three ClusterClass choices that are the difference between a cluster that works and one that fights you for an afternoon, the GPU Operator flags, ingress and TLS, and the measurements behind every tuning value.

As always, this is a lab walkthrough that stitches the public sources into one flow — it doesn’t replace Broadcom’s own documentation. Where they differ, the official docs win. And treat the generated answers as a reading aid rather than a source: retrieval returns real passages with real page numbers, but the synthesis step can still garble a summary. Read the cited passage.

Take the ideas, not the liability.