Skip to content

Configure Custom Ingress for KubeVirt HCP

This recipe walks through deploying a KubeVirt-based Hosted Control Plane with a custom baseDomain (without baseDomainPassthrough) on a bare-metal management cluster using MetalLB for LoadBalancer services.

This is the typical setup when the guest cluster needs its own DNS domain separate from the management cluster's *.apps domain, and an external load balancer (F5, HAProxy, etc.) or MetalLB handles VIP advertisement.

Prerequisites

  • A bare-metal OpenShift management cluster with KubeVirt (OpenShift Virtualization) installed
  • MetalLB Operator installed (see Optional MetalLB Configuration Steps)
  • A DNS zone you control for the custom baseDomain
  • VM network configured with a secondary bridge interface (the VMs must have machineNetwork connectivity, not just pod network)

Environment Variables

Set these once — all subsequent commands reference them:

export CLUSTER_NAME=my-kubevirt-hcp
export BASE_DOMAIN=example.com
export HCP_NAMESPACE="clusters-${CLUSTER_NAME}"
export PULL_SECRET="$HOME/pull-secret"
export MEM="6Gi"
export CPU="2"
export WORKER_COUNT="2"

Step 1 — Create the HostedCluster

hcp create cluster kubevirt \
  --name $CLUSTER_NAME \
  --node-pool-replicas $WORKER_COUNT \
  --pull-secret $PULL_SECRET \
  --memory $MEM \
  --cores $CPU \
  --base-domain $BASE_DOMAIN

Because --base-domain is provided, the webhook does not enable baseDomainPassthrough. The cluster will stay in Partial progress until ingress is manually configured.

Step 2 — Configure MetalLB

2.1 — Create the MetalLB instance

apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
  name: metallb
  namespace: metallb-system

2.2 — Create the IPAddressPool

Adjust the address range to match available IPs on your bare-metal network:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: hcp-ingress-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.216.200-192.168.216.220

Warning

The MetalLB address pool must be disjoint from the VM machineNetwork addresses. If the pool includes IPs assigned to VMs (e.g., 192.168.216.50, 192.168.216.51 in this example), MetalLB may allocate a VIP that conflicts with an existing VM address.

2.3 — Create the L2Advertisement

If your network uses a specific bridge interface (e.g., br-sdn), add interfaces and nodeSelectors as needed:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: hcp-ingress-l2
  namespace: metallb-system
spec:
  ipAddressPools:
  - hcp-ingress-pool

Step 3 — Retrieve the Guest Router NodePorts

Wait for the guest cluster to have running worker nodes, then extract the dynamically assigned NodePorts:

export CLUSTER_KUBECONFIG="${CLUSTER_NAME}-kubeconfig"
hcp create kubeconfig --name $CLUSTER_NAME > $CLUSTER_KUBECONFIG

export HTTP_NODEPORT=$(oc --kubeconfig $CLUSTER_KUBECONFIG get services \
  -n openshift-ingress router-nodeport-default \
  -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}')

export HTTPS_NODEPORT=$(oc --kubeconfig $CLUSTER_KUBECONFIG get services \
  -n openshift-ingress router-nodeport-default \
  -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')

echo "HTTP NodePort: $HTTP_NODEPORT"
echo "HTTPS NodePort: $HTTPS_NODEPORT"

Step 4 — Retrieve VM machineNetwork IPs

oc get vmi -n $HCP_NAMESPACE -o json | \
  jq -r '.items[] | "\(.metadata.name)\t\(.status.interfaces[] | select(.name != "default" and .ipAddress != null and .ipAddress != "") | .ipAddress | split("/")[0])"'

This filters out the pod network interface (default) and strips any CIDR suffix. If your VMs use a different interface layout, check all interfaces with oc get vmi -n $HCP_NAMESPACE -o yaml and adjust the filter.

Example output:

my-kubevirt-hcp-workers-abc12-xyz34    192.168.216.50
my-kubevirt-hcp-workers-abc12-xyz56    192.168.216.51

Warning

Use the machineNetwork IPs (the VM's network interface on the secondary bridge), not the virt-launcher pod IPs. The guest router's NodePort only listens on machineNetwork IPs. Using pod IPs causes connection refused errors. See Ingress and DNS - Troubleshooting for details.

Step 5 — Create the LoadBalancer Service (no selector)

cat << EOF | oc apply -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-apps-ingress
  namespace: ${HCP_NAMESPACE}
spec:
  ports:
  - name: https-443
    port: 443
    protocol: TCP
    targetPort: ${HTTPS_NODEPORT}
  - name: http-80
    port: 80
    protocol: TCP
    targetPort: ${HTTP_NODEPORT}
  type: LoadBalancer
EOF

The Service has no selector. Traffic routing is handled entirely by the EndpointSlice below.

Step 6 — Create the EndpointSlice

Replace IP addresses with the values from Step 4:

cat << EOF | oc apply -f -
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: ${CLUSTER_NAME}-apps-ingress
  namespace: ${HCP_NAMESPACE}
  labels:
    kubernetes.io/service-name: ${CLUSTER_NAME}-apps-ingress
    endpointslice.kubernetes.io/managed-by: manual
addressType: IPv4
ports:
- name: https-443
  port: ${HTTPS_NODEPORT}
  protocol: TCP
- name: http-80
  port: ${HTTP_NODEPORT}
  protocol: TCP
endpoints:
- addresses:
  - "192.168.216.50"
- addresses:
  - "192.168.216.51"
EOF

Step 7 — Configure Wildcard DNS

Get the VIP assigned by MetalLB:

export EXTERNAL_IP=$(oc -n $HCP_NAMESPACE get service ${CLUSTER_NAME}-apps-ingress \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "LoadBalancer VIP: $EXTERNAL_IP"

Create a wildcard DNS record:

*.apps.my-kubevirt-hcp.example.com.  IN  A  <EXTERNAL_IP>

Verify:

dig +short test.apps.${CLUSTER_NAME}.${BASE_DOMAIN}

Important

DNS must resolve correctly both externally and from inside the guest VMs. If the VMs resolve *.apps to the MetalLB VIP but the return traffic path is broken (asymmetric routing), the ingress canary checks will fail. Configure split-horizon DNS so guest VMs resolve directly to their own machineNetwork IPs. See the JSON patch tip below for an automated approach.

Tip

As an alternative to split-horizon DNS, you can inject custom DNS configuration directly into the KubeVirt VMs using a JSON patch on the NodePool. This overrides the VM's DNS resolver so it points to a nameserver that returns the correct IPs from inside the guest network:

apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  name: my-kubevirt-hcp
  namespace: clusters
  annotations:
    hypershift.openshift.io/kubevirt-vm-jsonpatch: |
      [
        {
          "op": "add",
          "path": "/spec/template/spec/dnsPolicy",
          "value": "None"
        },
        {
          "op": "add",
          "path": "/spec/template/spec/dnsConfig",
          "value": {
            "nameservers": ["10.0.0.53"]
          }
        }
      ]

Warning

Setting dnsPolicy: None removes the default cluster search domains (e.g., svc.cluster.local). Do not add your external baseDomain to the searches list — this causes internal .svc.cluster.local lookups to be appended with the external domain and resolve to public IPs, breaking services like the console. If your custom nameserver at 10.0.0.53 needs search domains, include only the cluster-internal ones:

"searches": ["svc.cluster.local", "cluster.local"]

See Configuring VMs with JSON Patch for full details on the JSON patch mechanism.

Step 8 — Verify

Check HostedCluster progresses to Completed:

oc get --namespace clusters hostedclusters

Expected output:

NAME              VERSION   KUBECONFIG                         PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
my-kubevirt-hcp   4.17.0    my-kubevirt-hcp-admin-kubeconfig   Completed   True        False         The hosted control plane is available

Verify ingress from outside:

curl -vk https://console-openshift-console.apps.${CLUSTER_NAME}.${BASE_DOMAIN}

Check the ingress operator is not degraded inside the guest:

oc --kubeconfig $CLUSTER_KUBECONFIG get co ingress

Maintenance

The EndpointSlice is not automatically managed. Update it when:

Event Action
Scale up (new VMs) Add new VM machineNetwork IPs to the EndpointSlice
Scale down Remove decommissioned VM IPs
Live migration Update IPs if machineNetwork address changed

Quick command to get current VM IPs:

oc get vmi -n $HCP_NAMESPACE -o json | \
  jq -r '.items[] | "\(.metadata.name)\t\(.status.interfaces[] | select(.name != "default" and .ipAddress != null and .ipAddress != "") | .ipAddress | split("/")[0])"'

Traffic Flow

Client
  └─> *.apps.my-kubevirt-hcp.example.com     (DNS wildcard)
       └─> MetalLB VIP (e.g. 192.168.216.200) (L2 advertisement)
            └─> VM machineNetwork IP          (EndpointSlice)
                 └─> NodePort (e.g. 31245)    (guest router)
                      └─> guest Route         (application)