AWS HostedCluster creation and deletion observations (5.0.0-rc.7)

These observations cover two separate AWS HostedClusters created and then deleted on the same development management cluster on 28 October 2026. Tables identify them as HostedCluster 1 and HostedCluster 2; chart labels use HC1 and HC2. Each has its own creation timestamp, infrastructure ID, startup measurements, and deletion measurements, listed below.

Both used release quay.io/openshift-release-dev/ocp-release:5.0.0-rc.7-multi and HO image quay.io/acm-d/rhtap-hypershift-operator@sha256:d7eb652dbbfac8ae9866b413c7125c86961a5cfb46396eb31774e0b3e171a3a5. Each had two m5.large workers in us-east-1a and public endpoints. Both used SingleReplica policies for the control plane and infrastructure. The management cluster and its caches were warm. The requested HO was healthy before each create.

Neither startup needed manual credential repair or bootstrap intervention. Both teardowns used the normal destroy path without force or manual finalizer removal. Both clusters and their tracked AWS infrastructure were removed.

See the startup roadmap, AWS startup debugging examples, and teardown roadmap for gate definitions and investigation paths.

Lifecycle summary

Startup times begin at HC creation. Teardown times begin immediately before invoking the destroy script, including STS refresh and CLI setup. ~ marks the first successful polling observation. The table lists idle review time separately and excludes it from both clocks. Startup milestones use the earliest captured successful transition. Later successful condition timestamps can differ.

These two samples do not establish a typical duration or explain the timing differences. HostedCluster 1 remained available for review before teardown. HostedCluster 2's teardown began after capture of its completion evidence and deletion inventory. HostedCluster 1's command logs retain a NAT-quota retry before HC creation. That retry does not contribute to startup timing.

Startup comparison

Each marker measures elapsed time since HC creation. It does not measure the duration of a separate phase. Both HostedClusters use HC creation as zero, and ~ marks the first successful polling observation. Select a chart to open the full-size SVG. The table below contains all values.

Observed startup waits

Both HostedClusters published valid OIDC configuration immediately and completed without manual credential repair, an HO rollout, or bootstrap intervention. HostedCluster 2 first reached HCP availability 35 seconds earlier and NodePool readiness 12 seconds earlier, while the full checklist was first observed at about 17m02s in both. These small differences are observations, not evidence of a reliable performance improvement.

The full trace captures HostedCluster 2's first Available=True transition at +3m57s. A later successful condition carries a +4m12s transition timestamp, also the control-plane rollout completion time. Both captured observations were True; no intervening False was captured. The tables retain the earliest successful transition rather than replacing it with the latest condition timestamp.

AWS identity validation passed at +5m34s for HostedCluster 1 and +5m33s for HostedCluster 2; each default worker security group appeared one second later. The startup debugging examples explain this wait using the observed condition messages and the pinned CPO source, including how to distinguish KAS unavailability from an unavailable EC2 client.

Monitoring was the last guest operator to converge in both HostedClusters. HostedCluster 1 reported missing cluster-monitoring-view and monitoring-alertmanager-edit RBAC dependencies. HostedCluster 2 reproduced the Alertmanager role-binding failure. Its log records the initial failure at 21:22:06Z, followed by a five-minute availability-validation wait and a failed task result at 21:27:06Z. The next reconciliation created Prometheus/k8s at +16m15s, and monitoring converged at +16m35s. The corresponding HostedCluster 1 times were +16m14s and +16m34s. Reconciliation resolved both failures. Checking only existing monitoring pods would have missed the absent Prometheus resource during this interval.

Teardown comparison

Deletion timestamps and observed absence are different signals. The first row below uses the HC API deletion timestamp. Resource-removal rows use the first successful collection that confirms absence or an AWS terminal state. The command-return row in the summary uses the wrapper's actual return time.

All chart markers except command return are first successful polling observations. Lines connect the two HostedClusters for comparison; they do not measure individual cleanup-task durations. The table lists all tracked resources.

Observed teardown waits

HCP deletion is an intermediate milestone. In both HostedClusters, the workers terminated and CAPI removed the Machine objects before HCP removal. The HCP namespace, component API objects, Services and remaining API-server pods remained afterward. Removing a component API object does not establish that its workload has stopped.

API-server grace periods explain part of the namespace wait. The HostedCluster 2 observation captured terminationGracePeriodSeconds and deletionGracePeriodSeconds: 135 seconds for KAS and 120 seconds for the OpenShift API and OAuth API servers. For example, the KAS pod was already observed terminating in the collection ending 21:32:45Z, with a future deletion timestamp of 21:34:34Z. That timestamp is a grace deadline, not a request time. HostedCluster 1 observations include pod deletion timestamps but not explicit grace-period fields. HostedCluster 2 observations include both.

Read namespace conditions alongside the remaining objects. In HostedCluster 2, the namespace reported no remaining content or finalizers while the captured polls still showed terminating API-server pods and component objects. Check each resource's presence before concluding that the namespace is empty.

The final CLI phase is separate from controller cleanup. After the HCP namespace disappeared, the destroy command continued removing remaining VPC dependencies, NAT infrastructure, IAM roles/profiles and the OIDC provider. The CLI finalizer kept the HC present during this work. The dataset records command return and the first collection that confirmed Kubernetes and AWS cleanup as separate measurements. Neither deletion used force or manual finalizer removal.

Both commands logged transient internet-gateway DependencyViolation errors for mapped public addresses while the NAT gateway was still deleting. They retried and completed without intervention. In this state, follow NAT and network-dependency progress before treating a retry message as a leak or removing a finalizer.

Control-plane component comparison

First successful transitions may precede later rollouts. These component transitions are separate from aggregate HCP availability and full completion.

The chart compares first availability in both HostedClusters. The table lists components alphabetically and retains both availability and rollout timestamps. These observations do not establish a dependency order. Rollout completion can follow availability. Open the image for the full-size vector chart.

Creation to component availability

These durations start at ControlPlaneComponent object creation and include controller and dependency waits. They do not measure container startup time.

Guest operator comparison

An operator converges when it reports Available=True, Progressing=False, and Degraded=False. It can resume progressing afterward. Full completion requires all operators to report these conditions in the same collection cycle.

The vertical reference lines mark NodePool readiness. Some operators converge before workers; others need guest workloads to run. Monitoring is the last operator in these two observations. Check all operators together before reporting full completion.

Measurement limits

Startup and teardown were polled at ten-second intervals, with query duration adding to the interval between observations. Startup polling timestamps mark query start; teardown absence timestamps mark query completion. API transition timestamps are used where available. Polling observations are approximate: queries span a time window, and AWS eventual consistency can delay a result. A failed query leaves the resource state unknown; it does not prove deletion.

Teardown checks covered the worker VPC and its resources, plus control-plane load balancers discovered through HCP Services in the management VPC. Shared management resources, the HO installation, OIDC bucket, and base-domain zone were outside the teardown scope.