HyperShift Operator Nightly Releases

The HyperShift Operator built from main is published as a nightly release stream, gated by the same OpenShift release controller that accepts and rejects OCP nightly payloads. Each nightly is a Konflux-built operator image that has been tagged into a CI imagestream and offered to the controller as a release candidate.

Browse the stream at hcp.releases.ci.openshift.org.

The stream gives you:

• The Konflux-built pull spec for each candidate, and the verification status the controller reached for it.
• A changelog between any two nightlies, rendered from the commits that went into them.

How it works

graph LR

    subgraph Release controller
        direction LR
        H["Release controller — release-namespace=hypershift - wait for new tags"]
        H --> I["Adopt newest candidate — supersede the rest as skipped"]
        I --> J["Verification Prow jobs, image injected as OVERRIDE_HYPERSHIFT_OPERATOR_IMAGE"]
        J --> K[["Gather Results for acceptance"]]
        K --"Accepted only"--> L[Create Konflux Release]
    end

    subgraph Konflux
        direction LR
        A["Push to main"] --> B["Konflux build — hypershift-operator-main-push"]
        B --> C["Snapshot — component hypershift-operator-main"]
        C --> D["Integration Service — Snapshot eligible, auto-release ReleasePlan"]
        D --> E["Release — admitted and configured by the ReleasePlanAdmission"]
        E --> F["Tekton pipeline — hypershift-nightly-release-tag"]
        F --> G["oc tag into imagestream hypershift/hypershift-operator-main-nightly on app.ci"]
    end

1. Konflux builds the operator

A push to main triggers .tekton/hypershift-operator-main-push.yaml, which runs the shared common-operator-build pipeline and pushes quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-operator/hypershift-operator-main:<revision>.

The build bakes /release-manifests/image-references into the image — see Changelogs.

2. Konflux creates a Snapshot and a Release

Konflux's Integration Service creates a Snapshot describing the built components, then creates a Release for it against each auto-release-enabled ReleasePlan for the application. Two separate things have to hold:

• The Snapshot must be eligible — its required integration tests have finished and passed, it is valid, it came from a push (not a pull request) and it is not a group snapshot or labelled auto-release: "false".
• A ReleasePlan in crt-redhat-acm-tenant must have auto-release enabled (the release.appstudio.openshift.io/auto-release label is "true" or absent).

The ReleasePlanAdmission (RPA) is the managed-side counterpart: it admits the ReleasePlan, names the pipeline to run, and carries the release configuration. It is not what creates the Release, so when a Release is missing, check the ReleasePlan and the Snapshot's eligibility first.

The RPA is where per-component tagging config lives:

spec:
  data:
    mapping:
      components:
      - name: hypershift-operator-main                     # must match the Snapshot component
        destination-imagestream: hypershift/hypershift-operator-main-nightly
        release-version: "0.0.1"                           # optional, see below

3. The tag pipeline runs

The Release runs .tekton/pipelines/nightly-release-tag.yaml (hypershift-nightly-release-tag), which:

1. verify-access-to-resources — checks the pipeline can read the release CRs.
2. collect-data — merges spec.data from the Release, ReleasePlan and RPA into data.json.
3. apply-mapping — merges the mapping into the Snapshot components, failing the release if nothing matches.
4. tag-image — computes a tag per component and runs oc tag against the CI cluster. Defined inline in the pipeline above, not in the catalog.

tag-image authenticates to app.ci with a kubeconfig from the hypershift-ci-cluster-credentials secret, mounted at /credentials/kubeconfig.

4. The release controller picks up the tag

A dedicated release controller instance (deploy-hypershift-controller.yaml) runs on app.ci with --release-namespace=hypershift, watching the hypershift-operator-main-nightly imagestream. Its configuration is annotated onto that imagestream from openshift/release — see Configuration.

When the controller is ready to start a cycle it adopts the most recently imported tag, marks any candidate it superseded as skipped, launches the verification Prow jobs for the adopted one, and moves it to Accepted or Rejected.

"minCreationIntervalSeconds": 14400

"maxUnreadyReleases": 2

How a nightly gets its name

A nightly tag is a version plus a semver pre-release segment:

0.1.86-148-g0bb9d28005.nightly-2026-10-02-174500
└───── version ──────┘ └─── nightly segment ───┘

The version

Either the RPA's release-version for that component, or — when it is omitted — derived by running git describe --tags against the component's source.git.revision.

The timestamp

<ts> is the time the release pipeline ran. tag-image takes it once per run with date -u, so every component tagged by one Release shares a timestamp.

Changelogs and the release-manifests contract

oc adm release info --changelog is payload-only: it extracts release-manifests/image-references from the image's topmost layer and errors out when the file is absent. The HyperShift Operator image is not a payload, so the build fakes a minimal one-tag ImageStream to satisfy it.

release-manifests/image-references.template.yaml is rendered by Containerfile.operator from three build args, wired up in .tekton/pipelines/common-operator-build.yaml:

oc generates the changelog entirely from the io.openshift.build.* annotations: it clones source-location and runs git log over the range between the two builds' commit.id values. A single tag naming this repository is therefore enough.

Configuration

Configuration is spread across this repository, openshift/release and the Konflux tenant namespace. Find the row for what you want to change:

Release controller config

release-hcp-main.json is annotated onto the imagestream by hack/annotate.sh in openshift/release:

annotate "hypershift" "hypershift-operator-main-nightly" "hcp-main.json" "hypershift"

The fifth argument is the subdirectory under _releases/.

The fields in release-hcp-main.jsonworth knowing:

A verify entry looks like:

"release-5.1-e2e-aks": {
    "maxRetries": 2,
    "prowJob": {
        "name": "periodic-ci-openshift-hypershift-release-5.1-periodics-e2e-aks"
    }
}

The key is a label of your choosing; prowJob.name must match an existing Prow job exactly. Job names are generated rather than hand-written, so look them up in openshift/release rather than constructing them:

Names are assembled as periodic-ci-<org>-<repo>-<branch>-<variant>-<test>, where <variant> is the __suffix on the ci-operator config filename and <test> is the test's as field. So periodic-ci-openshift-hypershift-release-5.1-periodics-hcm-azure-e2e-aks-upgrade-minor is the test e2e-aks-upgrade-minor from openshift-hypershift-release-5.1__periodics-hcm-azure.yaml. Every variant for a branch still lands in that branch's single generated periodics file.

Add "optional": true to make a job informing — it runs and is reported, but does not block acceptance. New jobs should always start optional and be promoted to blocking once they have proven stable against the stream.

Currently, jobs from the latest release branch are blocking with retries, and jobs from older branches run as informing back-compatibility signal. This configuration is arbitrary and may change in the future.

How verification jobs consume the image

The controller injects the nightly's pull spec into OVERRIDE_HYPERSHIFT_OPERATOR_IMAGE, which is read by the hypershift-install step-registry ref (source) in openshift/release. The step uses it for both the operator image and the hcp CLI, extracted from the same image, bypassing the usual pipeline ImageStream dependency resolution.

Override precedence inside the step, highest first:

1. OVERRIDE_HYPERSHIFT_OPERATOR_IMAGE
2. HO_MULTI
3. INSTALL_FROM_LATEST

Setting more than one logs a warning.

Konflux resources

The ReleasePlan, the ReleasePlanAdmission and the secrets they depend on (hypershift-ci-cluster-credentials, the trusted-artifacts push credential) live in the crt-redhat-acm-tenant namespace on the stone-prd-rh01 Konflux cluster. They are not in this repository — tenant configuration is managed by MR against konflux-release-data — so read the live objects with:

# If not already logged in
oc login --web https://api.stone-prd-rh01.pg1f.p1.openshiftapps.com:6443

oc get releaseplan,releaseplanadmission -n crt-redhat-acm-tenant -o yaml

Builds, Snapshots and Releases for the application are visible in the Konflux UI.

The RPA also references:

• policy: hypershift-nightly — the EnterpriseContractPolicy named for this release. The tagging pipeline itself does not evaluate it: it runs verify-access-to-resources, collect-data, apply-mapping and tag-image, with no EC verification step before oc tag. Treat this as a policy reference, and confirm where it is evaluated before relying on it as a pre-publication gate.
• serviceAccountName: release-service-account — needs RBAC to GET release/releaseplan/releaseplanadmission/snapshot, and a linked quay push secret for the trusted-artifacts steps.

Runbooks

Add a new e2e job to the gate

1. Find the job's exact name in ci-operator/jobs/openshift/hypershift/ (see the verify map).
2. Confirm the job installs HyperShift via the hypershift-install step-registry ref, by checking the workflow it uses in its ci-operator config.
3. Open a PR against openshift/release adding an entry to the verify map in _releases/hypershift/release-hcp-main.json, with "optional": true.
4. Merge it — the HyperShift OWNERS file means the team can approve this.
5. Watch a few nightlies. Once the job is reliably passing, drop optional and add "maxRetries": 2 to make it blocking — bearing in mind that this lengthens the critical path for every subsequent nightly.

A nightly was rejected

1. Open the release in the controller UI and find the failing job.
2. Follow through to the Prow job and its artifacts.
3. Confirm the job actually tested the nightly: the install step logs WARNING: Overriding OPERATOR_IMAGE with the override pull spec. If it does not, the job is not wired to the override and the result is not about this nightly.
4. Use the changelog between the last accepted nightly and this one to narrow down the commit range.

No new nightlies have appeared

Work down the chain:

1. Konflux build — did the push to main build successfully?
2. Snapshot — oc get snapshots -n crt-redhat-acm-tenant. Is the newest one eligible? Check its AppStudioTestSucceeded condition; a Snapshot whose integration tests have not finished or have failed is never auto-released.
3. Release — oc get releases -n crt-redhat-acm-tenant. If no Release was created, check auto-release on the ReleasePlan (oc get releaseplan -n crt-redhat-acm-tenant --show-labels), then that the RPA admits it — applications and origin must match.
4. Release pipeline — check the PipelineRun, and which task failed:
   • apply-mapping failing with ERROR: Resulting snapshot contains 0 components... means no mapping entry name matched a Snapshot component.
   • tag-image failing with mapped snapshot contains 0 components is its own defensive guard on the same condition, reached only if apply-mapping let an empty result through.
   • tag-image failing at oc tag is usually an expired CI kubeconfig or missing RBAC on app.ci.
5. Cadence — minCreationIntervalSeconds: 14400 and maxUnreadyReleases: 1 both legitimately suppress new releases. A nightly stuck in verification blocks the next one. Tags that arrived during the cooldown are not queued: when the controller next adopts, it takes the newest candidate and marks the rest skipped, so a quiet stream and a stream full of skipped tags look different and mean different things.

For archived Konflux PipelineRuns and logs, see Konflux CI Scripts.

Test a tag pipeline change before merging

The pipeline is only exercised by real Releases, so changes are easy to get wrong. Apply the pipeline to the tenant namespace and drive it with a manual PipelineRun that points at an existing Release's CRs:

oc apply -n crt-redhat-acm-tenant -f .tekton/pipelines/nightly-release-tag.yaml
oc get releases -n crt-redhat-acm-tenant       # pick a recent one
oc get release <name> -n crt-redhat-acm-tenant -o yaml   # read its snapshot/releasePlan/RPA refs

Then create a PipelineRun supplying the five params release-service would normally inject, each as <namespace>/<name> of a CR that already exists:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: test-nightly-release-tag-
  # The managed tenant namespace: where the release CRs, the
  # hypershift-ci-cluster-credentials secret and the trusted-ca configmap live.
  namespace: crt-redhat-acm-tenant
spec:
  pipelineRef:
    name: hypershift-nightly-release-tag
  taskRunTemplate:
    # Needs RBAC to read the release CRs below and push to ociStorage.
    serviceAccountName: release-service-account
  params:
  - name: release
    value: crt-redhat-acm-tenant/<release>
  - name: releasePlan
    value: crt-redhat-acm-tenant/<releaseplan>
  - name: releasePlanAdmission
    value: crt-redhat-acm-tenant/<rpa>
  - name: releaseServiceConfig
    value: release-service/<releaseserviceconfig>
  - name: snapshot
    value: crt-redhat-acm-tenant/<snapshot>
  # Pin the catalog for a reproducible run instead of tracking 'production'.
  # - name: catalogGitRevision
  #   value: <commit-sha>

spec:
  data:
    mapping:
      components:
      - name: hypershift-operator-main
        destination-imagestream: <your-namespace>/scratch-nightly   # not hypershift/...
        release-version: "0.0.1"

Manually release a Snapshot

With auto-release enabled on the ReleasePlan this should not normally be needed. If auto-release is turned off — or the Snapshot was ineligible at the time and you want to release it anyway — create a Release referencing the Snapshot and the ReleasePlan in crt-redhat-acm-tenant.

Rotate the CI cluster kubeconfig

Replace the kubeconfig key in the hypershift-ci-cluster-credentials secret in crt-redhat-acm-tenant. The secret name is a pipeline param (ci-cluster-credentials-secret), so a differently-named secret needs the param changed too — the ci-cluster-credentials-secret field in the RPA mapping is informational only and does not select the secret.

Glossary

See also

• HO/CPO Release Process — the versioned release process, distinct from this continuous nightly stream.
• Konflux CI Scripts
• Custom Images