Skip to content

Operations and Troubleshooting

Manual Trigger Strategies

There are two ways to manually trigger the release gating pipeline, each with different scope.

Full CronJob Run (All Service Gates)

This re-runs the entire nightly flow, triggering all managed service gates defined in ITS_NAMES:

oc create job --from=cronjob/hypershift-operator-nightly-promotion \
  ho-release-gate-manual-$(date +%s) -n crt-redhat-acm-tenant

Use this when you need to re-validate all services (e.g. after a shared infrastructure fix).

Snapshot Label (Single Service Gate)

This triggers only one specific ITS, useful for re-testing a single service without affecting others:

SNAPSHOT_NAME=$(oc get snapshot -n crt-redhat-acm-tenant \
  --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')

oc label snapshot "$SNAPSHOT_NAME" \
  test.appstudio.openshift.io/scenario=<its-name> \
  -n crt-redhat-acm-tenant --overwrite

Replace <its-name> with the target ITS name, for example hypershift-ho-release-gate-aro-hcp. The Integration Service will detect the label and create a new PipelineRun for that ITS only.

When to Use Which

Scenario Strategy
Re-validate all services after an infrastructure change CronJob
Re-test a single service after fixing a service-specific issue Snapshot label
Test a new ITS configuration Snapshot label
Nightly run failed due to a transient error Snapshot label (for the affected gate)

Inspecting a PipelineRun

Fetching the PipelineRun

List recent release gating PipelineRuns:

oc get pipelineruns -n crt-redhat-acm-tenant \
  --sort-by=.metadata.creationTimestamp | tail -5

Reading Task Logs

Find the pods for a specific PipelineRun, then read the logs for a specific task step:

oc get pods -n crt-redhat-acm-tenant -l tekton.dev/pipelineRun=<pipelinerun-name>

oc logs pod/<pod-name> -c step-<step-name> -n crt-redhat-acm-tenant

Warning

PipelineRun pods are subject to aggressive garbage collection on the Konflux cluster. If the pods have already been deleted, use the Konflux UI instead (see below), where logs are persisted, centralized, and aggregated across all tasks of the pipeline.

Konflux UI

PipelineRun logs and Release CR status are available in the Konflux PipelineRuns view.

Tip

Use the PipelineRun name from the oc get pipelineruns command above to filter the list in the UI.

Historical PipelineRun Data (KubeArchive)

The oc get pipelineruns command only returns PipelineRuns that still exist on the cluster. Due to aggressive garbage collection on stone-prd-rh01, PipelineRuns are deleted shortly after completion. For historical data (e.g. investigating a stale promotion streak or reviewing failures beyond what the Slack notification displays), query the KubeArchive REST API directly:

curl -s -H "Authorization: Bearer $(oc whoami -t)" \
  "https://kubearchive-api-server-product-kubearchive.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/apis/tekton.dev/v1/namespaces/crt-redhat-acm-tenant/pipelineruns?labelSelector=test.appstudio.openshift.io/scenario=<ITS_NAME>" \
  | jq -r '.items[] | "\(.metadata.name)  status=\(.status.conditions[-1].status // "?")  reason=\(.status.conditions[-1].reason // "?")"'

Note

The KubeArchive URL in the curl command above corresponds to the default value of the kubearchive-api-base pipeline parameter. If the pipeline has been reconfigured to point at a different KubeArchive instance, use that URL instead.

Replace <ITS_NAME> with the target service gate name (e.g. hypershift-ho-release-gate-aro-hcp). You must be logged in to the stone-prd-rh01 cluster (oc login).

The output lists all archived PipelineRuns for that ITS with their completion status and reason. To inspect a specific PipelineRun from the results, build the Konflux UI URL from its name:

https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/ns/crt-redhat-acm-tenant/applications/hypershift-operator/pipelineruns/<pipelinerun-name>/

This URL provides the full task logs, results, and pipeline visualization even after the PipelineRun has been garbage-collected from the cluster.

Common Failure Scenarios

Gangway Token Expired

Symptom: run-e2e task fails with HTTP 401 errors when triggering Prow jobs.

Fix: rotate the gangway-token Secret in crt-redhat-acm-tenant. The token is a Prow CI cluster OAuth token.

Slack Webhook 4xx

Symptom: notify-slack logs show repeated 4xx errors after 3 retries.

Fix: verify the webhook URL in the slack-webhook Secret is still valid. Slack webhooks can be revoked if the app is reinstalled.

clone-lib Failure

Symptom: clone-lib task fails with git errors.

Common causes:

  • Repository URL or branch changed
  • GitHub rate limiting on unauthenticated git clones
  • Network connectivity from the Konflux cluster

PVC Issues

Symptom: tasks fail with workspace mount errors or permission denied on shared files.

Common causes:

  • PVC quota exceeded in the tenant namespace
  • Storage class unavailable
  • Stale PVCs from previous failed runs (Konflux garbage-collects these, but delays can occur)

Prow Job Timeout

Symptom: run-e2e task reaches its 4-hour polling timeout with jobs still pending.

Common causes:

  • Prow cluster capacity issues (jobs queued but not scheduled)
  • The E2E test itself is stuck or abnormally slow
  • Gangway API returning stale status

Mitigation: check the Prow job directly in the Prow UI using the URL from the run-e2e task logs. If the job is stuck, it may need to be manually cancelled in Prow before re-triggering the gate.

KubeArchive Unreachable

Symptom: notify-slack or notify-slack-error logs show warnings about failing to fetch PipelineRun history from KubeArchive. The gate result notification is still sent, but no stale promotion alert appears.

Common causes:

  • KubeArchive service is down or restarting on stone-prd-rh01
  • The projected ServiceAccount token has expired or the audience (kubearchive) is misconfigured
  • Network policy changes blocking cluster-internal traffic

Impact: the stale check is a non-blocking operation. If KubeArchive is unreachable, the pipeline logs a warning and skips the stale alert. The gate verdict and notification are not affected. The stale check will resume automatically on the next run when KubeArchive becomes available again.

Unexpected Stale Alert

Symptom: a stale promotion alert is sent even though the gate has not been failing for long, or the streak count seems wrong.

Common causes:

  • stale-threshold-days is set too low in the ITS (e.g. 1 would alert on the first failure)
  • Test PipelineRuns from integration testing contribute to the real streak history because they are archived with the same ITS label. The streak resets automatically on the first successful nightly run
  • KubeArchive returned incomplete data (e.g. after a data migration or cleanup)