Triaging Presubmit Failures

Your PR has a failing check. Use the flowchart below to figure out what's going on and what to do about it.

Triage flowchart

On your PR, click Details on the failing check. The check name tells you which CI system it belongs to.

The rest of this page gives details for each branch in the flowchart.

GitHub Actions failures

Click Details on the failing check — it takes you to the GitHub Actions workflow run. On that page:

1. Look at the left sidebar for the job name with a red ❌.
2. Click it to expand the job's steps.
3. Find the red step and click it to see the log output.

These checks run on every PR:

These checks only run when relevant files change:

GHA failures are almost always caused by your code changes. Fix and push.

To see all runs of a particular workflow (useful for checking if a failure is widespread), go to the GitHub Actions page and select the workflow from the left sidebar.

Prow jobs

Prow checks appear on your PR as ci/prow/<job-name>. Click Details to open the Prow job page.

Prow non-e2e jobs

These jobs don't run e2e tests. They build images or run static checks.

If images fails, it usually means your code doesn't compile. Search the log for error: and fix. If you're unsure whether it's your code, check the job history — if the same job is red on other PRs, it's not you.

Prow e2e jobs

These jobs create real clusters and run tests against them. They are the most common source of failures.

Jobs marked Auto run when the PR receives /lgtm but can also be triggered manually with /test <job-name>. Jobs marked /test only must always be triggered manually.

Finding the failed e2e step

On the Prow job page, look at the step list on the left. The e2e pipeline runs in this order:

1. create-guests — Creates hosted clusters in parallel
2. run-tests — Runs Ginkgo test suites against the clusters
3. dump-guests — Collects diagnostic artifacts (always runs)
4. destroy-guests — Tears down clusters

Find the step that failed and click it to see the log.

create-guests failures

A hosted cluster failed to come up. To find out why:

1. Open the Artifacts tab and look for junit_hosted_cluster_*.xml files. These contain the HostedCluster and NodePool conditions at the time of failure.
2. If no JUnit file exists, the failure happened before the cluster reached the version rollout phase — check the create-guests step log directly.

Common causes:

After identifying the error, check the job history to determine if this is specific to your PR.

run-tests failures

A test assertion failed. To find which test:

1. Open the Artifacts tab and look for JUnit XML files (e.g., junit_self_managed_azure_public.xml). The failed test name and assertion message are in the XML.
2. Alternatively, search the run-tests step log for [FAIL] to find the Ginkgo failure output, which includes the test description, the failed assertion, and the source file and line number.

After identifying the failing test, check the job history to determine if this is specific to your PR.

For more details on reading test artifacts and Ginkgo output, see Debugging CI Failures.

destroy-guests failures

Cluster teardown failed. This is rarely caused by your PR — it usually means a cloud API issue or a resource stuck in a deleting state. /retest is almost always the right move.

Konflux failures

Konflux checks appear as Red Hat Konflux / <component>-on-pull-request or Red Hat Konflux / enterprise-contract-*. Click Details to open the Konflux pipeline run in the Konflux UI.

Common causes:

• Compilation error — Same as images failures. Your code doesn't compile. Fix and push.
• Stale or retired Tekton pipeline images — Konflux pipelines reference specific Tekton task images that get retired over time. When this happens, every PR fails on Konflux until the pipeline definitions are updated. The fix is to update the Tekton pipelines in the repo, merge that fix, and then rebase your PR onto the updated main branch.
• Enterprise contract failures — Policy violations on image provenance or signing. Usually not your code — /retest once.
• If the failure persists or you're unsure, escalate.

Checking Prow job history

This is the key step for determining whether a failure is your fault. Check whether the same job is failing on other PRs.

Non-e2e job history

E2e job history

Look at the last 10-20 runs:

• Mostly red → The job is failing for everyone. It's not your code. 🚨 Escalate.
• Mostly green, yours is red → The failure is likely related to your change. Continue to tracing the failure.
• Mixed → Could be a flaky test. Check if the same test is failing in the red runs.

Tracing the failure to your change

If the failure appears specific to your PR:

1. Find the test file. Search for the failing test name in test/e2e/:

   grep -r "test description from the failure" test/e2e/
   

2. Read the test. Understand what it's checking and what code paths it exercises.

3. Trace back to your changes. Common relationships:

   • Changed a controller → tests that verify that controller's behavior
   • Changed API types → tests that create or validate those resources
   • Changed a CPO component → compliance tests that check all control plane workloads

If your change clearly relates to the test, fix and push — the test is catching a real problem.

If it's not obvious, /retest once. If the same test fails again, escalate.

🚨 Escalate

If you've already retested and got the same failure, don't keep retesting — escalate.

Post in #forum-ocp-hypershift and tag @hypershift-engineering-ic with:

• The Prow job link
• The failing test or job name
• A one-line summary of what your PR changes
• Whether /retest produced the same failure

Further reading

• Debugging CI Failures — Reading JUnit XML, Ginkgo output, and dump-guests artifacts
• V2 E2E Testing Overview — Architecture of the v2 test framework
• CI Pipeline Configuration — How presubmit jobs are configured
• Daily CI Health — Monitoring periodic and presubmit job health