Global Pull Secret for Hosted Control Planes

Overview

The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster.

The implementation uses a DaemonSet approach that automatically detects when you create an additional-pull-secret in the kube-system namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration.

Adding your Pull Secret

To use this functionality, follow these steps:

1. Create your additional pull secret

Create a secret named additional-pull-secret in the kube-system namespace of your Hosted Cluster (DataPlane). The secret must contain a valid DockerConfigJSON format:

apiVersion: v1
kind: Secret
metadata:
  name: additional-pull-secret
  namespace: kube-system
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded-docker-config-json>

2. Example DockerConfigJSON format

Your .dockerconfigjson should follow this structure:

{
  "auths": {
    "registry.example.com": {
      "auth": "base64-encoded-credentials"
    },
    "quay.io/mycompany": {
      "auth": "base64-encoded-credentials"
    }
  }
}

3. Apply the secret

kubectl apply -f additional-pull-secret.yaml

4. Verification

After creating the secret, the system will automatically:

1. Validate the secret format
2. Merge it with the original pull secret
3. Deploy a DaemonSet to all nodes
4. Update the kubelet configuration on each node

You can verify the deployment by checking:

# Check if the DaemonSet is running
kubectl get daemonset global-pull-secret-syncer -n kube-system

# Check the merged pull secret
kubectl get secret global-pull-secret -n kube-system

# Check DaemonSet pods
kubectl get pods -n kube-system -l name=global-pull-secret-syncer

How it works

The Global Pull Secret functionality operates through a multi-component system:

Automatic Detection

• The Hosted Cluster Config Operator (HCCO) continuously monitors the kube-system namespace
• When it detects the creation of additional-pull-secret, it triggers the reconciliation process

Validation and Merging

• The system validates that your secret contains a proper DockerConfigJSON format
• It retrieves the original pull secret from the HostedControlPlane
• Your additional pull secret is merged with the original one
• If there are conflicting registry entries, the original pull secret takes precedence (the additional pull secret entry is ignored for conflicting registries)
• The system supports namespace-specific registry entries (e.g., quay.io/namespace) for better credential specificity

Deployment Process

• A global-pull-secret is created in the kube-system namespace containing the merged result
• RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both kube-system and openshift-config namespaces
• We use Role and RoleBinding in both namespaces to access secrets in kube-system and openshift-config namespaces
• A DaemonSet named global-pull-secret-syncer is deployed to eligible nodes

Node-Level Synchronization

• Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace
• When changes are detected, it updates /var/lib/kubelet/config.json on the node
• The kubelet service is restarted via DBus to apply the new configuration
• If the restart fails after 3 attempts, the system rolls back the file changes

Automatic Cleanup

• If you delete the additional-pull-secret, the HCCO automatically removes the global-pull-secret secret
• The system reverts to using only the original pull secret from the HostedControlPlane
• The DaemonSet continues running but now syncs only the original pull secret to nodes

Registry Precedence and Conflict Resolution

The Global Pull Secret system uses a specific precedence model when merging your additional pull secret with the original one:

Merge Behavior

• Original pull secret entries always take precedence over additional pull secret entries for the same registry
• If both secrets contain an entry for quay.io, the original pull secret's credentials will be used
• Your additional pull secret entries are only added if they don't conflict with existing entries
• Warnings are logged when conflicts are detected

Recommended Approach

To avoid conflicts and ensure your credentials are used, consider these strategies:

1. Use namespace-specific entries: Instead of quay.io, use quay.io/your-namespace
2. Target specific registries: Add entries only for registries not already in the original pull secret
3. Check existing entries: Review what registries are already configured in the HostedControlPlane

Example Merge Scenario

Original Pull Secret:

{
  "auths": {
    "quay.io": {
      "auth": "original-credentials"
    }
  }
}

Your Additional Pull Secret:

{
  "auths": {
    "quay.io": {
      "auth": "your-credentials"
    },
    "quay.io/mycompany": {
      "auth": "your-namespace-credentials"
    }
  }
}

Resulting Merged Pull Secret:

{
  "auths": {
    "quay.io": {
      "auth": "original-credentials"
    },
    "quay.io/mycompany": {
      "auth": "your-namespace-credentials"
    }
  }
}

Note how the quay.io entry keeps the original credentials, but quay.io/mycompany is added from your additional secret.

Implementation details

The implementation consists of several key components working together:

Core Components

1. Global Pull Secret Controller (globalps package)
2. Handles validation of user-provided pull secrets
3. Manages the merging logic between original and additional pull secrets
4. Creates and manages RBAC resources
5. Deploys and manages the DaemonSet

6. Node eligibility assessment: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions

7. Sync Global Pull Secret Command (sync-global-pullsecret package)

8. Runs as a DaemonSet on each node
9. Watches for changes to the global-pull-secret in kube-system namespace
10. Accesses the original pull-secret in openshift-config namespace
11. Updates the kubelet configuration file

12. Manages kubelet service restarts via DBus

13. Hosted Cluster Config Operator Integration

14. Monitors for the presence of additional-pull-secret
15. Orchestrates the entire process
16. Handles cleanup when the secret is removed

Architecture Diagram

Key Features

• Security: Only watches specific secrets in kube-system and openshift-config namespaces
• Robustness: Includes automatic rollback in case of failures
• Efficiency
• Only updates when there are actual changes
• The globalPullSecret implementation has their own controller so it cannot interfere with the HCCO reonciliation
• Security considerations: Uses specific RBAC for only the required resources in each namespace. The DaemonSet containers run in privileged mode due to the need to:
• Write to /var/lib/kubelet/config.json (kubelet configuration file)
• Connect to systemd via DBus for service management
• Restart kubelet.service, which requires root privileges
• Smart node targeting: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts

InPlace NodePool Handling

To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting:

Node Labeling Process

1. MachineSets Discovery: The controller queries the management cluster for MachineSets with InPlace-specific annotations (hypershift.openshift.io/nodePoolTargetConfigVersion)
2. Machine Enumeration: For each InPlace MachineSets, it lists all associated Machines
3. Node Identification: Maps Machine objects to their corresponding nodes via machine.Status.NodeRef.Name
4. Labeling: Applies hypershift.openshift.io/nodepool-inplace-strategy=true label to identified nodes

DaemonSet Scheduling Configuration

The DaemonSet uses NodeAffinity to exclude InPlace nodes:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: hypershift.openshift.io/nodepool-inplace-strategy
                operator: DoesNotExist

This ensures that: - Nodes without the label: ✅ Are eligible for DaemonSet scheduling - Nodes with the label (any value): ❌ Are excluded from DaemonSet scheduling

Conflict Prevention Benefits

• Prevents MCD failures: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades
• Maintains upgrade reliability: InPlace upgrade processes are not interrupted by Global Pull Secret modifications
• Automatic detection: No manual intervention required - the system automatically identifies and handles InPlace nodes

Error Handling

The system includes comprehensive error handling:

• Validation errors: Invalid DockerConfigJSON format is caught early
• Restart failures: If kubelet restart fails after 3 attempts, the file is rolled back
• Resource cleanup: If the additional pull secret is deleted, the HCCO automatically removes the globalPullSecret

This implementation provides a secure, autonomous solution that allows HostedCluster administrators to add private registry credentials without requiring Management Cluster administrator intervention.