After the Power Outage: An Image Cache for OpenShift with Zot

While recovering several OKD clusters after a power outage, I found images that CRI-O listed as present but that the node could not use. On one node, pulling an image again by digest brought OVN and Multus back. That left me with a question for the next recovery: what happens when several nodes need to pull the same images at once?

I described the incident in OpenShift After a Power Outage: The Images Were There, but They Weren’t. The diagnostic and repair commands are in Recovering CRI-O Image Storage After a Power Outage.

I set up Zot as a pull-through cache for OCI images on a RHEL server outside the clusters. The first pull of an uncached image still depends on the upstream registry. Subsequent pulls can reuse the internal copy across nodes and clusters. The cache provides another way to retrieve images during recovery. Any inconsistencies in CRI-O’s local storage still need to be repaired on the affected nodes.

Server, node, context, and internal domain names are fictional. Public registry names have been kept.

A shared registry outside the clusters

I installed Zot 2.1.23 on RHEL 9.7 x86_64 as a native systemd service. Nodes connect to registry.example.net:5000 over HTTPS. The service name is a DNS alias for the RHEL host, and the certificate is signed by a CA the nodes trust.

Workload manifests keep their original image references. For example, when a pod asks for docker.io/library/busybox:1.36, CRI-O tries registry.example.net:5000/docker.io/library/busybox:1.36. The docker.io prefix tells Zot which upstream registry to contact if the image is missing from the cache. If the mirror pull fails, CRI-O can try the upstream registry directly.

I configured the same scheme for eleven registries: docker.io, quay.io, registry.k8s.io, ghcr.io, docker.elastic.co, gcr.io, eu.gcr.io, us.gcr.io, asia.gcr.io, europe-docker.pkg.dev, mcr.microsoft.com.

NFS storage and Zot startup

The cache lives on a dedicated NFS 4.1 mount. Here is the corresponding /etc/fstab entry with example names:

nfs-cache.example.net:/exports/oci-cache /srv/zot-cache nfs vers=4.1,hard,timeo=600,rsize=262144,wsize=262144,_netdev 0 0

sudo mkdir -p /srv/zot-cache
sudo mount /srv/zot-cache
findmnt --mountpoint /srv/zot-cache

The exported directory must be writable by the user that runs Zot. I also tied the service to the mount. If NFS is unavailable at startup, Zot should stay stopped. Otherwise, it could start filling the server’s disk with a second cache in the directory beneath the mount.

This unit excerpt uses the example path. The .mount unit name changes with the mount point.

[Unit]
RequiresMountsFor=/srv/zot-cache
BindsTo=srv-zot-cache.mount
After=srv-zot-cache.mount
ConditionPathIsMountPoint=/srv/zot-cache

Configuring on-demand pulls

The executable is at /usr/bin/zot and reads /etc/zot/config.json. This excerpt shows storage, TLS, and one of the eleven synchronization rules. For Docker Hub, manifests use docker.io while Zot contacts registry-1.docker.io.

{
  "storage": {
    "rootDirectory": "/srv/zot-cache"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000",
    "tls": {
      "cert": "/etc/zot/tls/registry.crt",
      "key": "/etc/zot/tls/registry.key"
    }
  },
  "extensions": {
    "sync": {
      "enable": true,
      "registries": [
        {
          "urls": [
            "https://registry-1.docker.io"
          ],
          "onDemand": true,
          "tlsVerify": true,
          "content": [
            {
              "prefix": "**",
              "destination": "/docker.io"
            }
          ]
        }
      ]
    }
  }
}

With zot.service and the certificates in place, I checked that the service started and the OCI endpoint responded over HTTPS.

sudo systemctl daemon-reload
sudo systemctl enable --now zot
sudo systemctl status zot

curl --noproxy '*' -fsS -o /dev/null -w '%{http_code}
'   https://registry.example.net:5000/v2/

In my setup, /v2/ returned 200. This verified HTTPS connectivity and certificate trust from the test client. Checking upstream synchronization required an image pull.

Redirecting OpenShift image pulls

ImageDigestMirrorSet covers references by digest (@sha256:…), and ImageTagMirrorSet covers references by tag (:1.36). I applied both with the same source-to-mirror mapping. This shortened manifest shows the Docker Hub entry.

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  name: zot
spec:
  imageDigestMirrors:
    - source: docker.io
      mirrors:
        - registry.example.net:5000/docker.io
      mirrorSourcePolicy: AllowContactingSource
---
apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
  name: zot
spec:
  imageTagMirrors:
    - source: docker.io
      mirrors:
        - registry.example.net:5000/docker.io
      mirrorSourcePolicy: AllowContactingSource

I kept the manifests in Git. After applying them, I checked that the Machine Config Operator had propagated the change to the nodes. I also reviewed the existing mirror resources because part of the environment already had Nexus rules.

CTX=cluster-dev

oc --context="$CTX" apply --dry-run=server -f zot-mirrors.yaml
oc --context="$CTX" apply -f zot-mirrors.yaml
oc --context="$CTX" get imagedigestmirrorsets,imagetagmirrorsets
oc --context="$CTX" get mcp
oc --context="$CTX" get nodes

Verifying that the pull went through Zot

With AllowContactingSource, a successful crictl pull could mean CRI-O pulled directly from upstream. I ran the test from a node using the public image name and watched the Zot logs at the same time.

oc --context=cluster-dev debug node/worker-01

Inside the debug session:

chroot /host

curl --noproxy '*' -fsS -o /dev/null -w '%{http_code}
'   https://registry.example.net:5000/v2/

grep -n 'registry.example.net' /etc/containers/registries.conf
crictl pull docker.io/library/busybox:1.36

At the same time, on the RHEL server:

sudo journalctl -u zot -f

The Zot logs confirmed that the request from CRI-O reached the mirror and showed on-demand synchronization when content was missing. In the first rollout phase, I validated the integration in development and preproduction, with nodes Ready and the MachineConfigPool updates complete. I also removed the old Nexus rules in development. The production manifests had passed validation and still needed to be applied. The integration still had to be verified in the other two clusters.

What changes on the next startup

If a node needs to pull an image that Zot already holds, it can fetch it over the internal network even if its local copy is unusable. If the image is missing from the cache, Zot needs access to the upstream registry. If Zot or its NFS storage is unavailable, the node can still pull directly as long as it has connectivity and the upstream registry responds.

The check I took away from the incident is straightforward: to verify that a critical image can be recovered through the cache, pull it from CRI-O using its original reference and confirm that Zot served the request. A healthy registry endpoint or a successful pull on its own only verifies part of that path.

Documentation: Zot 2.1.23 synchronization and mirroring and OpenShift image mirror configuration.