Migrating Portainer Agent to Edge Agent Standard across several OpenShift/OKD clusters exposed a chain of independent issues: filesystem permissions, immutable Kubernetes Service fields, DNS resolution, incomplete credentials, and deployment automation. This technical report documents the observed symptoms, confirmed causes, and corrective actions.

Note: this is an anonymized real-world technical case. Organization, cluster, node, domain, IP address, environment ID, and credential details have been removed. All examples use generic names and must be adapted before running commands.

1. Background: from the conventional Agent to Edge Agent

The original architecture used Portainer Community Edition with a conventional agent deployed in each cluster. Each agent was exposed through a Kubernetes LoadBalancer Service backed by MetalLB, and the central server accessed its API over TCP port 9001.

Portainer Server -- TCP 9001 -- LoadBalancer -- Portainer Agent -- Kubernetes API

The target architecture used Edge Agent Standard, which reverses the direction in which connections are established. The agent initiates outbound HTTPS polling to the Portainer API and opens the Edge tunnel over TCP port 8000 when required. Each agent no longer needs its own externally exposed load balancer.

OpenShift / Edge Agent -- HTTPS ----> Portainer API
                       -- TCP 8000 -> Portainer Edge Tunnel

2. Initial failure: insufficient permissions to generate certificates

Early tests with the conventional agent produced a filesystem error:

unable to generate self-signed certificates
open cert.pem: permission denied

OpenShift enforces container security policies through Security Context Constraints (SCC). These policies may prevent containers from running as root and restrict writes to paths inside the container image. The agent attempted to create a certificate in a location that was not writable by its effective user.

Diagnosis: a ServiceAccount's Kubernetes RBAC privileges are entirely separate from the permissions of the process inside the container. Even a Kubernetes administrative role cannot fix a filesystem permission denied error.

The deployment was therefore adapted to support unprivileged execution rather than permanently broadening the container's security privileges.

3. Migration failure: an immutable ClusterIP

Applying the new Edge Agent manifest to the existing Service failed with:

spec.clusterIPs[0]: Invalid value: ["None"]:
may not change once set

The old Service had an allocated ClusterIP. The new agent needed a headless Service using clusterIP: None, but Kubernetes does not allow this field to be changed on an existing Service.

Fix: introduce a controlled migration step into the deployment procedure. The automation checks the existing Service, identifies the legacy LoadBalancer, and deletes and recreates only that specific Service when necessary. It does not remove the namespace or the Portainer server.

apiVersion: v1
kind: Service
metadata:
  name: portainer-agent
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: portainer-agent

Recreating a Service can cause a brief interruption and should be performed as a controlled maintenance operation. Subsequent deployments reuse the headless Service.

4. DNS resolution failure during startup

After replacing the Service, the agent logged name-resolution failures:

lookup portainer-agent on <dns-server>:53:
no such host

The agent uses the Service name for member discovery. A headless Service may have no published endpoints while none of its pods are considered Ready. This creates a startup dependency loop: the agent needs DNS discovery to initialize, but the endpoints are not yet published because initialization has not completed.

Fix: set publishNotReadyAddresses: true to make the pods discoverable while they initialize. In this deployment, that change resolved the DNS startup failure.

5. Empty Edge credentials and the secondary port 80 error

Another deployment failed with an error that initially resembled a privileged-port restriction:

Unable to start Edge server
listen tcp 0.0.0.0:80: bind: permission denied

Inspecting the Kubernetes Secret showed that EDGE_KEY was zero bytes long. The agent had not received the association key generated by Portainer. In this configuration, it then attempted to start a server on port 80; the permission error was a secondary symptom of an incorrectly initialized agent.

The fix was to provide the correct, non-empty Edge key and verify that the expected startup sequence completed. Granting the container extra capabilities to bind privileged ports was unnecessary.

Credentials were centralized in HashiCorp Vault, with one record per environment containing:

The deployment process reads both values from Vault, creates a ConfigMap for non-sensitive settings, and stores the key in a Kubernetes Secret. The Deployment refers to the key using secretKeyRef:

env:
  - name: EDGE_KEY
    valueFrom:
      secretKeyRef:
        name: portainer-agent-edge-key
        key: edge.key

Deployment now fails explicitly if either value is missing or empty. Credentials are not printed, committed to Git, or included in diagnostic logs. Their presence can be checked without revealing their contents.

6. An unwritable /tmp directory

With a valid key, the agent could start its API and establish some communication. However, internal features still failed:

failed to create healthy file:
open /tmp/.portainer-healthy: permission denied

failed to create alert rule evaluator:
open TSDB: mkdir /tmp/portainer-eval-tsdb-...:
permission denied

The container was running as an unprivileged user. The existing /data volume was writable, but the agent also needed /tmp for health files, metrics, and temporary storage. That directory was not writable under the OpenShift security context being enforced.

Fix: mount separate emptyDir volumes at /data and /tmp, while keeping an unprivileged security context:

spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
      containers:
        - name: portainer-agent
          volumeMounts:
            - name: agent-data
              mountPath: /data
            - name: agent-tmp
              mountPath: /tmp
      volumes:
        - name: agent-data
          emptyDir: {}
        - name: agent-tmp
          emptyDir: {}

emptyDir volumes are ephemeral: their contents are lost when the pod is deleted. This matters if agent features requiring persistence are enabled. In this case, the mounts resolved the write failures without running the container as root.

7. Readiness: internal HTTP is not the Edge connection

The Deployment checked the agent's internal API via /ping on port 9001:

readinessProbe:
  httpGet:
    path: /ping
    port: 9001
    scheme: HTTP
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3

In the tested configuration, the internal API used HTTP. This is distinct from the external Edge Agent communications using HTTPS and the TCP 8000 tunnel.

Three states must be distinguished: Running means the main container process is executing; Ready means its configured local readiness probe passes; and Connected means Portainer receives the required traffic and can manage the environment. Neither Running nor Ready alone proves the Edge connection works.

8. Deployment automation: duplicated Makefile targets

The migration also exposed GNU Make warnings:

warning: overriding recipe for target 'configure'
warning: ignoring old recipe for target 'configure'

The Makefile contained duplicate recipes for targets including configure and deploy. GNU Make retains the last recipe for a target and overrides earlier ones. As a result, a deployment could follow an obsolete sequence that skipped loading the Edge credentials from Vault.

The Makefile was consolidated so that each operation had a single definition. The corrected flow prepares the namespace, retrieves credentials, validates the ConfigMap and Secret, migrates the Service when needed, applies the manifests, and verifies the rollout.

deploy: configure check-config

Error handling was also tightened so failures when reading Vault or applying Kubernetes resources could not be hidden by command chains or pipelines. Because the shared configuration was referenced through Git submodules, each cluster's pinned revision had to be checked: updating the shared repository does not automatically update every submodule.

9. Verifying the fix

Validation covered Kubernetes, the application, and network connectivity. First, the Deployment, pods, and logs were checked:

oc -n portainer get deployment portainer-agent
oc -n portainer get pods -l app=portainer-agent -o wide
oc -n portainer logs deployment/portainer-agent --tail=100

Successful initialization included messages similar to:

edge key loaded from options
Agent info updated successfully
evaluator successfully initialized

Checks confirmed that the agent was no longer repeatedly restarting and that persistent /tmp permission errors were gone. Outbound connectivity to the Portainer HTTPS endpoint and TCP port 8000 was tested from a network representative of the pods.

During rolling updates, member-discovery warnings may temporarily refer to old terminating pods. Those events must be distinguished from a persistent failure in the new pod. Likewise, a firewall or routing problem should not be declared the root cause without reproducible network evidence.

The decisive test was performed in the Portainer UI: the environments appeared available, their status updated, and Kubernetes resources could be accessed. A healthy Deployment is not a substitute for an end-to-end functional test in Portainer.

10. Outcome and technical lessons

The migration did not fail for one reason. Filesystem permissions, Service design, DNS discovery, secret management, and automation defects overlapped. Resolving the incident required separating symptoms from root causes, correcting each component, and introducing repeatable checks.

Key lesson: repeatedly restarting a Deployment will not fix an empty Secret, an immutable field, an unwritable directory, or an incorrect deployment recipe. The diagnosis must end with a functional application-level check, not merely a pod reporting Running.

References

• Portainer — Edge Agent
• Portainer — Add a Kubernetes Edge Agent environment
• Kubernetes — Services
• Kubernetes — Volumes and emptyDir
• Kubernetes — Security Context
• HashiCorp Vault — KV Get