A real incident on OKD: blocked processes, orphaned agents, cgroups, pipelines that must not be resumed, and an unexpected problem when removing the recovery safeguards.
Just Another Friday
It all started when a worker in an OKD cluster stopped responding properly.
The cluster, running OKD 4.22, had three control-plane nodes and three workers. One of the workers, with 16 CPUs and 32 GiB of memory, had accumulated an extraordinary number of blocked processes. Its load average was approaching 900, I/O activity had surged, and Kubernetes had eventually marked the node NotReady.
Several persistent services shared that worker: Git repositories, databases, an artifact manager, monitoring services, Jenkins, and others.
The priority was not simply to reboot the machine. We needed to understand what was causing the blockage and recover the node without putting the persistent volumes at risk.
The investigation eventually revealed several related, but technically distinct, problems:
- Ephemeral Jenkins agents that had become stuck while terminating.
- A storm of processes and I/O operations that prevented the worker from functioning normally.
- Interrupted pipelines that Jenkins would try to recover automatically.
- A concurrency configuration that was too permissive for the available capacity.
- A deadlock between pipelines caused by reducing that concurrency too far.
- A leftover initialization file that prevented Jenkins from starting during the final cleanup.
What made the incident interesting was not any single error, but how every recovery decision required a better understanding of how Kubernetes and Jenkins behave together.
1. A Node With Almost 900 Blocked Processes
The initial symptom was a completely overwhelmed worker.
The usual checks showed the scale of the problem:
oc get nodes
oc adm top nodes
ssh core@worker-a '
uptime
free -h
vmstat 1 3
'
The load average had reached nearly 900. Hundreds of tasks were blocked, disk activity was abnormally high, and very little memory was available.
It is important to distinguish load average from CPU utilization. On Linux, load average also includes tasks waiting under certain uninterruptible I/O conditions. A system can have an enormous load average without its CPUs continuously executing useful code.
In this case, the blocked processes and I/O pressure pointed to something more serious than a build merely consuming too much CPU.
The first measure was to prevent Kubernetes from scheduling new workloads on the node:
oc adm cordon worker-a
That did not solve the existing problem, but it prevented additional work from being assigned while we investigated.
Jenkins Agents That Would Not Terminate
While inspecting the worker's workloads, we found several Jenkins pods in the Terminating state.
These were ephemeral agents used to run build pipelines. Some had been trying to disappear for far too long, and their processes were still associated with the node's cgroups.
An important detail was that several agents were classified as BestEffort: they had neither resource requests nor limits defined.
In a shared cluster, this is particularly risky. A build agent can consume substantial memory, CPU, and I/O without an explicit reservation to help the scheduler plan its placement, or a memory limit to cap its consumption.
The investigation then moved down from Kubernetes objects to the operating system's cgroups.
The affected processes were not under the burstable tree, but under besteffort. That explained why an initial search had failed to locate some of the problematic agents.
On the node, we identified the specific cgroup belonging to one of the stuck pods. We used its cgroup.kill interface to terminate the processes in that group.
The response was immediate.
The number of blocked processes fell from several hundred to single digits. I/O pressure returned to normal, the node became responsive again, and it eventually returned to Ready.
There was no need to reboot the worker.
To be precise, this intervention confirmed that the processes in that cgroup were directly involved in the blockage. It does not, by itself, establish what original failure triggered all the I/O activity.
Nor does it make cgroup.kill a generic solution for stuck pods. It is a low-level operation that requires unambiguous identification of the cgroup and an understanding of the consequences of terminating its processes.
2. Recovering the Node Was Not Enough
The worker was functioning again, but Jenkins still needed attention.
The controller retained information about numerous executions interrupted during the incident. If started normally, it might try to resume a collection of old pipelines and create Kubernetes agents all over again.
To avoid that, we temporarily stopped the Jenkins instances by scaling their StatefulSets down to zero replicas, while preserving their PVCs.
The recovery strategy had two goals:
- Start the controller so we could inspect its state and configuration.
- Prevent it from immediately running the queued work.
Quiet Down From Startup
Jenkins provides a Quiet Down mode, which prevents new builds from starting while allowing existing ones to finish.
To ensure the controller entered this state from the very beginning, we temporarily mounted a Groovy script in its initialization directory:
import jenkins.model.Jenkins
Jenkins.get().doQuietDown()
println("RECOVERY: Jenkins started in Quiet Down mode")
The idea was to let Jenkins start and load its configuration without triggering new builds.
However, Quiet Down alone does not solve the problem of interrupted pipelines that the execution engine attempts to restore.
Preventing Old Executions From Resuming
We temporarily used the following JVM property of the Pipeline component:
-Dorg.jenkinsci.plugins.workflow.job.WorkflowRun.maxResumptionAge=1s
With such a short threshold, Jenkins rejected the resumption of old executions.
The logs confirmed the behavior:
Refusing to resume running build ...
because it is too old and might be corrupt
This distinction matters: we were not merely pausing pipelines; we were preventing their recovery.
It was a deliberate emergency measure, not an appropriate setting for normal Jenkins operation.
The controller eventually started successfully, in Quiet Down, with no active executions.
For the first time since the incident, we had a stable environment to work with.
3. The Problem With Agents Without Resource Limits
The next question was obvious: how could we prevent another burst of builds from overwhelming the worker again?
The Jenkins Kubernetes Cloud configuration allowed up to ten concurrent agents.
In addition, the agents did not have a consistent resource policy. Jenkins used a global template for the connection container (jnlp), while other containers were defined dynamically through a shared pipeline library.
This separation had important consequences.
The Global Template Was Not Enough
In the Jenkins interface, we configured resources for the jnlp container:
| Resource | Request | Limit |
|---|---|---|
| CPU | 100m | 500m |
| Memory | 256Mi | 1Gi |
But changing that template did not guarantee that the build containers themselves had limits.
The pipelines used different Java versions and specialized tools. Maven containers, for example, were assembled by a Groovy function in the shared library.
We had to change the actual source of the YAML definitions.
Limits in the Shared Library
For the Maven containers, both Java 8 and Java 17, we set:
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
For the static analysis container:
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
A typical agent with Maven and JNLP therefore had the following aggregate resources:
| Resource | Total request | Total limit |
|---|---|---|
| CPU | 600m | 2500m |
| Memory | 1280Mi | 5Gi |
The distinction between requests and limits is essential.
Requests influence Kubernetes scheduling decisions. Limits cap permitted consumption, although the precise behavior depends on the resource: exceeding a memory limit can cause an OOMKill, while CPU usage is normally constrained through throttling.
In our case, memory limits were especially important to prevent a build from consuming the node's memory uncontrollably.
The change was published to the shared library's repository and subsequently verified on the actual pods.
Kubernetes showed exactly the expected resources for both Maven and JNLP containers.
4. A Concurrency Limit of One Looked Safe… Until It Wasn't
For the first test, we reduced the global Kubernetes Cloud limit from ten agents to one.
The intention was reasonable: validate the whole process without allowing Jenkins to create multiple build pods simultaneously.
We selected a controlled run of a Java project that used different stages with Java 8 and Java 17.
The build began to progress.
Pods appeared and disappeared as Jenkins moved between stages, some with final states that initially looked concerning.
However, the pipeline continued, and the new resource limits were being applied correctly.
The real problem appeared later.
A Pipeline Waiting for Another Pipeline
During one stage, the main pipeline invoked a second job to perform a deployment:
build job: 'deploy-application', wait: true
That second job also needed a Kubernetes agent.
The logs showed:
[Pipeline] podTemplate
[Pipeline] node
Still waiting to schedule task
All nodes of label 'deploy-agent-...' are offline
The main pipeline was still occupying an agent while waiting for the second job. But Kubernetes Cloud was allowed to run only one agent at a time.
We had created a circular wait:
Main pipeline
|
| occupies the only permitted agent
|
+-- launches deployment pipeline
|
| requires another agent
|
+-- blocked by the global limit
This was not a Kubernetes failure. It was a consequence of combining the pipeline's structure with the concurrency policy.
From One Agent to Two
The solution was to increase the global limit to two.
The second agent was created immediately, and the deployment was able to continue.
Both pods ran simultaneously, with their CPU and memory limits correctly applied.
The final result was satisfactory:
| Execution | Result | Duration |
|---|---|---|
| Main Java pipeline | SUCCESS | 11 minutes |
| Deployment pipeline | SUCCESS | 6 min 49 s |
The test confirmed that the new limits allowed the complete process to run and that, for this particular pipeline structure, a concurrency limit of one was not viable.
The minimum concurrency level depends not only on how many builds we want to allow, but also on how many simultaneous agents a single logical execution may require.
5. Why Did Kubernetes Always Choose the Same Worker?
Another question arose during testing.
All agents were running on the same worker that hosted the Jenkins controller.
Our first hypothesis was that the volumes used by the agents tied them to that node.
The pods mounted shared PVCs for the Maven repository and other build data.
However, inspection showed that both volumes used the ReadWriteMany (RWX) access mode.
Therefore, the usual ReadWriteOnce restriction preventing simultaneous access from other nodes did not apply.
Nor was there a nodeSelector in the template pinning agents to the controller's worker. The only requirement was Linux.
The memory distribution observed during the test was approximately:
| Worker | Memory used |
|---|---|
| Worker A | 23% |
| Worker B | 82% |
| Worker C | 76% |
It was consistent with the scheduler preferring the least-loaded worker, although Kubernetes schedules primarily according to requested resources and other constraints, not simply the memory utilization percentage reported by oc adm top.
We did not establish any automatic affinity between Jenkins and its agents.
Agent placement remained an open improvement: investigate affinity rules, spreading, or scheduling policies that avoid excessive concentration without unnecessarily reducing resource utilization.
6. The Unexpected Problem When Removing the Temporary Configuration
With both pipelines completed successfully and the worker stable, it was time for the final step: removing the recovery safeguards.
We needed to remove the quiet-start.groovy script mount and the JVM property that prevented old pipelines from resuming from the StatefulSet.
We prepared a controlled StatefulSet update using its RollingUpdate strategy's partition, separating the template modification from the moment of restart.
After removing both settings, we restarted Jenkins.
And Jenkins stopped starting.
CrashLoopBackOff After an Apparently Correct Cleanup
The pod began entering CrashLoopBackOff.
Startup probes returned HTTP 503 errors and, in some attempts, connection refusals.
The first log excerpts showed exceptions from internal components and the plugin system during shutdown.
But those exceptions were not the root cause.
Retrieving the complete log from a failed attempt revealed the relevant error:
Failed GroovyInitScript.init
/var/jenkins_home/init.groovy.d/quiet-start.groovy
can not be read
Jenkins was still detecting the initialization file even though it was no longer mounted from the ConfigMap.
Inspecting the persistent volume, we found:
-rwxrw---- root jenkins 0 quiet-start.groovy
It was an empty regular file, zero bytes long, left behind in the init.groovy.d directory.
The original script had been introduced through a ConfigMap mount using subPath. When the mount was removed, a residual file remained on persistent storage.
The log indicated that Jenkins could not read it. Although the file had an unusual combination of owner and permissions, those details alone did not establish the exact reason for the read failure. What we did verify was that moving it out of the directory fixed the startup problem.
Recovery From a Maintenance Pod
There was no reason to reinstall Jenkins or change its plugins.
The problem was isolated to a single file on the PVC.
We scaled the StatefulSet down to zero and created a temporary pod that mounted the same persistent volume without starting Jenkins.
From that pod, we inspected the file and moved it out of init.groovy.d, preserving it under a different name.
We then removed the maintenance pod and scaled Jenkins back up to one replica.
This time, it started correctly.
The controller was 1/1 Running, with no restarts, Quiet Down disabled, and the two-agent limit preserved.
It was a useful reminder that removing a Kubernetes mount does not necessarily remove files left behind in the volume's filesystem.
7. The Final Check: Helm and Persistent Configuration
Functional recovery was complete, but one question remained: would the new concurrency limit survive an operator deployment?
We ran the usual deployment through Helm and checked the active Kubernetes Cloud configuration again.
The result was:
Kubernetes Cloud: Kubernetes
Concurrency Limit: 2
Quiet Down: false
Queue: 0
The configuration had been preserved.
However, a review of the chart's values revealed a detail: the two-agent limit was not declared in the deployment repository.
The chart had a default of ten agents, but automatic Jenkins Configuration as Code configuration was disabled.
The new value therefore survived because Jenkins stored its configuration on the persistent volume.
That is sufficient for a deployment over the existing installation, but it does not guarantee that a fresh installation will produce exactly the same result.
Moving that configuration to a declarative source remains outstanding, without accidentally overwriting the other Kubernetes Cloud settings that Jenkins currently maintains.
Simply adding a parameter to values.yaml is not always enough. First, we need to establish whether the chart actually applies that parameter to the service's effective configuration.
8. What We Learned
I took away several technical lessons from this incident.
Pods are not the whole story. When Kubernetes shows agents in Terminating, their associated processes may still be alive in the operating system. Understanding cgroups makes it possible to investigate situations that the Kubernetes API alone cannot explain.
Resource limits must be defined where containers actually originate. In Jenkins, a global template can coexist with dynamic templates generated by shared libraries. Configuring one does not mean that all the others are configured.
Lower concurrency does not always mean better configuration. A pipeline that launches another job and waits for its result may need several agents at once. Reducing concurrency without reviewing those dependencies can introduce deadlocks that did not exist before.
Jenkins recovery requires distinguishing new executions from old ones. Quiet Down prevents jobs from starting, but it is not the same as preventing interrupted pipelines from being restored. They are different mechanisms with different consequences.
Shutdown errors are not always the cause of the problem. During CrashLoopBackOff, several exceptions appeared as Jenkins shut down. The original cause lay a few seconds earlier, during the execution of a Groovy initialization script.
Configuration that survives a deployment is not necessarily reproducible. A PVC can preserve a change made through the interface perfectly well, even if that change is not declared in Helm.
Outcome
The worker was recovered without restarting the operating system. Jenkins returned to service, a complete Java build and its associated deployment were validated, and the agents gained explicit CPU and memory limits.
The global concurrency limit was reduced from ten agents to two, which allowed the tested workflow to complete without overwhelming the worker again.
The temporary recovery safeguards were also removed successfully, after resolving the leftover initialization-file problem.
There is still work to do, particularly on agent placement and declarative Kubernetes Cloud configuration.
But the main objective was achieved: restore service and reduce the risk of another Jenkins run compromising the node.
Perhaps the most useful lesson is that, in systems combining Kubernetes, persistent storage, and dynamic pipelines, recovery does not end when all the pods turn green. It ends when we understand what caused them to fail, which mechanisms we changed to bring them back, and what will happen the next time the system has to start from scratch.