Pod disruption budgets: the Kubernetes setting that decides whether a node upgrade breaks your quorum

One of the scenarios we put to DevOps candidates has a RabbitMQ quorum queue losing its leader during a rolling Kubernetes node upgrade. The strongest answers ask, early, whether the pod disruption budget allowed more than one broker down at once. That question deserves its own explanation, because the setting is easy to misunderstand in both directions.

Voluntary and involuntary disruptions

Kubernetes separates disruptions into two kinds [1]. Involuntary disruptions are failures nobody chose: a hardware failure, a kernel panic, a cloud provider losing a VM, a node partitioned off the network [1]. Voluntary disruptions are actions an application owner or cluster administrator takes — including draining a node for repair or upgrade, or draining nodes to scale the cluster down [1].

What a PodDisruptionBudget does

A PodDisruptionBudget limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions [1]. The Kubernetes documentation's own example is a quorum-based application that needs the number of running replicas never to fall below the number required for a quorum [1].

It only works if the tooling asks permission. Cluster managers and hosting providers should use tools that respect budgets by calling the Eviction API rather than deleting pods directly [1]. `kubectl drain` does: it tries to evict every pod on the node, an eviction that cannot proceed may be temporarily rejected, and drain keeps retrying until the pods are gone or a timeout is reached [1].

What it cannot do

A budget cannot prevent involuntary disruptions — but those disruptions do count against it [1]. That combination is the trap. If one broker has already failed, the budget may already be spent, and a node drain that would normally be safe can stall waiting for capacity that is not coming back.

Setting one for a quorum system

For multi-instance stateful applications such as ZooKeeper or etcd, the concern the documentation names is not reducing the number of instances below quorum, otherwise writes fail [2]. It offers two approaches [2]:

• `maxUnavailable: 1` — works as the application scales up or down.
• `minAvailable` set to the quorum size — for example 3 when running 5 — which allows more disruptions at once.

A single budget can specify only one of `maxUnavailable` and `minAvailable` [2].

The wider point

Node upgrades are routine, which is exactly why they cause outages: the procedure feels safe, so nobody re-checks the budget against the current state of the cluster. A pod disruption budget is a promise about voluntary change. It says nothing about what has already broken.