The short answer
When a quorum queue loses its leader during a rolling upgrade, leadership re-election is expected behaviour and not the incident. The actual bug is usually the dashboard reporting the cluster healthy. A senior answer separates those two failures before touching anything; a mid answer restarts the pod first.
This is one of the scenarios we put to DevOps candidates, published in full on the role page. It is worth walking through in public, because the gap between a strong answer and an adequate one is unusually legible here — and because it shows what we mean when we say a recruiter cannot run this conversation.
A RabbitMQ quorum queue loses its leader during a rolling Kubernetes node upgrade. Publishers keep accepting writes, consumer lag climbs, and the on-call dashboard shows the cluster as healthy. Walk me through the first ten minutes.
The trap in the question
There are two failures described, and only one of them is a failure. A quorum queue losing its leader during a rolling node upgrade is expected — that is what a rolling upgrade does, and Raft re-elects. The genuinely broken thing is the dashboard reporting healthy while consumer lag climbs.
Candidates who miss that separation start remediating the re-election, which is not a problem, and leave the monitoring defect in place, which is.
What a senior answer contains
- Asks what the quorum size and `cluster_partition_handling` are before touching anything. The correct first action depends on both, and a candidate who starts acting without asking is guessing on a production cluster.
- Separates the two failures explicitly — re-election expected, healthy dashboard the actual bug.
- Checks whether the pod disruption budget allowed more than one broker down at once. This is the root cause in a large share of real incidents of this shape, and it is a Kubernetes question rather than a RabbitMQ one.
- Knows that publisher confirms and consumer acknowledgements answer “did we lose messages” — and says so rather than speculating.
Where a mid answer stops
- Restarts the pod first and reasons afterwards. Under-quorum, this turns a recoverable re-election into genuine unavailability.
- Treats consumer lag as the incident rather than as a symptom of it.
- Cannot articulate what the cluster does when a quorum is lost versus merely reduced. These are materially different states with different correct responses.
The wider point
We publish these scenarios because publishing them is the only way to make a quality claim checkable. Any staffing firm can assert that its engineers are senior. Very few will show you the question they asked and the answer they expected, because doing so invites you to judge the bar.