RabbitMQ Khepri is now the default metadata store: what changes when a majority of nodes is gone

Every RabbitMQ cluster keeps a metadata store: the database that records everything except queue and stream messages, including users, virtual hosts, topology, runtime parameters and policies [1]. Until 3.13.x that store was always Mnesia [1]. It is now Khepri.

The version line

• Khepri is a fully supported metadata store starting with RabbitMQ 4.0.x [1].
• Khepri became the default backend as of RabbitMQ 4.2.0 [1].
• Khepri is based on the Raft consensus algorithm [1] — the same family of design as quorum queues.

What Raft changes operationally

With Khepri, all operations that update the metadata store, and sometimes even queries, require a quorum number of nodes to be available [2]. Changing cluster membership — adding or removing a member — needs a Raft leader and a quorum to commit; without one, the operation times out [2].

Enabling it on an existing cluster

• Khepri is enabled with the khepri_db feature flag — via rabbitmqctl enable_feature_flag khepri_db, the management UI, or the RABBITMQ_FEATURE_FLAGS environment variable [3].
• It can only be enabled when all cluster nodes are online and healthy; not while a node or the cluster is stopped [3].
• The migration uses resources and pauses other activity briefly near the end, so client operations may time out; RabbitMQ advises running it away from peak load [3].
• If the migration hits an error, everything is rolled back and RabbitMQ continues on Mnesia [3].

What we ask

We ask RabbitMQ engineers what happens to a queue declaration on the minority side of a partition once Khepri is enabled, and how they would schedule the migration on a busy cluster. Engineers who have done it answer with quorum arithmetic and a maintenance window; the rest answer with the feature flag command.