A NatsCluster joining a supercluster founds its own JetStream meta group and can lead the supercluster's #51

New issue

Open

opened 2026-10-05 12:22:40 +00:00 by mikluko · 1 comment

mikluko commented

2026-10-05 12:22:40 +00:00

(Migrated from github.com)

Summary

A NatsCluster with JetStream that joins an existing supercluster can become the meta leader of the whole supercluster while knowing only its own servers as peers.
Its servers form a JetStream meta group of their own before any gateway connects, then bring that group's leader into the supercluster's.

Steps to reproduce

Run a supercluster whose clusters share one JetStream meta group (no JetStream domain), with reject_unknown on every member.
Deploy a NatsCluster new with jetstream, gateway.discovery: Explicit and every member in gateway.remotes.
Add new to the gateway lists of the existing members.

Observed with nats-operator 0.4.0 and nats-server 2.15.0.

What happens

Until the members list new, they reject its gateways, so its three servers start isolated and elect a meta leader among themselves.
Once the gateways connect, every server in the supercluster reports that leader, while the leader's view of the group holds four peers against the other servers' seventeen:

new-0..2            meta leader new-2, cluster_size 4, peers new-0, new-1, new-2, other-nats-1
every other server  meta leader new-2, cluster_size 17

With four peers, three servers of new make a quorum on their own, so a cluster with no streams can commit metadata for the whole supercluster.

Expected

A NatsCluster joining a supercluster joins its meta group rather than founding one.
Some ways to get there:

Render JetStream off until the gateways to the listed remotes are connected, and turn it on afterwards.
Refuse, or warn on, a NatsCluster with both jetstream and gateway remotes and no JetStream domain, and document the order of steps in "Add a NATS cluster to a supercluster that you did not deploy".

Workaround

Delete the NatsCluster and the PersistentVolumeClaims of its servers, so that the remaining servers elect a meta leader among themselves.

## Summary A `NatsCluster` with JetStream that joins an existing supercluster can become the meta leader of the whole supercluster while knowing only its own servers as peers. Its servers form a JetStream meta group of their own before any gateway connects, then bring that group's leader into the supercluster's. ## Steps to reproduce 1. Run a supercluster whose clusters share one JetStream meta group (no JetStream domain), with `reject_unknown` on every member. 1. Deploy a `NatsCluster` `new` with `jetstream`, `gateway.discovery: Explicit` and every member in `gateway.remotes`. 1. Add `new` to the gateway lists of the existing members. Observed with nats-operator 0.4.0 and nats-server 2.15.0. ## What happens Until the members list `new`, they reject its gateways, so its three servers start isolated and elect a meta leader among themselves. Once the gateways connect, every server in the supercluster reports that leader, while the leader's view of the group holds four peers against the other servers' seventeen: ```text new-0..2 meta leader new-2, cluster_size 4, peers new-0, new-1, new-2, other-nats-1 every other server meta leader new-2, cluster_size 17 ``` With four peers, three servers of `new` make a quorum on their own, so a cluster with no streams can commit metadata for the whole supercluster. ## Expected A `NatsCluster` joining a supercluster joins its meta group rather than founding one. Some ways to get there: - Render JetStream off until the gateways to the listed remotes are connected, and turn it on afterwards. - Refuse, or warn on, a `NatsCluster` with both `jetstream` and gateway remotes and no JetStream domain, and document the order of steps in "Add a NATS cluster to a supercluster that you did not deploy". ## Workaround Delete the `NatsCluster` and the PersistentVolumeClaims of its servers, so that the remaining servers elect a meta leader among themselves.

mikluko commented

2026-10-05 14:08:26 +00:00

(Migrated from github.com)

Fix proposed in #60, for review: a NatsCluster whose servers may already have a JetStream meta group of their own (a data claim older than the NatsCluster, or running servers leading a meta group with no peer outside their NATS cluster) is not joined to the supercluster's meta group by its gateway remotes; it reads Progressing False, reason OwnMetaGroup.

Fix proposed in #60, for review: a `NatsCluster` whose servers may already have a JetStream meta group of their own (a data claim older than the `NatsCluster`, or running servers leading a meta group with no peer outside their NATS cluster) is not joined to the supercluster's meta group by its gateway remotes; it reads `Progressing` False, reason `OwnMetaGroup`.