internal/natscluster, internal/sysobs, docs: a NatsCluster whose servers may hold a JetStream meta group of their own is not joined to a supercluster's #60

Merged

mikluko merged 10 commits from mikluko/octant/2403-meta-join into mikluko/octant/2406-probe-port

2026-10-05 15:06:52 +00:00

mikluko commented

2026-10-05 14:08:17 +00:00

(Migrated from github.com)

What it does

A NatsCluster with jetstream, no jetstream.domain and a remote other than itself is held at its last render, with Progressing=False, in three cases:

OwnMetaGroup, retained claims: a server about to be created has a data volume claim older than the NatsCluster, for example one left behind by an earlier NatsCluster of the same name. No server is created. This is the likeliest way the reporter's new ended up with a log.
OwnMetaGroup, running servers: the render would change a running server's gateway remote names, its domain or whether it runs JetStream, while the snapshot shows a meta group led by a server of this NATS cluster that lists no peer outside it. To tell these apart, sysobs.Group gains an Outside field, filled from the leader's meta_cluster.replicas.
ObservationFailed: the same kind of change, but the servers cannot be judged. Either the snapshot is missing, a server that would change is silent, or the leader is one of this cluster's servers and its peer list was not read.

The message names the claims or the servers, and both ways out: delete the claims, or set a domain. To make the check before anything is created, the observation in reconcile now runs before server creation. That function is shared with the sibling tickets.

Costs

A standalone JetStream NatsCluster that holds streams and grows into a supercluster by listing a new member is refused as well. From its own side, a fresh remote looks the same as one that has a log. The way out today is a domain, or a wipe.
A cluster whose servers cannot be observed cannot have its remotes, domain or JetStream changed until they answer. Other changes are not blocked.
These cases are not caught:
- A meta group with no leader, which is never judged to be the cluster's own.
- A store JetStream does not run on.
- A fresh claim bound to a retained PV that already holds data.
- A scale-up that reuses a claim created after the NatsCluster.
- A split that is already running. Story 13 now says how to recognise it.

Not built: JetStream off until `GatewaysConnected`

In 2.15.0 a reload cannot enable JetStream properly. TestJetStreamEnabledAfterGateways_Live shows that after a reload, west joins east's meta group and east's stream stays intact, but placing a stream on west fails for an account the server already held. After a restart it works. So jetstream stays restart-only in the allow-list, and the allow-list doc now says why.
Enabling JetStream by restart would deadlock the rollout gate for a supercluster created all at once (stories 6 and 9). Every member holds JetStream back, the first server restarted in each cluster cannot elect a meta leader, the meta group reads NoLeader, and the gate never opens. Avoiding that needs either an all-at-once restart or a gate exception in rollout.go.
It adds no protection. Q2402 showed that fresh stores join cleanly, and a retained store would still be read whenever JetStream turned on. The claim-age check is what catches that case.

Verification

Unit tests, written to fail on the old code: TestReconcile_OwnMetaGroup and TestReconcile_ClaimOlderThanNatsCluster.
Against live in-process nats-server 2.15.0, TestMetaJoinHold_Live:
- Story 1's servers, running JetStream with no gateway, are held when remotes are added.
- Story 6's east and west, which share a meta group, are not judged to have one of their own.
just test, just envtest, just lint, just verify and just site-check pass. Story 13's steps are unchanged, so its e2e runs as before.

The design (section 4.3), story 13 and the changelog are updated.

Issue #51, octant X2403. Q2402 found the cause: a joining NATS cluster's own JetStream meta log. Two meta groups that meet over a gateway do not merge, and the side that loses deletes every stream the winner's snapshot does not list (`jetstream_cluster.go:2413-2418`). A store that starts empty joins cleanly. **This PR is for the maintainer to rule on.** It builds the refusal and leaves out the JetStream hold that Q2402 recommended. The reason is under "Not built". ## What it does A `NatsCluster` with `jetstream`, no `jetstream.domain` and a remote other than itself is held at its last render, with `Progressing=False`, in three cases: - **`OwnMetaGroup`, retained claims**: a server about to be created has a data volume claim older than the `NatsCluster`, for example one left behind by an earlier `NatsCluster` of the same name. No server is created. This is the likeliest way the reporter's `new` ended up with a log. - **`OwnMetaGroup`, running servers**: the render would change a running server's gateway remote names, its domain or whether it runs JetStream, while the snapshot shows a meta group led by a server of this NATS cluster that lists no peer outside it. To tell these apart, `sysobs.Group` gains an `Outside` field, filled from the leader's `meta_cluster.replicas`. - **`ObservationFailed`**: the same kind of change, but the servers cannot be judged. Either the snapshot is missing, a server that would change is silent, or the leader is one of this cluster's servers and its peer list was not read. The message names the claims or the servers, and both ways out: delete the claims, or set a domain. To make the check before anything is created, the observation in `reconcile` now runs before server creation. That function is shared with the sibling tickets. ## Costs - A standalone JetStream `NatsCluster` that holds streams and grows into a supercluster by listing a new member is refused as well. From its own side, a fresh remote looks the same as one that has a log. The way out today is a domain, or a wipe. - A cluster whose servers cannot be observed cannot have its remotes, domain or JetStream changed until they answer. Other changes are not blocked. - These cases are not caught: - A meta group with no leader, which is never judged to be the cluster's own. - A store JetStream does not run on. - A fresh claim bound to a retained PV that already holds data. - A scale-up that reuses a claim created after the `NatsCluster`. - A split that is already running. Story 13 now says how to recognise it. ## Not built: JetStream off until `GatewaysConnected` - In 2.15.0 a reload cannot enable JetStream properly. `TestJetStreamEnabledAfterGateways_Live` shows that after a reload, west joins east's meta group and east's stream stays intact, but placing a stream on west fails for an account the server already held. After a restart it works. So `jetstream` stays restart-only in the allow-list, and the allow-list doc now says why. - Enabling JetStream by restart would deadlock the rollout gate for a supercluster created all at once (stories 6 and 9). Every member holds JetStream back, the first server restarted in each cluster cannot elect a meta leader, the meta group reads NoLeader, and the gate never opens. Avoiding that needs either an all-at-once restart or a gate exception in `rollout.go`. - It adds no protection. Q2402 showed that fresh stores join cleanly, and a retained store would still be read whenever JetStream turned on. The claim-age check is what catches that case. ## Verification - Unit tests, written to fail on the old code: `TestReconcile_OwnMetaGroup` and `TestReconcile_ClaimOlderThanNatsCluster`. - Against live in-process nats-server 2.15.0, `TestMetaJoinHold_Live`: - Story 1's servers, running JetStream with no gateway, are held when remotes are added. - Story 6's east and west, which share a meta group, are not judged to have one of their own. - `just test`, `just envtest`, `just lint`, `just verify` and `just site-check` pass. Story 13's steps are unchanged, so its e2e runs as before. The design (section 4.3), story 13 and the changelog are updated.