Adding auth to a running NatsCluster never rolls out #39

New issue

Closed

opened 2026-10-05 08:12:07 +00:00 by mikluko · 3 comments

mikluko commented

2026-10-05 08:12:07 +00:00

(Migrated from github.com)

Summary

Adding auth to a running NatsCluster that has none never rolls out.
The rollout waits for Settled before it restarts the first server, and the cluster controller cannot observe Settled until a server has restarted with the new config.

Steps to reproduce

Deploy a NatsCluster without auth, with JetStream and three servers, and wait for Ready and Settled.
Add auth with trustRef, systemCredentials and accountTrustRefs, and apply it.

Observed with nats-operator 0.3.1 and nats-server 2.15.0.

What happens

The conditions stay as follows, and no server restarts:

GatewaysConnected=Unknown ObservationFailed: receive $SYS.REQ.SERVER.PING.STATSZ: nats: no responders available for request
Ready=True AllServersReady: 3 of 3 servers ready
Settled=Unknown ObservationFailed: receive $SYS.REQ.SERVER.PING.STATSZ: nats: no responders available for request
Progressing=True GateBlocked: before restarting demo-2 (1 of 3); waiting for Settled: the NATS cluster is not observed

status.rollout keeps all three servers pending, with the gate waiting for Settled:

{"gate":{"waitingFor":"Settled"},"pending":["demo-2","demo-1","demo-0"],"targetRevision":"d93c210f52"}

Why

The controller observes the cluster over the system account, with the credentials that auth.systemCredentials names.
A server that still runs the config without auth has no system account, so nothing answers $SYS.REQ.SERVER.PING.STATSZ.
Settled stays Unknown, and the gate before the first restart never opens.

The cluster keeps serving on the old config.

Expected

The rollout from no auth to auth completes, one server at a time, as other rollouts do.

Some ways to get there:

Observe a cluster whose running config has no auth without $SYS, for example from each server's monitoring port, until the servers run the new config.
Treat "the running config has no system account" as a gate condition of its own, and let the first restart through.

Workaround

Delete the NatsCluster and the PersistentVolumeClaims of its servers, then apply it with auth.
The servers start on the new config, and the JetStream data is lost.

## Summary Adding `auth` to a running `NatsCluster` that has none never rolls out. The rollout waits for `Settled` before it restarts the first server, and the cluster controller cannot observe `Settled` until a server has restarted with the new config. ## Steps to reproduce 1. Deploy a `NatsCluster` without `auth`, with JetStream and three servers, and wait for `Ready` and `Settled`. 1. Add `auth` with `trustRef`, `systemCredentials` and `accountTrustRefs`, and apply it. Observed with nats-operator 0.3.1 and nats-server 2.15.0. ## What happens The conditions stay as follows, and no server restarts: ```text GatewaysConnected=Unknown ObservationFailed: receive $SYS.REQ.SERVER.PING.STATSZ: nats: no responders available for request Ready=True AllServersReady: 3 of 3 servers ready Settled=Unknown ObservationFailed: receive $SYS.REQ.SERVER.PING.STATSZ: nats: no responders available for request Progressing=True GateBlocked: before restarting demo-2 (1 of 3); waiting for Settled: the NATS cluster is not observed ``` `status.rollout` keeps all three servers pending, with the gate waiting for `Settled`: ```json {"gate":{"waitingFor":"Settled"},"pending":["demo-2","demo-1","demo-0"],"targetRevision":"d93c210f52"} ``` ## Why The controller observes the cluster over the system account, with the credentials that `auth.systemCredentials` names. A server that still runs the config without `auth` has no system account, so nothing answers `$SYS.REQ.SERVER.PING.STATSZ`. `Settled` stays `Unknown`, and the gate before the first restart never opens. The cluster keeps serving on the old config. ## Expected The rollout from no `auth` to `auth` completes, one server at a time, as other rollouts do. Some ways to get there: - Observe a cluster whose running config has no `auth` without `$SYS`, for example from each server's monitoring port, until the servers run the new config. - Treat "the running config has no system account" as a gate condition of its own, and let the first restart through. ## Workaround Delete the `NatsCluster` and the PersistentVolumeClaims of its servers, then apply it with `auth`. The servers start on the new config, and the JetStream data is lost.

mikluko commented

2026-10-05 12:02:23 +00:00

(Migrated from github.com)

With 0.4.0 the gate opens and the first server restarts, but the rollout still cannot finish. A server with auth and a server without it cannot route to each other.

After the first restart, the restarted server stays at 1/2 ready, and its log repeats:

[WRN] Account fetch failed: will only fetch valid account keys
[INF] [...]:54192 - rid:84433 - Router connection closed: Missing Account - Remote: demo-0
[INF] [...]:6222 - rid:84432 - Router connection closed: Parse Error - Remote: demo-1

The servers still on the config without auth carry the global account $G over their routes. Under operator mode $G is not a valid account key, so the restarted server drops every route. It cannot rejoin the cluster or the JetStream meta group, never goes Ready, and the gate before the next restart waits for Settled indefinitely:

{"current":"demo-2","gate":{"waitingFor":"Settled"},"pending":["demo-0","demo-1"],"targetRevision":"d93c210f52"}

Every intermediate state of a one-server-at-a-time rollout from no auth to auth is a split cluster, so that rollout cannot succeed whatever the gate does. It takes restarting every server together, with the outage that implies, or recreating the cluster. Observed with nats-operator 0.4.0 and nats-server 2.15.0.

With 0.4.0 the gate opens and the first server restarts, but the rollout still cannot finish. A server with `auth` and a server without it cannot route to each other. After the first restart, the restarted server stays at 1/2 ready, and its log repeats: ```text [WRN] Account fetch failed: will only fetch valid account keys [INF] [...]:54192 - rid:84433 - Router connection closed: Missing Account - Remote: demo-0 [INF] [...]:6222 - rid:84432 - Router connection closed: Parse Error - Remote: demo-1 ``` The servers still on the config without `auth` carry the global account `$G` over their routes. Under operator mode `$G` is not a valid account key, so the restarted server drops every route. It cannot rejoin the cluster or the JetStream meta group, never goes Ready, and the gate before the next restart waits for `Settled` indefinitely: ```json {"current":"demo-2","gate":{"waitingFor":"Settled"},"pending":["demo-0","demo-1"],"targetRevision":"d93c210f52"} ``` Every intermediate state of a one-server-at-a-time rollout from no `auth` to `auth` is a split cluster, so that rollout cannot succeed whatever the gate does. It takes restarting every server together, with the outage that implies, or recreating the cluster. Observed with nats-operator 0.4.0 and nats-server 2.15.0.

mikluko commented

2026-10-05 12:13:32 +00:00

(Migrated from github.com)

Reopened: 0.4.0 (#42) opens the gate, but a server with auth and one without cannot route to each other, so no one-at-a-time rollout across that boundary can finish.

The fix will refuse the in-place transition: the NatsCluster keeps its current servers and Progressing goes False with a reason naming the way out. An annotation will opt into restarting every server together, accepting the outage; recreating the cluster stays the other path. Removing auth will be checked for the same split.

Reopened: 0.4.0 (#42) opens the gate, but a server with `auth` and one without cannot route to each other, so no one-at-a-time rollout across that boundary can finish. The fix will refuse the in-place transition: the `NatsCluster` keeps its current servers and `Progressing` goes False with a reason naming the way out. An annotation will opt into restarting every server together, accepting the outage; recreating the cluster stays the other path. Removing `auth` will be checked for the same split.

mikluko commented

2026-10-05 13:28:41 +00:00

(Migrated from github.com)

Fix in #55: adding or removing auth on a running NatsCluster is held with Progressing False, reason AuthChangeBlocked; the annotation cluster.nats-operator.io/restart-all restarts every server together, or recreate the NatsCluster. Removing auth splits the same way (each side refuses the route pinned to the other's system account), so both directions are held.

Fix in #55: adding or removing `auth` on a running `NatsCluster` is held with `Progressing` False, reason `AuthChangeBlocked`; the annotation `cluster.nats-operator.io/restart-all` restarts every server together, or recreate the `NatsCluster`. Removing `auth` splits the same way (each side refuses the route pinned to the other's system account), so both directions are held.