Drift: NodeClass changes that do not show on the server never drift nodes #24

New issue

Open

opened 2026-10-27 21:46:32 +00:00 by mikluko · 0 comments

mikluko commented

2026-10-27 21:46:32 +00:00

Owner

Changing a HCloudNodeClass does not drift the nodes made from it unless the change shows up on the running server. IsDrifted in pkg/cloudprovider/cloudprovider.go compares the server against the NodeClass for image ID, network, firewalls, server type, location and labels. Nothing else in the spec is compared, so an edit to any of these leaves existing nodes in place indefinitely:

bootstrap (beforeJoin, afterJoin, k0sVersion)
imageSelector: the recorded image ID is compared with the server's own image, which never changes, so a new selector is never detected
sshKeyIDs
enablePublicIPv4, enablePublicIPv6

Seen with 0.7.2: a sysctl added to beforeJoin reached new nodes only; existing NodeClaims never got a Drifted condition, and restarting the controller did not change that. The only way to apply it was to delete the NodeClaims by hand.

Most of these fields cannot be read back from the server, so comparing them against the server cannot work. The AWS provider solves the same problem with a hash: the NodeClass carries a hash of its spec (karpenter.k8s.aws/ec2nodeclass-hash, plus -hash-version), each NodeClaim is stamped with the hash at launch, and a mismatch is reported as NodeClassDrift. Fields that are checked against the server, or that change without needing new nodes, are excluded from the hash. The hash version lets a later change to the hashed fields avoid drifting every node at upgrade.

Written by an AI, reviewed by a human being.

Changing a `HCloudNodeClass` does not drift the nodes made from it unless the change shows up on the running server. `IsDrifted` in `pkg/cloudprovider/cloudprovider.go` compares the server against the NodeClass for image ID, network, firewalls, server type, location and labels. Nothing else in the spec is compared, so an edit to any of these leaves existing nodes in place indefinitely: - `bootstrap` (`beforeJoin`, `afterJoin`, `k0sVersion`) - `imageSelector`: the recorded image ID is compared with the server's own image, which never changes, so a new selector is never detected - `sshKeyIDs` - `enablePublicIPv4`, `enablePublicIPv6` Seen with 0.7.2: a sysctl added to `beforeJoin` reached new nodes only; existing NodeClaims never got a `Drifted` condition, and restarting the controller did not change that. The only way to apply it was to delete the NodeClaims by hand. Most of these fields cannot be read back from the server, so comparing them against the server cannot work. The AWS provider solves the same problem with a hash: the NodeClass carries a hash of its spec (`karpenter.k8s.aws/ec2nodeclass-hash`, plus `-hash-version`), each NodeClaim is stamped with the hash at launch, and a mismatch is reported as `NodeClassDrift`. Fields that are checked against the server, or that change without needing new nodes, are excluded from the hash. The hash version lets a later change to the hashed fields avoid drifting every node at upgrade. --- *Written by an AI, reviewed by a human being.*

mikluko referenced this issue from a commit

2026-10-27 21:56:09 +00:00

fix: drift nodes on node class edits the server cannot show, cut 0.7.2

mikluko referenced this issue from a commit

2026-10-27 22:14:07 +00:00

fix: drift nodes on node class edits the server cannot show, cut 0.8.0

mikluko referenced this issue from a pull request that will close it,

2026-10-27 22:14:15 +00:00

fix: drift nodes on node class edits the server cannot show, cut 0.8.0 #26

mikluko referenced this issue from a commit

2026-10-27 22:16:43 +00:00

fix: drift nodes on node class edits the server cannot show, cut 0.8.0