Kubernetes v1.36 Introduces Staleness Mitigation and Enhanced Controller Observability
The release of Kubernetes v1.36 addresses the critical issue of staleness in controllers, a problem where outdated local caches lead to incorrect or delayed actions in production environments. This update introduces significant improvements in both client-go and kube-controller-manager to mitigate these risks and enhance observability. In client-go, a new feature called atomic FIFO processing ensures queue consistency even when events arrive out of order, allowing clients to verify the latest resource version via the LastStoreSyncResourceVersion function. Consequently, kube-controller-manager leverages this capability for four high-contention controllers: DaemonSet, StatefulSet, ReplicaSet, and Job. These controllers now check cache freshness before executing actions, preventing operations based on stale data. The feature is enabled by default but can be disabled via specific feature gates. This advancement aims to stabilize cluster behavior by ensuring controllers act only on accurate, up-to-date state information, thereby reducing subtle errors caused by cache inconsistencies during restarts or API server disruptions.
Wire timeline
Kubernetes v1.36 Introduces Staleness Mitigation and Enhanced Controller Observability
The release of Kubernetes v1.36 addresses the critical issue of staleness in controllers, a problem where outdated local caches lead to incorrect or delayed actions in production environments. This update introduces significant improvements in both client-go and kube-controller-manager to mitigate these risks and enhance observability. In client-go, a new feature called atomic FIFO processing ensures queue consistency even when events arrive out of order, allowing clients to verify the latest resource version via the LastStoreSyncResourceVersion function. Consequently, kube-controller-manager leverages this capability for four high-contention controllers: DaemonSet, StatefulSet, ReplicaSet, and Job. These controllers now check cache freshness before executing actions, preventing operations based on stale data. The feature is enabled by default but can be disabled via specific feature gates. This advancement aims to stabilize cluster behavior by ensuring controllers act only on accurate, up-to-date state information, thereby reducing subtle errors caused by cache inconsistencies during restarts or API server disruptions.
Kubernetes Blog