Critical Upgrade Path for etcd v3.6 to Prevent Zombie Cluster Members
The etcd community has identified a critical bug affecting clusters upgrading from version 3.5 to 3.6, which can cause previously removed nodes, known as zombie members, to reappear and disrupt cluster consensus. This issue stems from inconsistencies between the deprecated v2store and the v3store, which becomes the sole source of truth for membership data in v3.6. To mitigate this risk, maintainers strongly advise users to upgrade to etcd v3.5.26 or later before proceeding to v3.6. This intermediate step includes an automatic synchronization mechanism that repairs potential data inconsistencies. Users unable to access v3.5.26 should delay their upgrade to v3.6 until it is available, as no safe workaround exists for skipping this version. The inconsistency may be triggered by historical bugs in snapshot restoration, forced new cluster creation, or unsafe synchronization settings. This guidance is essential for administrators managing Kubernetes and other distributed systems relying on etcd for state management, ensuring operational stability during the transition to the latest major version.
Wire timeline
Critical Upgrade Path for etcd v3.6 to Prevent Zombie Cluster Members
The etcd community has identified a critical bug affecting clusters upgrading from version 3.5 to 3.6, which can cause previously removed nodes, known as zombie members, to reappear and disrupt cluster consensus. This issue stems from inconsistencies between the deprecated v2store and the v3store, which becomes the sole source of truth for membership data in v3.6. To mitigate this risk, maintainers strongly advise users to upgrade to etcd v3.5.26 or later before proceeding to v3.6. This intermediate step includes an automatic synchronization mechanism that repairs potential data inconsistencies. Users unable to access v3.5.26 should delay their upgrade to v3.6 until it is available, as no safe workaround exists for skipping this version. The inconsistency may be triggered by historical bugs in snapshot restoration, forced new cluster creation, or unsafe synchronization settings. This guidance is essential for administrators managing Kubernetes and other distributed systems relying on etcd for state management, ensuring operational stability during the transition to the latest major version.
Kubernetes Blog