fix: StatefulSet stuck with 0 replicas#1902
Open
dcoppa wants to merge 4 commits intoAltinity:0.26.0from
Open
Conversation
…tion loops The IsStatefulSetReady() function incorrectly returned false for StatefulSets with spec.replicas=0 and status.readyReplicas=0, treating a successfully scaled-to-zero StatefulSet as "not ready". This caused infinite reconciliation loops during StatefulSet recreation: 1 - doDeleteStatefulSet() scales to 0 replicas for graceful termination 2 - recreateStatefulSet() deletes and creates the StatefulSet 3 - During recreation, the StatefulSet temporarily has 0 replicas 4 - updateStatefulSet() checks IsStatefulSetReady(), gets false 5 - Defaults to ErrCRUDRecreate action, triggering deletion again 6 - Loop repeats indefinitely The fix recognizes that a StatefulSet with 0/0 replicas is in a valid ready state, allowing the recreation flow to complete successfully. This issue manifested during rolling updates when manifest changes triggered StatefulSet recreation, and only affected some replicas due to timing/race conditions in the reconciliation loop. Signed-off-by: David Coppa <dcoppa@gmail.com>
Author
|
I'm now using a dedicated function that checks for StatefulSets stuck at 0 replicas and forces them to scale up to the desired replica count: With this, I can't reproduce the problem anymore. |
dcoppa
changed the title
Member
|
@dcoppa , is there any easy way to reproduce the original problem? |
Author
clickhousekeeperinstallation_before.txt Basically, I updated my ClickHouseKeeperInstallation manifest by adding the necessary configuration for Prometheus metrics. Please see the two attached files (before and after). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The ClickHouse operator enters an infinite reconciliation loop when updating StatefulSets that require recreation, causing some keeper pods to remain stuck with 0 replicas indefinitely.
Root Cause
IsStatefulSetReady()inpkg/model/k8s/stateful_set.goincorrectly returnsfalsefor StatefulSets withspec.replicas=0andstatus.readyReplicas=0. This breaks the operator's own StatefulSet recreation flow:doDeleteStatefulSet()scales to 0 replicas for graceful pod termination (line 477 instatefulset-reconciler.go)WaitHostStatefulSetReady()expecting it to succeed at 0/0 replicas (line 484)updateStatefulSet()checksIsStatefulSetReady()and getsfalse(line 254)ErrCRUDRecreate, triggering deletion again instead of allowing the update to proceedReproduction
Trigger a rolling update that requires StatefulSet recreation (e.g., adding a metrics port)
Observed behavior
kubectl scale --replicas=1required to break the loopNote: This is a race condition - some environments/replicas succeed while others get stuck, depending on reconciliation timing.
Logs:
Important items to consider before making a Pull Request
Please check items PR complies to:
next-releasebranch, not intomasterbranch1. More info--
1 If you feel your PR does not affect any Go-code or any testable functionality (for example, PR contains docs only or supplementary materials), PR can be made into
masterbranch, but it has to be confirmed by project's maintainer.