[Bugfix] Fix slow recovery when server down because valid follower servers

[Bugfix] Fix slow recovery when server down because valid follower servers #1057