[fix] Bookie Info lost by notification race condition. #20642

[gaozhangmin]

According to pulsar's fix: apache/pulsar#20642
race conditions problem is also exists in bk's ZKRegistrationClient.java

[StevenLuMT]

Can you provide some information about the process of locating the problem you encountered, or is this just a simple logic synchronization with Pulsar?

[gaozhangmin]

@StevenLuMT
After switching the bookie from readonly to writable status online, the broker side continuously generated error logs: "Bookie Handle not available." The issue was resolved only after restarting the bookie. It was discovered that there was a bug in the ZkRegisterClient relied upon by the Pulsar broker, which caused the broker's bookie status cache update to be lost. Although the bookie was actually in a normal state, the broker failed to resolve the bookie's hostname due to the missing cache, resulting in the error: "Bookie Handle not available."

[Copilot]

Pull Request Overview

This PR addresses a race condition in BookKeeper's ZKRegistrationClient where bookie information could be lost due to notification timing issues. The fix separates the cache for writable and read-only bookies and ensures sequential processing of ZooKeeper events to prevent race conditions.

• Splits the single bookieServiceInfoCache into separate caches for writable and read-only bookies
• Introduces sequential processing of ZooKeeper notifications using a Sequencer utility class
• Adds comprehensive test coverage to verify the fix handles network delays during bookie transitions

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
ZKRegistrationClient.java	Core fix implementing separate caches and sequential event processing
FutureUtils.java	Adds Sequencer utility class for sequential task execution
ZKRegistrationClientTest.java	New test to verify race condition fix with network delay simulation
FaultInjectableZKRegistrationManager.java	Test utility for fault injection during registration operations

[gaozhangmin]

rerun failure checks

[gaozhangmin]

rerun failure checks

[wenbingshen]

@gaozhangmin Is it different from this fix #4481

[gaozhangmin]

@gaozhangmin Is it different from this fix #4481

it's different case

[gaozhangmin]

rerun failure checks