[Bug] Topic list watcher doesn't filter system topics when they get created

[lhotari]

Search before reporting

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

All released versions

Issue Description

The topic list watcher is using org.apache.pulsar.broker.namespace.NamespaceService#getListOfPersistentTopics for listing topics and in addition, org.apache.pulsar.broker.resources.TopicResources#handleNotification doesn't filter out system topics. When a new system topic is added to the namespace, and the pattern is matching all topics, it would get passed to the client as a new topic to be subscribed to.
The expectation is that pattern (regex) topic consumer doesn't subscribe to system topics.
New system topics can be created by Pulsar's transaction support. The transaction support uses system topics that end with certain suffixes, for example with the __transaction_pending_ack suffix.

An additional detail is that topic list watcher initialization hasn't been able to benefit from deduplication added in #23049 since it wasn't using that method for listing topics. Since the listing was ignored, system topics didn't get subscribed when the pattern consumer was initialized. However, it was performing the listing on the broker side unnecessarily.

Error messages

Reproducing the issue

n/a

Additional information

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!