This is temporary measure, before we introduce proper room membership managemet and room lifecycle management for Synapse.
Currently, we are open to infinite number of edge cases, what user see does not have full consistency, and for user which is member of active spaces, this will be one big mess.
Ideally, at the monument, user must see only those rooms which belongs to conversation.