Avoid creating transient index patterns in the global tenant #208
Description
When Opensearch is under heavy load, tenant creation requests can time out. Even if the creation request doesn't time out, tenants can take some minutes to appear in the Opensearch API.
Currently the index-pattern creation logic assumes that tenants created in the "current sync run" will be available. If the tenants haven't appeared in the Opensearch API yet, then Opensearch will ignore securitytenant: xxx headers and newly created index patterns will appear in the Global tenant.
This isn't usually a big problem, since they will be removed and recreated in the correct tenant on the next sync run. But having index patterns incorrectly created in the Global tenant even for a few minutes isn't ideal.
Maybe Opensearch sync could check the tenant API to ensure a tenant exists before creating an index pattern in that tenant? Alternatively, if a tenant creation request times out or errors in a particular sync run, maybe index pattern creation for that tenant should not be attempted?