Collect logs from system namespaces

[Schnitzel]

We had a couple of cases where we really needed logs from system namespaces.
I'm not sure though if this is really the right way to do this. Basically we're sending many thousand of log entries to the elasticsearch which is also used by our customers.
If I would have time I probably would create a separate logging infrastructure for system logs from our clusters, but as we don't really have time I think the easiest right now is to just send them in the same place.
Any opinions @smlx ?

[smlx]

Here are some problems I see with doing this:

• many system services are very verbose, this would add significant load on elasticsearch
• fluentd currently looks for lagoon.sh/project labels and otherwise puts all logs into a catch-all index.
• I'm not sure how well parsing will work, since these log formats have not been tested with the current fluent.conf

I agree that it would be much better to split these out to a different log sink. However that doesn't need to be elasticsearch and could be something much simpler since we don't need the multi-tenancy or the extra processing we currently do with fluentd in the logs-dispatcher. I've got some ideas for this 😄

Another thought: do we really need all these namespaces? one or two might be ok as a quick fix for your use-case?