[feature]: Prometheus/OpenTelemetry metrics for queue

[gustavclausen]

Is your feature request related to a problem? Please describe.

Hi!

Thanks for this great project.

I see that some Prometheus metrics was once collected and exposed (#4) but I don't see this feature included anymore.

Describe the solution you would like

I'm particularly interested in monitoring the queue - more specifically:

• Count of read & write events
• Queue depth
• Count of delayed/requeue events

Is this something that can be implemented in the default TimedEntityQueue implementation? Or is the suggestion to implement a custom queue (as per these docs) that collects the metrics?

I would be happy to contribute with any changes in this regard.

Additional Context

No response

[kimpenhaus]

Hey @gustavclausen

thanks for reaching out - this seems to be long time gone 😄 hadn't had an idea it was existing.

from my perspective open telemetry metrics should be the way to go and therefore needs to have an HTTP pull endpoint as mentioned in the docs on top of collecting the data - maybe you have more experience with that and can elaborate a little on this?

generally I think there is still some design issue in the operator-sdk, unfortunately, as the queue is not in front of the reconciler itself but the watcher and the queue call the reconciler side-by-side. this leads to a potentially race condition #977 and limits the parallelisation of the reconciler (which could be more efficient when reconciling same kinds in parallel - but not same uid's) - see discussion #656

that said collecting metrics inside the queue would currently be just a partial overview.

I think integrating otel metrics in the queue is a good first way to get started - keeping in mind that it doesn't give the total picture. I guess this should be implemented in the operator, so it doesn't needs to be implemented as a custom behaviour over and over again - maybe configurable by the operator settings.

@buehler, @gustavclausen - what's your opinion on this?

[gustavclausen]

Thanks for providing the additional context @kimpenhaus. I wasn't aware of this.

therefore needs to have an HTTP pull endpoint as mentioned in the docs on top of collecting the data - maybe you have more experience with that and can elaborate a little on this?

Yes, a HTTP endpoint for scraping the metrics would be the way to go. There are some examples in the official OTEL docs for reference (example of custom counter and example in ASP.NET Core).

I would be happy to build the OTEL foundation and support queue metrics for starters.