GIE-226: Add LLM-friendly errors and handle empty response cases

[saswatamcode]

This commit transforms query errors to be LLM-friendly with guidance on how to proceed, and also detects and adds guidance for LLMs on empty query responses.

[iNecas]

I wonder if we could better target the case when the metric is purely hallucinated and not present in the system at all. Rather than giving the guidance to the LLM to check the list_metrics, we could then right away return query as error, if the targeted time-series doesn't exist, both saving tokens and getting more deterministic behavior.

[saswatamcode]

There's no real way of doing this here directly when we are executing the query, but...I forgot we already do check for this in the guardrails method. I can move those errors into isSafeQuery method then

[iNecas]

Yes, that's what I was thinking of.

[saswatamcode]

Tried to address this in b923bf5. This is a bit more complex now, but essentially we check parsing, metric and label name existence in guardrails (as an always-on check). And then we check if query response is still empty post execution and guide accordingly

[iNecas]

I've encountered 2 issues during the testing:

1. the thanos-querier in OpenShift doesn't seem to provide TSDB endopoint, see use Prometheus instead of Thanos by default #7
2. (more importantly) even when I query against the correct metric, I get a failure:

curl -X POST -H 'Content-Type: application/json' http://localhost:9100/mcp -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"execute_range_query","arguments":{"query":"up","step":"1m","duration":"5m"}}}'

{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"query validation failed: the metric 'up' does not exist in Prometheus. You can use the 'list_metrics' tool to see all available metrics, or check if the metric name is spelled correctly"}],"isError":true}}

The list_metrics definitely has the metric. But the TSDB stats contain only small subset:

apiserver_request_body_size_bytes_bucket:3776 apiserver_request_duration_seconds_bucket:12422 apiserver_request_sli_duration_seconds_bucket:12100 apiserver_response_sizes_bucket:4040 apiserver_watch_cache_read_wait_seconds_bucket:3136 etcd_request_duration_seconds_bucket:17664 resource_verb:apiserver_request_duration_seconds_bucket:rate:1m:6743 resource_verb:apiserver_request_duration_seconds_bucket:rate:5m:6743 workqueue_queue_duration_seconds_bucket:6054 workqueue_work_duration_seconds_bucket:6054

Is there some pagination going when loading the data?

[saswatamcode]

Hmm yes this is probably due to the specific setup on OpenShift, where we use Querier on top of Thanos sidecars. I have an idea of addressing this, will follow up

[iNecas]

@saswatamcode what's the plan with this PR? Revisit after #15 gets in?

[saswatamcode]

Yes exactly!