-
Notifications
You must be signed in to change notification settings - Fork 89
Description
Request
In our hybrid IR + archive search deployment, KQL queries against timestamps can target either IR or archive. While this PR adds timestamp type support for archive, timestamp type support for IR has not yet been implemented. This gap creates two issues:
- Duplicate query maintenance: Without a unified timestamp type, we must maintain separate KQL queries for IR and archive. Querying against the timestamp type requires a specialized API (timestamp()), which is only available for archive.
- Scalability concerns: Without native timestamp support in IR, timestamp handling must go through our conversion during ingestion. This approach won't scale as we deploy more production tables, since timestamp formats vary across the logging library.
Current deployment behavior:
To temporarily work around these gaps, our current deployment behaves as follows:
- IR stores timestamps as Unix epoch in milliseconds (integer type) after ingestion-side conversion
- Archive already supports the timestamp type
Note, to maintain unified KQL during the transition, we implemented a workaround that allows issuin epoch millisecond KQL queries against timestamp types.
Backward compatibility considerations:
Our ultimate goal is to migrate to the official timestamp() API. However, given our 30-day retention period, any new API must coexist with the old API for at least 30 days. We have proposed below migration timestamp:
- Implement IR timestamp type support with backward-compatible epoch millisecond handling
- Wait 30 days until all IR and archive data uses the timestamp type
- Migrate search queries to use the timestamp() API and remove the workaround
Possible implementation
Follow the same implementation proposed in this PR with the support of epoch millisecond handling.