Skip to content

Support Timestamp Type in IR #1960

@20001020ycx

Description

@20001020ycx

Request

In our hybrid IR + archive search deployment, KQL queries against timestamps can target either IR or archive. While this PR adds timestamp type support for archive, timestamp type support for IR has not yet been implemented. This gap creates two issues:

  1. Duplicate query maintenance: Without a unified timestamp type, we must maintain separate KQL queries for IR and archive. Querying against the timestamp type requires a specialized API (timestamp()), which is only available for archive.
  2. Scalability concerns: Without native timestamp support in IR, timestamp handling must go through our conversion during ingestion. This approach won't scale as we deploy more production tables, since timestamp formats vary across the logging library.

Current deployment behavior:
To temporarily work around these gaps, our current deployment behaves as follows:

  • IR stores timestamps as Unix epoch in milliseconds (integer type) after ingestion-side conversion
  • Archive already supports the timestamp type

Note, to maintain unified KQL during the transition, we implemented a workaround that allows issuin epoch millisecond KQL queries against timestamp types.

Backward compatibility considerations:
Our ultimate goal is to migrate to the official timestamp() API. However, given our 30-day retention period, any new API must coexist with the old API for at least 30 days. We have proposed below migration timestamp:

  1. Implement IR timestamp type support with backward-compatible epoch millisecond handling
  2. Wait 30 days until all IR and archive data uses the timestamp type
  3. Migrate search queries to use the timestamp() API and remove the workaround

Possible implementation

Follow the same implementation proposed in this PR with the support of epoch millisecond handling.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions