Skip to content

feat: client side tracing upload path#545

Open
hweawer wants to merge 2 commits intofeat/tracing-httputil-infrastructurefrom
feat/tracing-origin-client-side
Open

feat: client side tracing upload path#545
hweawer wants to merge 2 commits intofeat/tracing-httputil-infrastructurefrom
feat/tracing-origin-client-side

Conversation

@hweawer
Copy link
Collaborator

@hweawer hweawer commented Jan 26, 2026

Summary

Implement end-to-end distributed tracing for Docker blob upload paths in Kraken, enabling observability from proxy → origin. This PR adds OpenTelemetry instrumentation to the client upload flow with hierarchical span creation and trace context propagation.

Motivation

Debugging slow or failed Docker pushes currently requires piecing together logs across multiple services with no correlation. This PR enables:

  • End-to-end visibility: Trace a blob upload from Docker client through proxy, origin, and writeback
  • Performance analysis: Identify bottlenecks in chunked uploads, replication, or backend writes
  • Retry debugging: See which origins were attempted and why they failed

@hweawer hweawer force-pushed the feat/tracing-origin-client-side branch from 3ac9e8f to f1c7884 Compare January 26, 2026 14:12
@hweawer hweawer changed the base branch from feat/tracing-writeback-executor-logging to feat/tracing-httputil-infrastructure January 26, 2026 14:12
@hweawer hweawer self-assigned this Jan 26, 2026
if err := t.originCluster.UploadBlob(ctx, namespace, d, blob); err != nil {
t.failureStats.Counter("upload_blob").Inc(1)
span.RecordError(err)
span.SetStatus(codes.Error, "upload failed")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

considering we will be trying to debug which part of the upload failed, does it make sense to enrich this err message?

// UploadBlob uploads blob to origin cluster. See Client.UploadBlob for more details.
func (c *clusterClient) UploadBlob(namespace string, d core.Digest, blob io.ReadSeeker) (err error) {
func (c *clusterClient) UploadBlob(ctx context.Context, namespace string, d core.Digest, blob io.ReadSeeker) (err error) {
ctx, span := otel.Tracer("kraken-origin-cluster").Start(ctx, "cluster.upload_blob",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

considering the normal client's tracer is called kraken-origin-client, wouldnt it make sense to call this one kraken-origin-cluster-client?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants