Added the option to disconnect ContinueAsNew traces

[bincyber]

What was changed

Added two new options to the Tracing interceptor and support in the OpenTelemetry implementation:

1. DisconnectContinueAsNewWorkflows - setting this to true will create a new root span for any workflows that are ContinueAsNew
2. EnableSpanLinks - setting this to true will create a span link to connect the new root span to the parent span

This change is backwards compatible with these two new options defaulting to false.

Why?

Slack context: https://temporalio.slack.com/archives/CTDTU3J4T/p1702921125553459

Currently, when tracing is enabled on a workflow that ends as ContinueAsNew, the new workflow is attached to the existing trace:

The total number of spans can become extremely large and overwhelm the tracing backend (eg, Grafana Tempo, Jaeger, etc.).

Instead, it would be nice if the new workflow was created as a new, disconnected trace and linked to the existing trace:

This idea can be extended to disconnecting Child Workflows as well.

Checklist

1. How was this tested:

Create the Temporal worker with the tracing interceptor and enable EnableSpanLinks and DisconnectContinueAsNewWorkflows:

tracingInterceptor, err := opentelemetry.NewTracingInterceptor(opentelemetry.TracerOptions{
	EnableSpanLinks:                  true,
	DisconnectContinueAsNewWorkflows: true,
})
if err != nil {
	// handle the error
}

w := worker.New(temporalClient, "example", worker.Options{
	Interceptors:                       []interceptor.WorkerInterceptor{
		tracingInterceptor,
	},
})

I don't currently use DataDog, so I have not been able to test this change on the DataDog tracing interceptor.

[CLAassistant]

All committers have signed the CLA.

[Quinn-With-Two-Ns]

Hey @bincyber thanks for the contribution. Before I start reviewing the implementation we need to first design the API. Did you look at other SDKs open telemetry interceptor or our data dog interceptor for how they support this case? We need to be consistent across our SDKs and tracing interceptors. Generally we would discuss the design in a github issue before we dive into an implementation, but we can do the design on the github issue as well if you prefer

[bincyber]

Hey @bincyber thanks for the contribution. Before I start reviewing the implementation we need to first design the API. Did you look at other SDKs open telemetry interceptor or our data dog interceptor for how they support this case? We need to be consistent across our SDKs and tracing interceptors. Generally we would discuss the design in a github issue before we dive into an implementation, but we can do the design on the github issue as well if you prefer

Let's discuss the design in a GitHub issue. I've created on here: #1976

[jlegrone]

Does this need to be an additional option? It seems like span links (an otel-specific feature?) could be added automatically if the disconnect continue as new workflows option is set.

[bincyber]

I preferred to make this the default, but decided not to for backwards compatibility. Adding links results in additional tracing data which may increase costs depending on your observability vendor

TIL DataDog's SDKs also support span links: https://docs.datadoghq.com/tracing/trace_collection/span_links/