💥 Payload limit options and validation

[jmaeagle99]

What was changed

• 💥 Update worker to get payload error limits from Temporal server and enforce those limits by default. The SDK will now eagerly fail a task that has too large payloads instead of uploading them to the server and having the server fail the workflow. If the server doesn't report error limits, error limits are not enforced in the worker.
• Add client payload limits options for setting payload size warning thresholds. Warnings are emitted in client, worker, and replayer for any payloads that are above these thresholds.

💥 Behavioral change details

Within workers, if a payload exceeds the server limits, the worker will eagerly fail the current task instead of uploading the object with the too large payload. This allows the task to be retried instead of entirely failing the workflow from within the server.

Customers who use gRPC proxies that alter payloads before they are passed to the server (encryption in the proxy, offloading to external storage within proxy) should disable this new behavior on the worker using the new DisablePayloadErrorLimit option on the worker. For example:

c, err := client.Dial(...)

w := worker.New(c, "queue-name", worker.Options{
    DisablePayloadErrorLimit: true,
})

Examples

Log output when an activity attempts to return a result that exceeds the warning limit:

WARN  [TMPRL1103] Attempted to upload payloads with size that exceeded the warning limit. Namespace integration-test-namespace TaskQueue tq-238573ce-2bd2-4ee3-a455-aaf22d640463-TestIntegrationSuite/TestPayloadSizeWarningCustomSize WorkerID <worker_id> WorkerType payload-size-warning-activity-result WorkflowID TestIntegrationSuite/TestPayloadSizeWarningCustomSize RunID 019c2a6b-3833-758d-a30c-4796c230ad05 ActivityID 5 Attempt 1 ActivityType large-payload-activity

The above example will upload the payload to the server as normal.

Log output when a workflow attempts to return a result or provide an activity some input that exceeds the warning limit:

WARN  [TMPRL1103] Attempted to upload payloads with size that exceeded the warning limit. Namespace integration-test-namespace TaskQueue tq-238573ce-2bd2-4ee3-a455-aaf22d640463-TestIntegrationSuite/TestPayloadSizeWarningCustomSize WorkerID <worker_id> WorkflowType payload-size-warning-activity-result WorkflowID TestIntegrationSuite/TestPayloadSizeWarningCustomSize RunID 019c2a6b-3833-758d-a30c-4796c230ad05

The above example will upload the payload to the server as normal.

Log output when a client attempts to provide input to a workflow that exceeds the warning limit:

WARN  [TMPRL1103] Attempted to upload payloads with size that exceeded the warning limit.

The above example will upload the payload to the server as normal.

Why?

Users need to know when payload sizes are approaching or have exceeded size limits. This will help prevent workflow outages and inform users to adjust their workflows to make use of alternate storage methods or to break down their payloads more granularly.

Checklist

1. Closes SDK should fail workflow task if payloads size it known to be too large #2165

2. Closes Warn if the SDK tried to send a payload above a specific size #2167

3. How was this tested: Unit tests

4. Any docs updates needed? Yes

[jmaeagle99]

This is supposed to be similar to https://github.com/temporalio/sdk-python/pull/1288/changes#diff-dca556db4f4a9a3950b33cf90d75886e8c3216b0955fa950e71cf60aadee3946R1211, but the payload visitor in Go doesn't allow access to Memo fields. The visitor (https://github.com/temporalio/api-go/blob/master/proxy/interceptor.go#L652) will pull out the fields of the Memo and pass that map[string]*common.Payload back to the visitor and is handled at https://github.com/temporalio/api-go/blob/master/proxy/interceptor.go#L266. The visitor will see each payload individually rather than as a collection. The collection is what is needed to validate the the memo size.

Thinking of some options to allow this alternative behavior for memos:

• Add a new bool field to VisitPayloadsOptions that means "I want all payloads of a memo at once". Maybe call it VisitMemoPayloadsAsSequence bool. This would allow current uses of the VisitPayloadsOptions` struct to work as-is if they are using named parameters, but would break those using position parameters (less likely).
• Replicate the visitor behavior in the current repo but just for memo fields. Not a great option from a maintenance perspective.
• Don't check memo fields for their limits. This would be a behavioral difference across SDKs.

Looking for other suggestions or alternatives.