-
Notifications
You must be signed in to change notification settings - Fork 857
Description
What problem do you want to solve?
The current AWS Lambda instrumentation library flushes all metrics after each invocation, which results in metric data-points being generated after each invocation, regardless of the configured export interval. This issue becomes increasingly problematic for Lambda functions with short execution durations (e.g. ~100ms) since even datapoints with the same attributes, will not have the ability to be aggregated if they span more than a single invocation. In addition to being operationally inefficient, because almost all observability vendors charge based on the number of datapoints ingested, this pattern results in dramatic increases in costs.
Describe the solution you'd like
There are several changes which can be employed to help reduce the number of metric datapoints being generated. The two initial changes are as follows:
- Provide a configuration option/environment variable to disable force flushing metrics after each invocation.
- Provide a configuration option/environment variable to register a signal handler for the
SIGTERMshutdown event to flush all metrics before the Lambda instance is cleaned up. (see https://aws.amazon.com/blogs/compute/python-3-12-runtime-now-available-in-aws-lambda/) - Either: add an enhancement to the existing PeriodicExportingMetricReader to add the ability to suspend/resume metric collection to ensure that metric collection stops before the Lambda environment is potentially frozen OR create a new
SuspendableExportingMetricReaderwhich has the same functionality.
Describe alternatives you've considered
Metric aggregation in the collector, however there seems to be limited support for this pattern.
Additional Context
No response
Would you like to implement a fix?
Yes
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.