Add a config property descriptor along with a custom resolver and validators#642
Conversation
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/tests/unit/config/test_custom_resolver.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/src/smithy_aws_core/config/custom_resolvers.py
Outdated
Show resolved
Hide resolved
| # Get max_attempts | ||
| max_attempts, attempts_source = resolver.get("max_attempts") | ||
|
|
||
| if retry_mode is None or max_attempts is None: |
There was a problem hiding this comment.
This is ignoring one of these values if both aren't set. It's valid for a customer to only set retry_mode and not max_attempts and expect that to be respected.
There was a problem hiding this comment.
Think of a scenario where we need to resolve retry_strategy, but the resolver only finds retry_mode in the sources without max_attempts. Should we mix the resolved retry_mode with a default max_attempts value? This approach doesn't make sense because when a client needs to resolve retry_strategy, it will have a default retry_strategy value configured. To me it doesn’t make sense to mix a partially resolved strategy with default components. Instead, if either component (retry_mode or max_attempts) is missing, we should use the complete default retry_strategy.
There was a problem hiding this comment.
Not sure I agree with that; I should be able to set either mode OR strategy and have that be reflected. That's how this input works today - you can set only one of the two inputs on the RetryStrategyOptions object.
As a user, I can trust the SDK to pick the right number of max attempts, but realize that I'm likely to get throttled so I'd want to choose adaptive retry mode. If I only set AWS_RETRY_MODE=adaptive in the environment, that needs to be respected.
While I don't think there's a scenario where it would make sense to ignore the user input here, even if there were, we would at least need to raise an error or warning to tell the customer that we are ignoring their input. This would ignore it silently.
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
packages/smithy-aws-core/src/smithy_aws_core/config/validators.py
Outdated
Show resolved
Hide resolved
| self, | ||
| key: str, | ||
| validator: Callable[[Any, str | None], Any] | None = None, | ||
| resolver_func: Callable[[ConfigResolver], tuple[Any, str | None]] | None = None, |
There was a problem hiding this comment.
As a follow-up, I'd like to talk about getting the source to be an enum instead of a string
| # Get max_attempts | ||
| max_attempts, attempts_source = resolver.get("max_attempts") | ||
|
|
||
| if retry_mode is None or max_attempts is None: |
There was a problem hiding this comment.
Not sure I agree with that; I should be able to set either mode OR strategy and have that be reflected. That's how this input works today - you can set only one of the two inputs on the RetryStrategyOptions object.
As a user, I can trust the SDK to pick the right number of max attempts, but realize that I'm likely to get throttled so I'd want to choose adaptive retry mode. If I only set AWS_RETRY_MODE=adaptive in the environment, that needs to be respected.
While I don't think there's a scenario where it would make sense to ignore the user input here, even if there were, we would at least need to raise an error or warning to tell the customer that we are ignoring their input. This would ignore it silently.
| max_attempts = validate_max_attempts(max_attempts, attempts_source) | ||
|
|
||
| options = RetryStrategyOptions( | ||
| retry_mode=retry_mode or "standard", # type: ignore |
There was a problem hiding this comment.
| retry_mode=retry_mode or "standard", # type: ignore | |
| retry_mode=retry_mode |
| super().__init__(msg) | ||
|
|
||
|
|
||
| def validate_host_label(host_label: Any, source: str | None = None) -> str: |
There was a problem hiding this comment.
A host label has to be a string, right?
| def validate_host_label(host_label: Any, source: str | None = None) -> str: | |
| def validate_host_label(host_label: str, source: str | None = None) -> str: |
| if self.resolver_func: | ||
| value, source = self.resolver_func(obj._resolver) | ||
| else: | ||
| value, source = obj._resolver.get(self.key) |
There was a problem hiding this comment.
As another follow up, we should expose each source from the resolver so they can be called directly, for example for use in environment only configs like the bedrock token
| def __get__(self, obj: Any, objtype: type | None = None) -> Any: | ||
| """Get the config value with lazy resolution and caching. | ||
| On first access, the property checks if the value is already cached. If not, it resolves |
There was a problem hiding this comment.
Third follow up - we should have some caching on the resolver as well and make sure they share the config resolver.
Issue #, if available:
This PR adds a
ConfigPropertydescriptor that serves as the central instrument for configuration resolution. The descriptor is responsible for resolving config values from sources, applying custom resolvers for complex resolutions, validating values, caching them, and facilitating value updates.The
ConfigPropertydescriptor implements the Python descriptor protocol to provide:Initial config variables:
region- demonstrates simple resolutionretry_strategy- demonstrates complex resolution (aggregates retry_mode and max_attempts)These two variables ensure that the design handles both simple and complex resolution.
Tests
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.