Feat/m1 config based job loading #20997
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements M1 milestone for loading job specifications from TOML configuration at node startup. Features: - LoadJobsFromConfig with dependency injection pattern - ParseJobSpecsFromTOML for [[Jobs]] section parsing - Idempotent loading via ExternalJobID - Comprehensive test coverage (10 tests) - Success/failure observability Breaks import cycles using ValidatorFunc injection. Backward compatible with existing job loading.
- Add ConfigTOMLPath to ApplicationOpts - Add job loading after jobSpawner creation - Support cron and directrequest validators - Build verified
Changed integration to use correct job format: - type="standardcapabilities" with command field (cron, consensus, streams) - Not separate types like type="cron", type="directrequest" Updates: - application.go: Changed validators map to use standardcapabilities key - loader_test.go: Updated all tests to use standardcapabilities format This aligns with the actual job architecture where all standard capabilities jobs use the same type with different command values.
Changed job loading to use cfg.ConfigTOML() which returns the effective merged configuration from all TOML files, rather than trying to read a specific file path. This properly supports the CLP operator's config layering where multiple configs are passed via -c flags and merged. - Removed ConfigTOMLPath field from ApplicationOpts - Read from effective config using cfg.ConfigTOML() - Works with any config source (file, env, CLP ConfigMaps)
M1 reads from the effective configuration which is the merged result of all TOML files passed via -c flags. No hardcoded paths needed. For K8s/CLP deployments, job config should be added to ConfigMaps with descriptive keys (e.g., standardcap-jobs.toml) which the operator will automatically mount and include in the -c flags.
Contributor
|
I see you updated files related to
|
|
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requires
Supports