Add on-demand 'trigger-ml' CLI command for machine learning workers (fixes #2920)#3624
Add on-demand 'trigger-ml' CLI command for machine learning workers (fixes #2920)#3624sohampirale wants to merge 3 commits intochaoss:mainfrom
Conversation
MoralCode
left a comment
MoralCode
left a comment
There was a problem hiding this comment.
Thanks for the contribution. After reviewing the specific code changes, im a little confused on what your intent is with this PR.
Augur uses celery as a task broker and starts up workers for all enabled processing phases/task pipelines at startup. It is then up to the scheduler to send tasks into the pipeline to be run.
If you are having issues getting the ML workers to show up in the flower dashboard for monitoring running tasks, you may need to enable the ML workers in your config and restart augur.
If you see the ML workers but they arent doing anything, there may be an issue with the scheduler that you should report so we can look into it.
I'd be happy to chat about this in more detail over on the CHAOSS Slack in the #wg-augur-8knot channel if you'd like to discuss what your goal is and how we can help you get there.
| logger = logging.getLogger(trigger_ml_phase.__name__) | ||
| enabled_phase_names = get_enabled_phase_names_from_config(engine, logger) | ||
|
|
||
| if RUNNING_DOCKER or machine_learning_phase.__name__ not in enabled_phase_names: |
There was a problem hiding this comment.
why is docker being excluded here?
There was a problem hiding this comment.
oh i see, because it was excluded in the original logic
| logger.info("Starting ML tasks") | ||
| trigger_ml_phase.delay() | ||
| logger.info("ML tasks started") |
There was a problem hiding this comment.
If the ml workers are not enabled in the config, this will still print a message saying they were started - this seems like it could very easily be confusing
| ml_interval = config.get_value('Tasks', 'ml_collection_interval_days') or 40 | ||
| ml_phase_request.append(build_ml_repo_collect_request(session, logger, enabled_phase_names, ml_interval)) | ||
|
|
||
| main_routine = AugurTaskRoutine(logger, ml_phase_request) |
There was a problem hiding this comment.
Im not sure if creating a new instance of the augur task routine here could lead to issues or not, especially since we already have one
MoralCode
added
the
more info needed
3 participants
Description
This PR implements on-demand starting of machine learning workers in Augur. Previously, ML workers were triggered periodically according to a fixed schedule. With these changes, they now start only when an explicit command is issued to an Augur instance, giving more control over when ML processing runs (e.g., via CLI or other command-based triggers).
This addresses the feature request to move away from always-scheduled execution toward user/command-driven activation, which can help with resource efficiency and targeted usage.
This PR fixes Start Machine Learning Workers on Demand #2920
Notes for Reviewers
trigger-mlbranch.trigger-mlunder theaugur tasksgroup, allowing to manually start the ML workers when needed.Signed commits