Improve dynamic pipeline step monitoring

[schustmi]

Describe changes

This PR updates the way isolated steps are executed and monitored in dynamic pipelines.

Previously, it worked as follows: When the user decided to run an isolated step, we would call an orchestrator method that would launch the step and then wait for the step to finish before returning. This meant that the thread was blocked for the entire duration.

With this PR, the submitting of an isolated step is now separated from the monitoring. When an isolated step needs to be run, the orchestrator submits that steps to its backend. After that, the thread is freed up and can be used to launch other steps. The monitoring of all isolated steps happens in a single thread that is just responsible for the monitoring.

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• IMPORTANT: I made sure that my changes are reflected properly in the following resources:
  • ZenML Docs
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
  • Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[bcdurak]

What a complicated but beautifully crafted piece of code <3

I am leaving a few comments to start with, and I will take another look into the runner tomorrow. Since it touches upon the main orchestrators, I feel like a small test session would be nice. Which ones have you tested so far?

[bcdurak]

I am just going to leave a small comment here. In our case, I don't think it is a big problem, but wanted to mention just in case. There are scenarios where this call might skip the creation of certain metadata without failing (data too large, or unsupported type), in which case, there might be a drift between the step_run in the info and in the DB.