feat: support for flux framework as hpc manager

Flux supports the majority of MPI flavors/variants, and can be used
to bootstrap MPI as a plugin. It adds other features for scheduling
and topology that can be used for simulations and ai/ml jobs.
This changeset adds the plugin implementation, including the
plugin module, tests, and an example with a small README to
serve as documentation for the time being.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Author: vsoch

api/openapi-spec/swagger.json

@@ -47,6 +47,16 @@ spec:
                 description: mlPolicy provides the ML-specific parameters for the
                   model training.
                 properties:
+                  flux:
+                    description: flux defines the configuration for the Flux runtime.
+                    properties:
+                      numProcPerNode:
+                        default: 1
+                        description: numProcPerNode is the number of processes per
+                          node.
+                        format: int32
+                        type: integer
+                    type: object
                   mpi:
                     description: mpi defines the configuration for the MPI Runtime.
                     properties:

api/python_api/kubeflow_trainer_api/models/__init__.py

@@ -47,6 +47,16 @@ spec:
                 description: mlPolicy provides the ML-specific parameters for the
                   model training.
                 properties:
+                  flux:
+                    description: flux defines the configuration for the Flux runtime.
+                    properties:
+                      numProcPerNode:
+                        default: 1
+                        description: numProcPerNode is the number of processes per
+                          node.
+                        format: int32
+                        type: integer
+                    type: object
                   mpi:
                     description: mpi defines the configuration for the MPI Runtime.
                     properties:

api/python_api/kubeflow_trainer_api/models/trainer_v1alpha1_flux_ml_policy_source.py

@@ -0,0 +1,32 @@
+# This example deploys the LAMMPS Molecular Dynamic Simulator
+# with MPI orchestrated by the Flux workload manager on 4 nodes.
+# The problem size is defined by the coordinates x,y,z, and the
+# parameter file reaxc.hns.
+# The image has the application, LAMMPS, installed (no Flux)
+# A Flux view will be added on the fly by the Kubeflow trainer
+# The 4 pods ideally map 1:1 to nodes, encompassing a cluster
+# The underlying abstraction is a JobSet with a headless service
+# Flux supports low-latency with Infiniband, EFA, etc., however
+# standard ethernet is used here.
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: TrainJob
+metadata:
+  name: lammps-flux-interactive
+spec:
+  # Reference the pre-defined runtime by name
+  runtimeRef:
+    name: flux-runtime
+  trainer:
+    numNodes: 4
+    image: ghcr.io/converged-computing/metric-lammps:latest
+    # You do not need to write "flux run, etc" here. It will be wrapped
+    command: ["lmp", "-v", "x", "2", "-v", "y", "2", "-v", "z", "2", "-in", "in.reaxc.hns", "-nocite"]
+    # Comment out the command above to make an interactive cluster! Then shell into the 0-0 pod:
+    #  # Source environment
+    #  . /mnt/flux/flux-view.sh
+    #  # Connect to the running lead broker socket
+    #  flux proxy $fluxsocket bash
+    #  # See Flux resources!
+    #  flux resource list
+    #  Run lammps!
+    #  flux run -N 4 -n 4 lmp -v x 2 -v y 2 -v z 2 -in in.reaxc.hns -nocite

api/python_api/kubeflow_trainer_api/models/trainer_v1alpha1_hpcml_policy_source.py

@@ -47,6 +47,16 @@ spec:
                 description: mlPolicy provides the ML-specific parameters for the
                   model training.
                 properties:
+                  flux:
+                    description: flux defines the configuration for the Flux runtime.
+                    properties:
+                      numProcPerNode:
+                        default: 1
+                        description: numProcPerNode is the number of processes per
+                          node.
+                        format: int32
+                        type: integer
+                    type: object
                   mpi:
                     description: mpi defines the configuration for the MPI Runtime.
                     properties: