Help deployment path of TrinityX

[Wodel]

Hi,

Could you help with deployment path of an HPC.

This is the hardware and the software that will be used, along side with the architecture of the deployment.

Hardware used

• 3 nodes : a Proxmox HCI cluster will be deployed here, this cluster will contain several infrastructure VMs as well as the 2 controllers in HA mode and 1 login node of TrinityX.
• 5 gpu nodes with nvme disks, those nodes will be the compute node, but also they will contain a ceph cluster (HCI mode), cephfs will be configured to hold /scratch and HPC modules via Easybuild.
• a NAS, that will be used for users home directories /home, and it will be used for backup.
• a management and ipmi switch 1Gbps
• an external switch to let users access the HPC 10Gbs
• a storage and HPC switch (400Gbs) will be used for, ceph storage, nfs storage, backup and also for mpi/gpu compute (using different vlans).

Can you helps us with the path of deployment, especially

• The order of deployment.
• The HA storage for the controllers, especially /home, /scratch and /modules...
• any good advice will be appreciated.

Regards.

[aphmschonewille]

Hi Wodel,

Deployment is determined by the dependencies. If you want to use an external nas based storage from the get-go (used in trix_shared_fs), then that one comes first. If prefer to do that at a later stage, then it can be done manually after install. In principle, almost all changes can be done afterwards (manually).

In your case, incorporating ceph, is beyond the get-go installation. How and what are too complex and error prone to support all kind of external dependencies out of the box. This particular step I'd do afterwards in the images or (post) scripts.
Switches etc come highly recommended to have setup and configured first.

If you use any supported (get-go) external storage for HA, I'd set that up first and use it straight away (trix_shared_fs), especially the {{ trix_ha }} part. Have a look here as well: HA architecture

Is there a golden rule? No, but the more you stick to defaults, the higher chance of getting a working setup, out of the box, which can be altered afterwards. Some things you're better of doing right of the bat, like HA storage.

Antoine