Building Ginkgo on Cray + AMD machines

[etiennemlb]

I'm a new user of the Ginkgo Project and I'm attempting to build Ginkgo on a machine similar to Frontier (MI250X+Trento+Cray env). Everything in cmake seems to pass and it's detecting HIP and MPI correctly. I use the Cray wrappers and ROCm's amdclang as the HIP compiler.

module load cpe/24.07
module load craype-accel-amd-gfx90a craype-x86-trento
module load PrgEnv-gnu
module load rocm/6.4.3
module load cmake ninja

cmake .. -G Ninja \
    -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_HIP_COMPILER="${ROCM_PATH}/bin/amdclang++" \
    -DCMAKE_BUILD_TYPE=Release -DGPU_TARGETS=gfx90a -DCMAKE_HIP_ARCHITECTURES=gfx90a \
    -DGINKGO_BUILD_HIP=ON \
    -DGINKGO_BUILD_OMP=ON \
    -DGINKGO_BUILD_MPI=ON -DGINKGO_FORCE_GPU_AWARE_MPI=ON \
    -DGINKGO_BUILD_EXAMPLES=OFF -DGINKGO_BUILD_TESTS=ON \
    -DGINKGO_BUILD_BENCHMARKS=ON -DGINKGO_BENCHMARK_ENABLE_TUNING=OFF \
    -DGINKGO_MIXED_PRECISION=OFF -DGINKGO_ENABLE_BFLOAT16=OFF -DGINKGO_ENABLE_HALF=OFF \
    -DGINKGO_BUILD_HWLOC=OFF -DGINKGO_BUILD_PAPI_SDE=OFF -DGINKGO_WITH_CCACHE=OFF

It builds fine until it tries to build benchmark/utils/hip_linops.hip.cpp, then it fails with 5 for these errors:

In file included from ginkgo/build_PrgEnv-gnu/include/ginkgo/ginkgo.hpp:37:
ginkgo/include/ginkgo/core/base/mpi.hpp:25:10: fatal error: 'mpi.h' file not found
   25 | #include <mpi.h>

Upon further investigation, and similarly to #1367 (comment), it would seem that CMake when it detects MPI via the Cray wrappers, does not extract include/linker flags. But then, when we:

target_link_libraries(
        hipsparse_linops_${type}
        PRIVATE Ginkgo::ginkgo ${HIPSPARSE_LIBRARIES}
    )

..., even if ginkgo is linked against MPI, there is no transitive passing of the MPI build flags:

/opt/rocm-6.4.3/bin/amdclang++ -DGKO_BENCHMARK_USE_DOUBLE_PRECISION -DGKO_COMPILING_HIP -DUSE_PROF_API=1 -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_AMD__=1 -D__HIP_ROCclr__=1 -Dhipsparse_linops_d_EXPORTS -Iginkgo/build/include -Iginkgo/include -Iginkgo -munsafe-fp-atomics -Wno-unused-command-line-argument -O3 -DNDEBUG -std=gnu++17 --offload-arch=gfx90a -fPIC -MD -MT benchmark/CMakeFiles/hipsparse_linops_d.dir/utils/hip_linops.hip.cpp.o -MF benchmark/CMakeFiles/hipsparse_linops_d.dir/utils/hip_linops.hip.cpp.o.d -o benchmark/CMakeFiles/hipsparse_linops_d.dir/utils/hip_linops.hip.cpp.o -x hip -c ginkgo/benchmark/utils/hip_linops.hip.cpp 

I'm not sure this is an issue in itself, more of a note on the correct way to use the Cray env. This means that without havnig to explicitly specify the MPI include/link flags, we can't mix compilers (which is unfortunatly what spack does, and probably needs fixing).

[MarcelKoch]

The benchmarks are only relevant if you want to actually run our Ginkgo benchmarks. If you do not plan to do this, you can just set -DGINKGO_BUILD_BENCHMARKS=OFF. I'm pretty confident that you will not see this kind of error in your application.

We will look into fixing this error, but it might take a while, since this is not related to our actual library.

[etiennemlb]

I do plan to use the benchmarks, so this was on purpose, thanks for looking at the issue.

[MarcelKoch]

Here is a patch. Since this file doesn't need the Ginkgo mpi header, I just removed it. Let me know if you still experience any issues.

restrict_includes1.patch