test/solver/solver_hip & test/solver/solver_omp tolerance failure

[etiennemlb]

I noticed that these tests where failing on a machine with 8 MI250X GCD an a trento CPU. I used OMP_NUM_THREADS=8 and the GPU device 0.

I use gcc 13.2 and rocm 6.3.3.

This issue was reproducible using the dev branch.

test/solver/solver_omp:
[ RUN      ] Solver/PipeCg.ApplyIsEquivalentToRef
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:950: Failure
Relative error between x.ref and x.dev is 2.043868787015054e-07
	which is larger than 1.2 * this->tol(x) (which is 1.6852000649168166e-07)
x.ref saved as SolverPipeCg.ApplyIsEquivalentToRef.x.ref.mtx
x.dev saved as SolverPipeCg.ApplyIsEquivalentToRef.x.dev.mtx

Google Test trace:
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:878: Multivector with 40 columns
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:751: Preconditioned solver with 4 iterations
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:665: Sparse Matrix with variable row nnz (50x50)

/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:950: Failure
Relative error between x.ref and x.dev is 2.043868787015054e-07
	which is larger than 1.2 * this->tol(x) (which is 1.6852000649168166e-07)
x.ref saved as SolverPipeCg.ApplyIsEquivalentToRef.x.ref.mtx
x.dev saved as SolverPipeCg.ApplyIsEquivalentToRef.x.dev.mtx

Google Test trace:
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:883: Strided multivector with 40 columns
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:751: Preconditioned solver with 4 iterations
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:665: Sparse Matrix with variable row nnz (50x50)

[  FAILED  ] Solver/PipeCg.ApplyIsEquivalentToRef, where TypeParam = PipeCg (23 ms)


test/solver/solver_hip:

[ RUN      ] Solver/PipeCg.ApplyIsEquivalentToRef
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:950: Failure
Relative error between x.ref and x.dev is 2.008522553101075e-07
	which is larger than 1.2 * this->tol(x) (which is 1.6852000649168166e-07)
x.ref saved as SolverPipeCg.ApplyIsEquivalentToRef.x.ref.mtx
x.dev saved as SolverPipeCg.ApplyIsEquivalentToRef.x.dev.mtx

Google Test trace:
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:878: Multivector with 40 columns
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:751: Preconditioned solver with 4 iterations
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:665: Sparse Matrix with variable row nnz (50x50)

/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:950: Failure
Relative error between x.ref and x.dev is 2.008522553101075e-07
	which is larger than 1.2 * this->tol(x) (which is 1.6852000649168166e-07)
x.ref saved as SolverPipeCg.ApplyIsEquivalentToRef.x.ref.mtx
x.dev saved as SolverPipeCg.ApplyIsEquivalentToRef.x.dev.mtx

Google Test trace:
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:883: Strided multivector with 40 columns
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:751: Preconditioned solver with 4 iterations
/home/malaboeuf/malaboeuf/repository.backedup/hpc-cines-tests/Tests/em-validation-suite/ginkgo.mi250x/ginkgo/test/solver/solver.cpp:665: Sparse Matrix with variable row nnz (50x50)

[  FAILED  ] Solver/PipeCg.ApplyIsEquivalentToRef, where TypeParam = PipeCg (37 ms)

[gojakuch]

@etiennemlb thank you for opening the issue!

@pratikvn the PIPECG algorithm can have some precision issues, as we've already seen. would it be ok if we just increase the allowed error a bit for these tests?

[pratikvn]

Yes, unfortunately that is a issue of the PIPECG algorithm. Temporarily, we can increase the tolerance to fix the issue.