Test times increased significantly

[maleadt]

I updated a local copy of GPUArrays.jl from 97bdfdb to f516fc2, and test times increased from 4 minutes 31 to a whopping 7m32.

Before:

[ Info: Running 5 tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the JULIA_CPU_THREADS environment variable.
                                                |          | ---------------- CPU ---------------- |
Test                                   (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
JLArray/alloc cache                         (2) |     1.00 |   0.01 |  0.8 |     192.21 |   557.28 |
JLArray/indexing scalar                     (3) |     5.81 |   0.12 |  2.0 |    1279.38 |   602.16 |
JLArray/indexing find                       (3) |     7.51 |   0.41 |  5.5 |    2602.77 |   765.70 |
JLArray/reductions/reducedim!               (4) |    28.30 |   0.86 |  3.0 |    6716.30 |   662.75 |
JLArray/math/power                          (6) |    29.88 |   1.32 |  4.4 |    7353.23 |   788.39 |
JLArray/reductions/any all count            (4) |     4.22 |   0.10 |  2.4 |    1054.88 |   732.34 |
JLArray/uniformscaling                      (4) |     4.40 |   0.08 |  1.9 |     685.22 |   768.08 |
JLArray/indexing multidimensional           (3) |    24.12 |   0.90 |  3.7 |    4996.91 |   866.39 |
JLArray/linalg/mul!/vector-matrix           (2) |    48.91 |   2.53 |  5.2 |   12632.54 |   814.38 |
JLArray/math/intrinsics                     (2) |     1.55 |   0.00 |  0.0 |     261.91 |   816.22 |
JLArray/reductions/mapreducedim!_large      (4) |    16.79 |   6.39 | 38.0 |   33495.93 |  1238.14 |
JLArray/linalg/NaN_false                    (2) |     9.20 |   0.57 |  6.2 |    2102.50 |   901.80 |
JLArray/statistics                          (2) |    35.87 |   2.51 |  7.0 |    7449.54 |   988.30 |
JLArray/reductions/minimum maximum extrema  (6) |    70.57 |   4.66 |  6.6 |   16305.19 |   992.94 |
JLArray/linalg/mul!/matrix-matrix           (3) |    63.02 |   3.71 |  5.9 |    8032.98 |   896.27 |
JLArray/ext/jld2                            (3) |    12.90 |   0.66 |  5.1 |    1569.69 |   973.78 |
JLArray/vectors                             (3) |     0.14 |   0.00 |  0.0 |      20.11 |  1019.12 |
JLArray/constructors                        (6) |    14.77 |   0.70 |  4.7 |    2026.22 |  1067.75 |
JLArray/linalg/norm                         (4) |    70.23 |   6.18 |  8.8 |   19389.82 |  1238.14 |
JLArray/random                              (3) |    11.97 |   0.82 |  6.8 |    1525.67 |  1058.08 |
JLArray/reductions/== isequal               (4) |    16.24 |   1.34 |  8.3 |    1812.29 |  1243.11 |
JLArray/base                                (6) |    34.23 |   6.37 | 18.6 |   10291.93 |  1337.23 |
JLArray/reductions/mapreduce                (2) |    54.59 |   4.71 |  8.6 |    8460.67 |  1112.97 |
JLArray/reductions/mapreducedim!            (4) |    16.72 |   1.65 |  9.8 |    2492.19 |  1286.08 |
Array/alloc cache                           (4) |     0.35 |   0.00 |  0.0 |      82.41 |  1317.64 |
Array/indexing scalar                       (4) |     1.68 |   0.00 |  0.0 |     133.01 |  1317.64 |
Array/reductions/reducedim!                 (4) |     0.01 |   0.00 |  0.0 |       2.43 |  1317.64 |
JLArray/reductions/sum prod                 (2) |    37.45 |   2.84 |  7.6 |    4407.01 |  1213.83 |
Array/linalg                                (4) |    21.69 |   3.05 | 14.0 |    3531.55 |  1424.41 |
Array/math/power                            (2) |     0.02 |   0.00 |  0.0 |       1.38 |  1245.14 |
JLArray/reductions/reduce                   (6) |    40.04 |   4.01 | 10.0 |    4899.03 |  1405.31 |
Array/linalg/mul!/vector-matrix             (4) |     1.59 |   0.00 |  0.0 |     256.59 |  1424.41 |
Array/reductions/any all count              (4) |     0.10 |   0.00 |  0.0 |      16.69 |  1424.41 |
Array/indexing find                         (2) |     2.44 |   0.00 |  0.0 |     409.17 |  1245.14 |
Array/reductions/minimum maximum extrema    (4) |     0.61 |   0.00 |  0.0 |      61.04 |  1424.41 |
Array/reductions/mapreducedim!_large        (4) |     0.51 |   0.06 | 12.0 |     825.78 |  1642.48 |
Array/uniformscaling                        (2) |     3.52 |   0.00 |  0.0 |     314.28 |  1245.14 |
Array/linalg/mul!/matrix-matrix             (4) |     1.44 |   0.00 |  0.0 |     157.98 |  1642.48 |
Array/math/intrinsics                       (2) |     0.00 |   0.00 |  0.0 |       0.03 |  1245.14 |
Array/indexing multidimensional             (6) |     8.25 |   0.00 |  0.0 |    1004.00 |  1405.31 |
Array/linalg/NaN_false                      (4) |     0.63 |   0.00 |  0.0 |     115.89 |  1642.48 |
Array/reductions/mapreduce                  (4) |     0.17 |   0.00 |  0.0 |      25.50 |  1642.48 |
Array/statistics                            (6) |     1.19 |   0.00 |  0.0 |     304.55 |  1406.94 |
Array/linalg/norm                           (2) |     8.27 |   0.00 |  0.0 |     859.41 |  1245.14 |
Array/ext/jld2                              (6) |     4.69 |   0.00 |  0.0 |     370.13 |  1443.97 |
Array/vectors                               (2) |     0.20 |   0.00 |  0.0 |      20.38 |  1279.97 |
Array/constructors                          (4) |     9.13 |   0.00 |  0.0 |    1047.21 |  1642.48 |
Array/reductions/== isequal                 (4) |     0.01 |   0.00 |  0.0 |       1.20 |  1642.48 |
Array/random                                (6) |     4.56 |   0.00 |  0.0 |     352.74 |  1443.97 |
Array/reductions/mapreducedim!              (6) |     0.57 |   0.00 |  0.0 |      38.10 |  1443.97 |
Array/reductions/reduce                     (6) |     0.01 |   0.00 |  0.0 |       2.18 |  1443.97 |
Array/reductions/sum prod                   (6) |     0.40 |   0.00 |  0.0 |      30.74 |  1443.97 |
JLArray/linalg                              (5) |   237.95 |  64.24 | 27.0 |  310193.91 |  1913.72 |
Array/broadcasting                          (4) |    16.89 |   0.59 |  3.5 |    2163.93 |  1642.48 |
Array/base                                  (2) |    22.55 |   7.00 | 31.0 |    8369.43 |  1479.42 |
JLArray/broadcasting                        (3) |   138.79 |  13.37 |  9.6 |   29606.22 |  1559.17 |
Testing finished in 4 minutes, 31 seconds, 745 milliseconds

Test Summary: |  Pass  Total  Time
  Overall     | 19257  19257
    SUCCESS
     Testing GPUArrays tests passed

After:

Running 5 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
                                                 │          │ ──────────────── CPU ──────────────── │
Test                                    (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
JLArray/linalg/norm                          (5) │    59.42 │   3.09 │  5.2 │   21069.36 │  1069.19 │
JLArray/linalg/core                          (4) │    65.88 │   2.93 │  4.4 │   28870.92 │  1160.56 │
JLArray/broadcasting                         (3) │   102.76 │   4.14 │  4.0 │   31210.01 │   985.23 │
JLArray/reductions/sum prod                  (4) │    41.08 │   1.10 │  2.7 │    9050.56 │  1160.56 │
Array/sparse                                 (5) │    56.39 │   2.09 │  3.7 │    5549.36 │  1149.64 │
JLArray/linalg/kron                          (2) │   126.50 │  35.69 │ 28.2 │  281439.52 │  1312.09 │
JLArray/base                                 (2) │    32.07 │   2.93 │  9.1 │   10910.85 │  1312.09 │
JLArray/linalg/mul!/matrix-matrix            (4) │    70.91 │   4.56 │  6.4 │    7932.93 │  1160.56 │
JLArray/reductions/minimum maximum extrema   (3) │    83.81 │   6.40 │  7.6 │   15226.11 │  1190.20 │
JLArray/linalg/mul!/vector-matrix            (5) │    70.96 │   6.03 │  8.5 │   11755.09 │  1201.41 │
JLArray/reductions/reducedim!                (2) │    34.96 │   2.91 │  8.3 │    5368.52 │  1312.09 │
JLArray/statistics                           (4) │    31.62 │   2.31 │  7.3 │    5396.39 │  1223.30 │
JLArray/indexing multidimensional            (3) │    36.53 │   2.25 │  6.1 │    4658.91 │  1359.70 │
JLArray/math/power                           (2) │    38.51 │   3.32 │  8.6 │    5912.38 │  1312.09 │
JLArray/sparse                               (1) │   251.77 │  12.85 │  5.1 │   39745.49 │  1238.30 │
JLArray/reductions/reduce                    (5) │    56.04 │   5.21 │  9.3 │    7216.90 │  1264.22 │
JLArray/random                               (3) │    28.96 │  10.40 │ 35.9 │   24492.82 │  1749.39 │
Array/broadcasting                           (2) │    20.02 │   1.07 │  5.3 │    2150.61 │  1312.09 │
JLArray/reductions/mapreduce                 (4) │    43.36 │   3.43 │  7.9 │    4818.84 │  1314.19 │
JLArray/reductions/mapreducedim!             (5) │    19.92 │   1.26 │  6.3 │    2680.92 │  1366.38 │
JLArray/constructors                         (1) │    22.82 │   2.31 │ 10.1 │    2523.19 │  1324.73 │
Array/linalg/core                            (2) │    22.04 │   2.40 │ 10.9 │    3272.20 │  1386.05 │
JLArray/reductions/== isequal                (4) │    24.54 │   2.49 │ 10.1 │    1876.42 │  1385.86 │
JLArray/linalg/diagonal                      (3) │    28.02 │   4.08 │ 14.6 │    5015.29 │  1749.39 │
JLArray/ext/jld2                             (1) │    27.92 │   2.34 │  8.4 │    1575.89 │  1326.95 │
JLArray/reductions/mapreducedim!_large       (5) │    34.05 │  19.74 │ 58.0 │   33419.02 │  1731.38 │
Array/indexing multidimensional              (4) │    11.80 │   0.00 │  0.0 │     995.70 │  1385.86 │
JLArray/indexing find                        (2) │    18.85 │   2.10 │ 11.1 │    2445.32 │  1430.20 │
Array/constructors                           (3) │    13.17 │   0.00 │  0.0 │    1105.23 │  1749.39 │
Array/linalg/norm                            (1) │    10.90 │   0.00 │  0.0 │     904.70 │  1437.20 │
Array/indexing find                          (2) │     2.05 │   0.00 │  0.0 │     167.49 │  1430.20 │
Array/ext/jld2                               (4) │     7.48 │   0.00 │  0.0 │     372.85 │  1490.95 │
JLArray/linalg/NaN_false                     (5) │    14.86 │   1.49 │ 10.1 │    1952.90 │  1731.38 │
JLArray/indexing scalar                      (3) │    10.09 │   0.00 │  0.0 │     797.48 │  1749.39 │
Array/linalg/diagonal                        (2) │    10.64 │   0.00 │  0.0 │     742.81 │  1430.20 │
Array/uniformscaling                         (4) │     6.27 │   0.00 │  0.0 │     265.56 │  1490.95 │
Array/linalg/mul!/vector-matrix              (3) │     2.91 │   0.00 │  0.0 │     259.91 │  1749.39 │
JLArray/reductions/any all count             (5) │    11.56 │   0.00 │  0.0 │     867.47 │  1731.38 │
JLArray/alloc cache                          (4) │     1.41 │   0.00 │  0.0 │     138.24 │  1490.95 │
JLArray/uniformscaling                       (2) │     7.45 │   0.00 │  0.0 │     469.79 │  1430.20 │
Array/reductions/== isequal                  (3) │     0.98 │   0.00 │  0.0 │      33.26 │  1749.39 │
Array/linalg/mul!/matrix-matrix              (5) │     3.70 │   0.00 │  0.0 │     277.96 │  1731.38 │
JLArray/math/intrinsics                      (2) │     2.17 │   0.00 │  0.0 │     251.88 │  1430.20 │
Array/random                                 (4) │     3.31 │   0.88 │ 26.6 │    1061.72 │  1541.16 │
Array/base                                   (1) │    44.72 │  19.57 │ 43.8 │    7169.32 │  1627.19 │
Array/linalg/NaN_false                       (3) │     1.14 │   0.00 │  0.0 │     171.83 │  1749.39 │
Array/reductions/minimum maximum extrema     (5) │     0.81 │   0.00 │  0.0 │      54.87 │  1731.38 │
Array/reductions/reduce                      (2) │     0.28 │   0.00 │  0.0 │      35.12 │  1430.20 │
Array/indexing scalar                        (4) │     2.30 │   0.00 │  0.0 │     114.52 │  1541.16 │
Array/vectors                                (1) │     0.25 │   0.00 │  0.0 │      18.85 │  1627.19 │
Array/linalg/kron                            (5) │     0.43 │   0.00 │  0.0 │      33.18 │  1731.38 │
Array/statistics                             (3) │     2.94 │   0.00 │  0.0 │     337.54 │  1749.39 │
Array/reductions/mapreducedim!_large         (2) │     1.58 │   0.68 │ 43.2 │     833.03 │  1430.20 │
Array/reductions/reducedim!                  (4) │     0.45 │   0.00 │  0.0 │      30.59 │  1541.16 │
JLArray/vectors                              (2) │     0.37 │   0.00 │  0.0 │      19.74 │  1430.20 │
Array/alloc cache                            (5) │     1.36 │   0.00 │  0.0 │      79.14 │  1731.38 │
Array/reductions/sum prod                    (1) │     1.31 │   0.00 │  0.0 │      71.08 │  1627.19 │
Array/reductions/mapreducedim!               (3) │     1.37 │   0.00 │  0.0 │      38.47 │  1749.39 │
Array/reductions/any all count               (4) │     0.84 │   0.00 │  0.0 │     116.10 │  1541.16 │
Array/math/intrinsics                        (2) │     0.10 │   0.00 │  0.0 │       8.21 │  1430.20 │
Array/reductions/mapreduce                   (5) │     0.18 │   0.00 │  0.0 │      19.96 │  1731.38 │
Array/math/power                             (1) │     0.01 │   0.00 │  0.0 │       1.70 │  1627.19 │

Test Summary: |  Pass  Total     Time
  Overall     | 24397  24397  7m32.8s
    SUCCESS
     Testing GPUArrays tests passed

The offenders look to be:

JLArray/linalg/kron                          (2) │   126.50 │  35.69 │ 28.2 │  281439.52 │  1312.09 │
JLArray/sparse                               (1) │   251.77 │  12.85 │  5.1 │   39745.49 │  1238.30 │

On CUDA.jl I also noticed the kron memory usage is excessive:

                                                   |          | ---------------- GPU ---------------- | ---------------- CPU ---------------- |
Test                                      (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
gpuarrays/linalg/kron                         (24) |   122.93 |   0.03 |  0.0 |    5381.25 |   406.00 |  12.56 | 10.2 |   32525.51 |  2849.95 |

cc @kshyatt @albertomercurio

[kshyatt]

I'll do some profiling and take a look. One possible culprit is just that we added a lot of tests for the sparse stuff, but probably there are some easy optimizations to do.

[christiangnrd]

Part of this seems to be that when the test runners used Distributed.jl, the initial code would only run once per worker, but now it runs before every test, and loading the GPUArrays test suite seems to be quite heavy.

julia> @time include("test/testsuite.jl")
  3.407996 seconds (4.06 M allocations: 199.437 MiB, 0.92% gc time, 30.60% compilation time: 37% of which was recompilation)

I'll have a PR up in ParallelTestRunner at some point between tonight and this weekend, but preliminary results below. (JuliaTesting/ParallelTestRunner.jl#85) Init time is a testing column I added that subtracts the total worker run time and test record time.

Before:

Running 5 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
                                                 │          │   Init   │ ──────────────── CPU ──────────────── │
Test                                    (Worker) │ Time (s) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
JLArray/linalg/norm                          (5) │    60.79 │     6.67 │   2.90 │  4.8 │   21219.35 │  1219.86 │
JLArray/linalg/core                          (4) │    65.58 │     6.56 │   2.74 │  4.2 │   28990.66 │  1467.30 │
JLArray/broadcasting                         (3) │   100.41 │     6.70 │   3.79 │  3.8 │   31414.54 │  1242.80 │
Array/sparse                                 (4) │    47.86 │     2.72 │   0.39 │  0.8 │    5093.86 │  1680.78 │
JLArray/reductions/minimum maximum extrema   (5) │    56.43 │     2.88 │   1.82 │  3.2 │   15802.80 │  1508.61 │
JLArray/linalg/kron                          (2) │   124.37 │     6.51 │  31.85 │ 25.6 │  281471.12 │  1856.66 │
JLArray/linalg/mul!/matrix-matrix            (3) │    47.79 │     2.90 │   0.87 │  1.8 │    8226.52 │  1547.27 │
JLArray/linalg/mul!/vector-matrix            (4) │    37.72 │     2.69 │   1.38 │  3.7 │   11147.87 │  1891.47 │
JLArray/reductions/sum prod                  (5) │    34.19 │     2.84 │   0.73 │  2.1 │    7656.59 │  1828.47 │
JLArray/reductions/mapreduce                 (2) │    41.59 │     2.57 │   1.09 │  2.6 │   10856.97 │  1856.66 │
JLArray/math/power                           (5) │    24.73 │     2.89 │   0.62 │  2.5 │    5730.06 │  1999.00 │
JLArray/reductions/reduce                    (3) │    31.99 │     2.85 │   0.73 │  2.3 │    6904.63 │  1809.94 │
JLArray/statistics                           (4) │    27.04 │     2.74 │   0.62 │  2.3 │    6994.55 │  2094.81 │
JLArray/base                                 (2) │    21.92 │     2.59 │   1.01 │  4.6 │   10585.47 │  1856.66 │
JLArray/sparse                               (1) │   198.83 │     6.48 │   5.22 │  2.6 │   39955.86 │  1548.36 │
JLArray/constructors                         (2) │    10.63 │     2.65 │   0.13 │  1.3 │    1766.91 │  1856.66 │
JLArray/reductions/mapreducedim!_large       (4) │    18.22 │     2.82 │   5.16 │ 28.3 │   34675.15 │  2695.27 │
JLArray/reductions/mapreducedim!             (5) │    21.55 │     3.01 │   0.36 │  1.7 │    3521.26 │  2004.17 │
JLArray/indexing multidimensional            (3) │    23.21 │     2.95 │   0.49 │  2.1 │    4661.49 │  1976.55 │
JLArray/random                               (1) │    16.73 │     3.04 │   3.62 │ 21.6 │   24505.17 │  1961.06 │
JLArray/reductions/== isequal                (2) │    11.23 │     2.67 │   0.13 │  1.2 │    1865.23 │  1856.66 │
Array/base                                   (4) │    12.08 │     2.87 │   0.62 │  5.2 │    7236.85 │  3083.91 │
JLArray/linalg/diagonal                      (5) │    11.89 │     3.02 │   0.46 │  3.8 │    4607.89 │  2004.17 │
Array/constructors                           (2) │     2.10 │     2.66 │   0.00 │  0.0 │     195.55 │  1856.66 │
Array/broadcasting                           (3) │    11.10 │     2.98 │   0.13 │  1.1 │    1578.73 │  1976.55 │
JLArray/ext/jld2                             (1) │    10.72 │     2.95 │   0.14 │  1.3 │    1643.92 │  2016.86 │
JLArray/indexing find                        (4) │     7.72 │     2.88 │   0.18 │  2.3 │    2418.57 │  3085.56 │
Array/linalg/norm                            (2) │     5.99 │     2.67 │   0.00 │  0.0 │     851.07 │  1871.59 │
Array/linalg/diagonal                        (3) │     5.18 │     3.06 │   0.07 │  1.3 │     888.64 │  1976.55 │
JLArray/linalg/NaN_false                     (5) │     7.31 │     3.02 │   0.15 │  2.0 │    1899.67 │  2004.17 │
Array/ext/jld2                               (2) │     3.87 │     2.70 │   0.00 │  0.0 │     370.00 │  1871.59 │
Array/linalg/core                            (4) │     5.26 │     2.90 │   0.04 │  0.7 │     777.33 │  3085.56 │
JLArray/uniformscaling                       (5) │     3.48 │     3.08 │   0.00 │  0.0 │     526.44 │  2004.17 │
Array/indexing multidimensional              (1) │     6.44 │     3.02 │   0.07 │  1.2 │    1201.22 │  2089.77 │
JLArray/indexing scalar                      (3) │     4.38 │     3.04 │   0.00 │  0.0 │     827.74 │  1976.55 │
JLArray/reductions/any all count             (2) │     2.53 │     2.68 │   0.00 │  0.0 │     556.51 │  1871.59 │
Array/random                                 (3) │     1.29 │     3.03 │   0.05 │  3.9 │    1063.30 │  2369.97 │
Array/uniformscaling                         (4) │     2.55 │     2.88 │   0.00 │  0.0 │     259.49 │  3085.56 │
Array/indexing find                          (5) │     2.36 │     3.08 │   0.00 │  0.0 │     524.40 │  2048.94 │
Array/linalg/mul!/matrix-matrix              (1) │     2.15 │     2.97 │   0.00 │  0.0 │     342.39 │  2089.77 │
Array/indexing scalar                        (3) │     0.61 │     2.99 │   0.00 │  0.0 │      64.91 │  2369.97 │
JLArray/reductions/reducedim!                (2) │     2.15 │     2.68 │   0.00 │  0.0 │     251.89 │  1871.59 │
JLArray/alloc cache                          (5) │     0.74 │     3.10 │   0.00 │  0.0 │     138.51 │  2049.27 │
Array/linalg/mul!/vector-matrix              (1) │     0.77 │     3.03 │   0.00 │  0.0 │     213.59 │  2089.77 │
JLArray/math/intrinsics                      (4) │     1.20 │     2.89 │   0.00 │  0.0 │     253.06 │  3085.56 │
Array/reductions/mapreducedim!               (3) │     0.44 │     3.06 │   0.00 │  0.0 │      41.28 │  2369.97 │
Array/reductions/mapreducedim!_large         (2) │     0.58 │     2.68 │   0.02 │  2.7 │     837.51 │  2317.30 │
Array/reductions/mapreduce                   (4) │     0.33 │     2.98 │   0.00 │  0.0 │      49.27 │  3085.56 │
Array/reductions/== isequal                  (5) │     0.34 │     3.10 │   0.00 │  0.0 │      33.03 │  2049.27 │
Array/reductions/minimum maximum extrema     (1) │     0.72 │     2.99 │   0.00 │  0.0 │      86.76 │  2089.77 │
Array/math/power                             (2) │     0.01 │     2.68 │   0.00 │  0.0 │       1.69 │  2317.97 │
Array/linalg/kron                            (3) │     0.14 │     3.06 │   0.00 │  0.0 │      33.46 │  2369.97 │
Array/alloc cache                            (5) │     0.03 │     3.08 │   0.00 │  0.0 │      30.76 │  2049.27 │
Array/reductions/reducedim!                  (4) │     0.43 │     2.91 │   0.00 │  0.0 │      29.46 │  3085.56 │
Array/math/intrinsics                        (1) │     0.09 │     3.01 │   0.00 │  0.0 │       8.41 │  2089.77 │
Array/reductions/any all count               (2) │     0.13 │     2.69 │   0.00 │  0.0 │      46.58 │  2317.98 │
Array/reductions/sum prod                    (3) │     0.24 │     3.03 │   0.00 │  0.0 │      31.42 │  2369.97 │
Array/reductions/reduce                      (5) │     0.17 │     3.09 │   0.00 │  0.0 │      23.77 │  2049.27 │
Array/vectors                                (1) │     0.13 │     3.02 │   0.00 │  0.0 │      18.93 │  2089.77 │
Array/linalg/NaN_false                       (4) │     0.45 │     2.89 │   0.00 │  0.0 │     120.64 │  3085.56 │
JLArray/vectors                              (2) │     0.10 │     2.68 │   0.00 │  0.0 │      17.91 │  2319.34 │
Array/statistics                             (3) │     0.75 │     2.96 │   0.00 │  0.0 │     338.85 │  2369.97 │

Test Summary: |  Pass  Total     Time
  Overall     | 24397  24397  4m31.9s
    SUCCESS     CPU Time: 1339.59s; CPU Test Time: 1141.48s
     Testing GPUArrays tests passed 

After:

Running 5 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
                                                 │          │   Init   │ ──────────────── CPU ──────────────── │
Test                                    (Worker) │ Time (s) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
JLArray/linalg/norm                          (5) │    60.77 │     6.70 │   2.90 │  4.8 │   21219.20 │  1186.48 │
JLArray/linalg/core                          (4) │    66.79 │     6.62 │   2.76 │  4.1 │   28990.74 │  1504.00 │
JLArray/broadcasting                         (3) │   100.59 │     6.69 │   3.96 │  3.9 │   31414.82 │  1287.94 │
JLArray/linalg/mul!/matrix-matrix            (4) │    47.28 │     0.55 │   0.66 │  1.4 │    7969.23 │  1711.94 │
JLArray/reductions/minimum maximum extrema   (5) │    56.64 │     0.66 │   1.85 │  3.3 │   15793.78 │  1495.86 │
JLArray/linalg/kron                          (2) │   124.49 │     6.54 │  31.98 │ 25.7 │  281471.17 │  1918.14 │
Array/sparse                                 (3) │    47.55 │     0.71 │   0.49 │  1.0 │    5069.70 │  1549.81 │
JLArray/reductions/mapreduce                 (4) │    37.85 │     0.62 │   0.85 │  2.2 │    9337.92 │  1975.55 │
JLArray/linalg/mul!/vector-matrix            (5) │    40.32 │     0.80 │   1.37 │  3.4 │   11832.83 │  1754.52 │
JLArray/reductions/sum prod                  (2) │    41.00 │     0.40 │   1.07 │  2.6 │   10470.40 │  1918.14 │
JLArray/statistics                           (4) │    21.47 │     0.71 │   0.51 │  2.4 │    5506.17 │  2195.08 │
JLArray/reductions/reduce                    (3) │    31.42 │     0.77 │   0.70 │  2.2 │    6989.61 │  1803.95 │
JLArray/math/power                           (5) │    24.54 │     0.86 │   0.54 │  2.2 │    5315.29 │  1949.50 │
JLArray/indexing multidimensional            (2) │    21.78 │     0.52 │   0.43 │  2.0 │    4703.57 │  1918.14 │
JLArray/reductions/mapreducedim!             (4) │    12.04 │     0.79 │   0.20 │  1.6 │    2681.13 │  2322.84 │
JLArray/sparse                               (1) │   197.24 │     6.48 │   5.22 │  2.6 │   39956.33 │  1578.22 │
Array/base                                   (4) │    12.39 │     0.90 │   0.66 │  5.3 │    7258.83 │  2439.56 │
JLArray/reductions/mapreducedim!_large       (5) │    17.82 │     0.91 │   5.32 │ 29.9 │   34313.55 │  2971.62 │
JLArray/base                                 (3) │    22.36 │     0.90 │   0.91 │  4.1 │   10360.93 │  2189.92 │
JLArray/random                               (2) │    16.18 │     0.56 │   2.96 │ 18.3 │   24509.22 │  1918.14 │
JLArray/linalg/diagonal                      (1) │    13.10 │     0.93 │   0.65 │  5.0 │    4993.38 │  1647.56 │
JLArray/ext/jld2                             (3) │     9.43 │     0.93 │   0.07 │  0.8 │    1289.76 │  2189.92 │
Array/broadcasting                           (4) │    12.48 │     0.86 │   0.14 │  1.1 │    2055.64 │  2439.56 │
JLArray/reductions/== isequal                (5) │    11.85 │     0.94 │   0.16 │  1.4 │    2007.77 │  2971.62 │
JLArray/constructors                         (2) │    10.79 │     0.60 │   0.14 │  1.3 │    1921.89 │  1918.14 │
Array/linalg/norm                            (5) │     0.62 │     0.94 │   0.00 │  0.0 │      54.51 │  2971.62 │
JLArray/indexing find                        (1) │     8.09 │     0.94 │   0.23 │  2.8 │    2459.47 │  1822.34 │
Array/indexing multidimensional              (4) │     5.79 │     0.88 │   0.00 │  0.0 │     923.98 │  2439.56 │
Array/linalg/diagonal                        (2) │     5.71 │     0.64 │   0.05 │  0.9 │    1009.88 │  1918.14 │
JLArray/linalg/NaN_false                     (3) │     7.44 │     0.94 │   0.15 │  2.0 │    1907.19 │  2189.92 │
JLArray/indexing scalar                      (1) │     3.03 │     0.95 │   0.00 │  0.0 │     524.28 │  1832.42 │
Array/indexing find                          (3) │     2.40 │     0.97 │   0.00 │  0.0 │     599.14 │  2189.92 │
Array/ext/jld2                               (4) │     4.23 │     0.87 │   0.00 │  0.0 │     376.03 │  2439.56 │
JLArray/uniformscaling                       (2) │     3.70 │     0.65 │   0.00 │  0.0 │     615.52 │  1918.14 │
Array/linalg/mul!/matrix-matrix              (4) │     0.22 │     0.90 │   0.00 │  0.0 │      19.21 │  2439.56 │
Array/uniformscaling                         (1) │     2.67 │     0.96 │   0.00 │  0.0 │     287.56 │  1842.45 │
Array/linalg/core                            (5) │    11.05 │     0.97 │   0.29 │  2.7 │    3360.42 │  2971.62 │
Array/random                                 (1) │     1.44 │     0.99 │   0.06 │  4.0 │    1086.15 │  2385.69 │
JLArray/reductions/any all count             (3) │     2.71 │     1.02 │   0.00 │  0.0 │     588.61 │  2189.92 │
JLArray/alloc cache                          (1) │     0.75 │     0.99 │   0.00 │  0.0 │     142.24 │  2385.69 │
JLArray/math/intrinsics                      (5) │     1.12 │     1.03 │   0.00 │  0.0 │     248.64 │  2971.62 │
Array/linalg/mul!/vector-matrix              (3) │     1.10 │     0.98 │   0.00 │  0.0 │     264.96 │  2189.92 │
Array/reductions/minimum maximum extrema     (5) │     0.01 │     1.00 │   0.00 │  0.0 │       3.86 │  2971.62 │
Array/statistics                             (1) │     0.92 │     1.00 │   0.00 │  0.0 │     369.45 │  2385.69 │
Array/indexing scalar                        (3) │     0.98 │     0.99 │   0.00 │  0.0 │      89.00 │  2189.92 │
Array/reductions/mapreducedim!               (5) │     0.47 │     1.02 │   0.00 │  0.0 │      41.45 │  2971.62 │
Array/reductions/== isequal                  (1) │     0.59 │     0.99 │   0.00 │  0.0 │      61.32 │  2385.69 │
Array/reductions/reducedim!                  (3) │     0.54 │     1.01 │   0.00 │  0.0 │      28.20 │  2189.92 │
Array/constructors                           (4) │     7.38 │     0.92 │   0.08 │  1.1 │    1068.79 │  2439.56 │
Array/linalg/NaN_false                       (5) │     0.51 │     1.04 │   0.00 │  0.0 │     122.18 │  2971.62 │
Array/reductions/mapreduce                   (1) │     0.19 │     1.04 │   0.00 │  0.0 │      36.10 │  2385.69 │
Array/reductions/sum prod                    (3) │     0.26 │     1.02 │   0.00 │  0.0 │      33.91 │  2189.92 │
Array/reductions/reduce                      (5) │     0.18 │     1.10 │   0.00 │  0.0 │      26.93 │  2971.62 │
Array/reductions/mapreducedim!_large         (4) │     0.57 │     0.94 │   0.02 │  3.0 │     828.85 │  2623.83 │
Array/linalg/kron                            (1) │     0.24 │     1.06 │   0.00 │  0.0 │      39.31 │  2385.69 │
Array/vectors                                (3) │     0.13 │     1.02 │   0.00 │  0.0 │      18.20 │  2189.92 │
Array/math/intrinsics                        (4) │     0.10 │     0.91 │   0.00 │  0.0 │       7.96 │  2624.06 │
JLArray/reductions/reducedim!                (2) │    10.97 │     0.68 │   0.06 │  0.5 │    1147.42 │  1918.14 │
Array/alloc cache                            (5) │     0.27 │     1.08 │   0.00 │  0.0 │      82.50 │  2971.62 │
Array/reductions/any all count               (1) │     0.27 │     1.03 │   0.00 │  0.0 │     114.91 │  2385.69 │
JLArray/vectors                              (3) │     0.13 │     0.98 │   0.00 │  0.0 │      12.58 │  2189.92 │
Array/math/power                             (4) │     0.01 │     0.88 │   0.00 │  0.0 │       1.12 │  2624.06 │

Test Summary: |  Pass  Total     Time
  Overall     | 24397  24397  4m07.8s
    SUCCESS     CPU Time: 1228.25s; CPU Test Time: 1144.65s
     Testing GPUArrays tests passed