Running python tests at the same time as async can cause memory issues

[pechersky]

Built with CUDA 13, CCCL 3.0, rdkit (pip installed) 2025.9.1, built at tag v0.2.0

Test command

python3.12 -m pytest --pyargs /app/nvMolKit/nvmolkit/tests

When running with -k "not async", many of the tests below pass.

Example error

/app/nvMolKit/nvmolkit/tests/test_types.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

device = None

    def synchronize(device: "Device" = None) -> None:
        r"""Wait for all kernels in all streams on a CUDA device to complete.
    
        Args:
            device (torch.device or int, optional): device for which to synchronize.
                It uses the current device, given by :func:`~torch.cuda.current_device`,
                if :attr:`device` is ``None`` (default).
        """
        _lazy_init()
        with torch.cuda.device(device):
>           return torch._C._cuda_synchronize()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E           torch.AcceleratorError: CUDA error: an illegal memory access was encountered
E           Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
E           CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
E           For debugging consider passing CUDA_LAUNCH_BLOCKING=1
E           Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

/opt/python/cp312-cp312/lib/python3.12/site-packages/torch/cuda/__init__.py:1083: AcceleratorError

All tests that failed:

FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-0-gpu_ids1] - AssertionError: Molecule 0, Conformer 0: energy mismatch: RDKit=26.874311, nvMolKit=125669.641792, abs_diff=125642.767481, rel_error=4675.199514
FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-2-gpu_ids1] - RuntimeError: Encountered CUDA error 101: invalid device ordinal
FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-5-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-0-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-2-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-5-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_allows_large_molecule_interleaved - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_mmff_optimization.py::test_error_case_throws_properly - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_cross_similarity_fp_mismatch[tanimoto] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_cross_similarity_fp_mismatch[cosine] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nvmolkit_cross_tanimoto_similarity_from_nvmolkit_fp - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims3] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_packing[nxmdims0] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_packing[nxmdims1] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_packing[nxmdims2] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nvmolkit_cross_cosine_similarity_from_nvmolkit_fp - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims3] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims4] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims5] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_self - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_cross[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_cross[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_cross[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_self - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_cross[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_cross[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_cross[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_segmented_path_large_cross[tanimoto] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_similarity.py::test_memory_constrained_segmented_path_large_cross[cosine] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED ../app/nvMolKit/nvmolkit/tests/test_types.py::test_async_gpu_result_release_frees_memory - torch.AcceleratorError: CUDA error: an illegal memory access was encountered

nvidia-smi:

$ nvidia-smi
Tue Oct 21 03:25:06 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.02              Driver Version: 581.42         CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4500 Laptop GPU    On  |   00000000:01:00.0 Off |                  Off |
| N/A   50C    P0             33W /   91W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |

[scal444]

Thanks! We should maybe add a generic cleanup to clear out the torch cache so that one failure doesn't pollute all the other tests.

I suspect the device ordinal issue is the root cause and then everything else is garbage. I think this may also be related to the MMFF issue in #29

[scal444]

@pechersky can you describe your hardware system? E.g. are you on a machine with multiple GPUs where you only have a view into one of them via vm or docker settings or env var, or is it actually just one device on the machine. Wondering if the bug might be upstream of the business logic, if we're accidentally targeting GPU 0 when the single GPU is GPU 1, or something like that.

[pechersky]

I actually have just one device on the machine where I ran these tests.

[scal444]

Thanks, one more question -

rdkit (pip installed) 2025.9.1, so are you using the "Building Against pip-installed RDKit" instructions? If possible, could you try to reproduce with the new conda packaging?

I'll try to reproduce your environment in the meantime.

[pechersky]

I followed the instructions in the README, but needed to have some modifications to make it work with conda.

On v0.2.0, needed to add the following code:

diff --git i/CMakeLists.txt w/CMakeLists.txt
index 416ffc8..ef3f5af 100644
--- i/CMakeLists.txt
+++ w/CMakeLists.txt
@@ -18,6 +18,23 @@ project(
   LANGUAGES CXX CUDA)
 set(CMAKE_CUDA_STANDARD 17)
 set(CMAKE_CXX_STANDARD 17)
+
+# Fetch CCCL for updated CUB with TransformReduce support
+include(FetchContent)
+FetchContent_Declare(
+  CCCL
+  GIT_REPOSITORY https://github.com/NVIDIA/cccl.git
+  GIT_TAG        v2.7.0
+  GIT_SHALLOW    TRUE
+)
+set(CCCL_ENABLE_TESTING OFF CACHE BOOL "" FORCE)
+set(CCCL_ENABLE_EXAMPLES OFF CACHE BOOL "" FORCE)
+set(CCCL_ENABLE_BENCHMARKS OFF CACHE BOOL "" FORCE)
+FetchContent_MakeAvailable(CCCL)
+include_directories(BEFORE SYSTEM ${cccl_SOURCE_DIR}/cub)
+include_directories(BEFORE SYSTEM ${cccl_SOURCE_DIR}/thrust)
+include_directories(BEFORE SYSTEM ${cccl_SOURCE_DIR}/libcudacxx/include)
+
 find_package(CUDAToolkit REQUIRED)
 
 set(CMAKE_MODULE_PATH "${CMAKE_MODULE_PATH};${CMAKE_CURRENT_SOURCE_DIR}/cmake")

Running in image 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:2.6.0-gpu-py312-cu124-ubuntu22.04-sagemaker:

cd
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash ~/Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
conda config --set always_yes true
conda create --name nvmolkit_dev_py312 python=3.12.1
conda activate nvmolkit_dev_py312
conda install -c conda-forge rdkit=2024.09.6 rdkit-dev=2024.09.6
conda install -c conda-forge libboost libboost-python libboost-devel libboost-headers libboost-python-devel
pip install torch torchvision torchaudio
python -c "import torch; print(torch.__version__); print(f'Is a CUDA build? {torch.cuda.is_available()}')"
cd /nvmolkit
conda install -c conda-forge cmake eigen  # this is not in README
CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) pip -v install .

test run:

$ pytest nvmolkit/tests -k "not async"
[...]
=============================================================================================== short test summary info ===============================================================================================
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-0-gpu_ids1] - AssertionError: Molecule 0, Conformer 0: energy mismatch: RDKit=26.874311, nvMolKit=26.939253, abs_diff=0.064942, rel_error=0.002416
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-2-gpu_ids1] - AssertionError: Molecule 3, Conformer 0: energy mismatch: RDKit=-207.435992, nvMolKit=-206.847232, abs_diff=0.588760, rel_error=0.002838
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-5-gpu_ids1] - AssertionError: Molecule 0, Conformer 0: energy mismatch: RDKit=26.874311, nvMolKit=26.939253, abs_diff=0.064942, rel_error=0.002416
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-0-gpu_ids1] - AssertionError: Molecule 2, Conformer 0: energy mismatch: RDKit=-18.732622, nvMolKit=-17.874977, abs_diff=0.857646, rel_error=0.045784
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-5-gpu_ids1] - AssertionError: Molecule 2, Conformer 0: energy mismatch: RDKit=-18.732622, nvMolKit=-17.874977, abs_diff=0.857646, rel_error=0.045784
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_allows_large_molecule_interleaved - AssertionError: Molecule 0, Conformer 0: energy mismatch: RDKit=0.401026, nvMolKit=9.277273, abs_diff=8.876246, rel_error=22.133825
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_self - AssertionError: 
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_self - AssertionError: 
========================================================================== 8 failed, 74 passed, 28 skipped, 1 deselected in 88.09s (0:01:28) ==========================================================================

which matches what I got before.

If I don't do not async, then after a few tests, all remaining tests fail with

=============================================================================================== short test summary info ===============================================================================================
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-0-gpu_ids1] - AssertionError: Molecule 0, Conformer 0: energy mismatch: RDKit=26.874311, nvMolKit=26.939253, abs_diff=0.064942, rel_error=0.002416
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-2-gpu_ids1] - RuntimeError: Encountered CUDA error 101: invalid device ordinal
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[1-5-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-0-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-2-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_batch_vs_rdkit[3-5-gpu_ids1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_mmff_optimization.py::test_mmff_optimization_allows_large_molecule_interleaved - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_mmff_optimization.py::test_error_case_throws_properly - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_cross_similarity_fp_mismatch[tanimoto] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_cross_similarity_fp_mismatch[cosine] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nvmolkit_cross_tanimoto_similarity_from_nvmolkit_fp - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_nvmolkit_fp[nxmdims3] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_packing[nxmdims0] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_packing[nxmdims1] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_tanimoto_similarity_from_packing[nxmdims2] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nvmolkit_cross_cosine_similarity_from_nvmolkit_fp - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims3] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims4] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_nxm_cross_cosine_similarity_from_nvmolkit_fp[nxmdims5] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_self - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_cross[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_cross[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_tanimoto_cross[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_self - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_cross[nxmdims0] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_cross[nxmdims1] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_cosine_cross[nxmdims2] - RuntimeError: Encountered CUDA error 700: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_segmented_path_large_cross[tanimoto] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
FAILED nvmolkit/tests/test_similarity.py::test_memory_constrained_segmented_path_large_cross[cosine] - torch.AcceleratorError: CUDA error: an illegal memory access was encountered
========================================================================= 35 failed, 49 passed, 26 skipped, 1 deselected in 78.74s (0:01:18) ==========================================================================

[scal444]

Didn't mean to close this, only partially addressed

[scal444]

See comment in #29 , hoping this issue is resolved but was unable to fully reproduce the bug.