Cellranger arc running out of disk space #436
Description
I'm running the latest version of Cumulus (3.0.0). I was able to successfully run CellRanger arc on some of my samples, but two of them have been crashing, it appears because they do not have enough disk space.
I have rerun setting disk space and memory very high:
arc_disk_space 1250
arc_memory "400G"
atac_disk_space 1250
atac_memory "250G"
count_disk_space 1000
However they are still crashing (after running for longer) and this is getting to be very expensive.
Any advice?
Here is the end of the log file:
2025-04-04 13:25:19 [runtime] (update) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS.fork0 chunks running (144/239 completed)
2025-04-04 13:30:33 [runtime] (update) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS.fork0 chunks running (144/239 completed)
2025-04-04 13:35:43 [runtime] (update) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS.fork0 chunks running (144/239 completed)
2025-04-04 13:40:45 [runtime] (update) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS.fork0 chunks running (144/239 completed)
2025-04-04 13:45:52 [runtime] (update) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS.fork0 chunks running (145/239 completed)
2025-04-04 13:50:56 [runtime] (update) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS.fork0 chunks running (150/239 completed)
2025-04-04 13:56:08 [runtime] (update) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS.fork0 chunks running (156/239 completed)
2025-04-04 13:58:20 [runtime] Pipestance directory out of disk space.
/cromwell_root/results/SC_ATAC_GEX_COUNTER_CS has only 9004kB remaining space available.
To ignore this error, set MRO_DISK_SPACE_CHECK=disable in your environment.
2025-04-04 13:58:23 [runtime] (failed) ID.results.SC_ATAC_GEX_COUNTER_CS.SC_ATAC_GEX_COUNTER._ATAC_MATRIX_COMPUTER.ALIGN_ATAC_READS
[error] Pipestance failed. Error log at:
results/SC_ATAC_GEX_COUNTER_CS/SC_ATAC_GEX_COUNTER/_ATAC_MATRIX_COMPUTER/ALIGN_ATAC_READS/fork0/chnk158-u0a3bef7605/_errors
Log message:
/cromwell_root/results/SC_ATAC_GEX_COUNTER_CS has only 9004kB remaining space available.
To ignore this error, set MRO_DISK_SPACE_CHECK=disable in your environment.
Waiting 6 seconds for UI to do final refresh.
Pipestance failed. Use --noexit option to keep UI running after failure.
2025-04-04 13:58:29 Shutting down.
cellranger-arc count --id=results --libraries=libraries.csv --reference=genome_dir --jobmode=local --localcores=64 --localmem=500
Traceback (most recent call last):
File "", line 40, in
tee: /cromwell_root/stderr: No space left on device
File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cellranger-arc', 'count', '--id=results', '--libraries=libraries.csv', '--reference=genome_dir', '--jobmode=local', '--localcores=64', '--localmem=500']' returned non-zero exit status 1.
/cromwell_root/script: line 78: echo: write error: No space left on device
sh: write error: No space left on device