Add GPU acceleration, crash recovery, execution metrics, and rootless container build infrastructure to nf-pySpade pipeline

[Copilot]

Optimize the nf-pySpade pipeline with GPU acceleration for hypergeometric tests in DEobs/DErand processes, enable automatic crash recovery, add comprehensive execution reporting for SLURM deployments, and provide container build infrastructure with rootless Podman support for custom deployments.

Changes

GPU Acceleration

• DEobs, DEobsFDR, DErand processes: Added GPU queue priority, --gres=gpu:1 resource allocation, and --use-gpu flag
• Set CUDA_VISIBLE_DEVICES for GPU visibility
• Queue priority: GPU,GPUp40,GPUp100,256GB,256GBv1,384GB,512GB (prioritizes GPU partitions, falls back to memory-based queues if unavailable)

process pySpadeDEobs {
    queue 'GPU,GPUp40,GPUp100,256GB,256GBv1,384GB,512GB'
    clusterOptions = '--gres=gpu:1'
    script:
        """
        export CUDA_VISIBLE_DEVICES=\${CUDA_VISIBLE_DEVICES:-0}
        pySpade DEobs ... --use-gpu
        """
}

Crash Recovery

• resume = true in config, -resume flag in execution script
• publishDir on all 14 processes with mode: 'copy', overwrite: false
• Pipeline automatically continues from last successful task after interruption

Execution Metrics & Reporting

• report.html: Execution statistics per task
• timeline.html: Visual timeline of parallel execution
• trace.txt: CPU%, memory%, I/O, duration, queue time for each task
• dag.svg: Workflow dependency graph

Infrastructure

• Work directory: ./work for all Nextflow caching and staging
• Container: Updated to docker://igvf/pyspade:pyspade_0.1.7 with auto-pull (60min timeout)
• .gitignore: Excludes work/, logs, reports, container cache

Container Build Infrastructure

• Dockerfile: Build custom container with helper scripts included at /opt/nf-pyspade/script/
• Singularity.def: HPC-compatible Singularity container definition
• build_container.sh: Automated build script supporting Docker, Podman, and Singularity with auto-detection
• .dockerignore: Optimized build context excluding temporary files
• GitHub Actions workflow: Optional CI/CD template for automated builds

Build custom containers with:

./build_container.sh docker      # Auto-detects Docker/Podman
./build_container.sh podman      # Explicit Podman (rootless)
./build_container.sh singularity # Singularity image
./build_container.sh both        # Both formats

The custom container includes all helper scripts (calculate_FDR.py, filtered_local_df.py, find_DErand_range.py, randomized_sgrna.py) bundled at /opt/nf-pyspade/script/ with scripts added to PATH for easy access.

Rootless Container Builds with Podman

• Podman support: Build containers without root access or Docker daemon
• Auto-detection: Build script automatically prefers Podman for rootless environments
• HPC-friendly: Ideal for systems where Docker isn't available or root access is restricted
• Full compatibility: Uses same Dockerfile, fully compatible with Docker commands
• Comprehensive workflows: Documentation includes Podman installation, HPC transfer strategies, and Singularity conversion

Build rootless with Podman:

podman build -t nf-pyspade:0.1.7-nf .
podman save -o nf-pyspade_0.1.7-nf.tar nf-pyspade:0.1.7-nf
# Transfer to HPC and load or convert to Singularity

Documentation

• OPTIMIZATION_NOTES.md: Usage guide, troubleshooting, file organization
• GPU_IMPLEMENTATION_NOTES.md: GPU setup verification, rollback instructions
• CONTAINER_BUILD.md: Complete container build guide for Docker, Podman, and Singularity with rootless build workflows
• README.md: Updated with optimization overview and container build instructions

Notes

⚠️ GPU implementation assumes pySpade 0.1.7 supports --use-gpu flag. If unsupported, remove flag from three processes—GPU resources still allocated for parallelism. See GPU_IMPLEMENTATION_NOTES.md for removal instructions.

Original prompt

I want to optimize the code on the dev_mn branch. Never merge with the main branch. I want you to use GPU acceleration for the hypergeometric test (DEobs and DErand) , then update the nextflow script to generate the report at the end of the run, and also to continue where the code left off if there is a crash. This will be running in slurm. make sure the working directory is in the work/ folder for the python funcitons. the final nextflow report should show time and compute needed to run each step

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[mnzima]

@copilot to be clear, the GPU options on my SLURM system are: GPU, GPUp40, GPUp100

[Copilot]

@copilot to be clear, the GPU options on my SLURM system are: GPU, GPUp40, GPUp100

Updated the GPU queue configuration to match your SLURM partitions. The three GPU-enabled processes (DEobs, DEobsFDR, and DErand) now use:

queue 'GPU,GPUp40,GPUp100,256GB,256GBv1,384GB,512GB'

This prioritizes your GPU partitions (GPU, GPUp40, GPUp100) and falls back to memory-based queues if GPUs are unavailable. Documentation updated in OPTIMIZATION_NOTES.md and GPU_IMPLEMENTATION_NOTES.md.

Commit: aacc85a

[mnzima]

@copilot create a container image with the updated codebase

[Copilot]

@copilot create a container image with the updated codebase

Created container build infrastructure for nf-pySpade. You can now build custom images with the helper scripts included:

Quick Build:

./build_container.sh docker    # Docker image
./build_container.sh singularity # Singularity image
./build_container.sh both       # Both formats

Files Added:

• Dockerfile - Docker container definition
• Singularity.def - Singularity definition for HPC
• build_container.sh - Automated build script
• CONTAINER_BUILD.md - Complete documentation

The container includes all helper scripts at /opt/nf-pyspade/script/ and is based on igvf/pyspade:pyspade_0.1.7.

See CONTAINER_BUILD.md for detailed instructions on building, testing, and deploying to Docker Hub or HPC systems.

Commit: 36737bc

[mnzima]

@copilot manually build the container here in github so that it can be pulled manually

[Copilot]

@copilot manually build the container here in github so that it can be pulled manually

I cannot directly build and push containers from this environment, but I can set up GitHub Actions to automatically build and publish the container to GitHub Container Registry (GHCR).

Would you like me to:

1. Enable the GitHub Actions workflow (currently commented out) to build on push/merge
2. Update it to publish to ghcr.io/hon-lab/nf-pyspade
3. Add instructions for pulling from GHCR

Alternatively, you can manually build and push using the existing files:

docker build -t ghcr.io/hon-lab/nf-pyspade:0.1.7-nf .
docker push ghcr.io/hon-lab/nf-pyspade:0.1.7-nf

Let me know which approach you prefer, or if you'd like me to activate the automated GitHub Actions workflow.

[mnzima]

@copilot for the container code, create a version that can be built with podman since i do not have root access

[Copilot]

@copilot for the container code, create a version that can be built with podman since i do not have root access

Added comprehensive Podman support for rootless container builds (no root access required).

Quick Start:

# Build with Podman (rootless)
./build_container.sh podman

# Or auto-detect (prefers Podman)
./build_container.sh docker

Manual Build:

podman build -t nf-pyspade:0.1.7-nf .
podman run --rm nf-pyspade:0.1.7-nf pySpade --help

Key Features:

• No root/sudo required
• No daemon process
• Fully compatible with Dockerfile
• Same commands as Docker
• Perfect for HPC systems

Transfer to HPC:

# Save as tar
podman save -o nf-pyspade_0.1.7-nf.tar nf-pyspade:0.1.7-nf

# On HPC: Load or convert to Singularity
podman load -i nf-pyspade_0.1.7-nf.tar
# OR
singularity build nf-pyspade.sif docker-archive://nf-pyspade_0.1.7-nf.tar

See CONTAINER_BUILD.md for complete documentation including installation, HPC workflows, and troubleshooting.

Commit: b173a96