Skip to content

[Feature]: Add S3 Support for CO2 Footprint Report Output #309

@chris-rands

Description

@chris-rands

Describe the feature

Thank you again for this plug-in!

Summary

The nf-co2footprint plugin currently does not support writing CO2 footprint reports directly to S3 paths. When S3 URLs are specified in the file configuration parameters, the plugin appears to write to local filesystem instead of the specified S3 location.

Current Behavior

When configuring the plugin with S3 paths:

co2summary {
    enabled = true
    file    = "s3://my-bucket/pipeline_info/co2footprint_summary_${trace_timestamp}.txt"
}
co2trace {
    enabled = true
    file    = "s3://my-bucket/pipeline_info/co2footprint_trace_${trace_timestamp}.txt"
}
co2report {
    enabled = true
    file    = "s3://my-bucket/pipeline_info/co2footprint_report_${trace_timestamp}.html"
}

The plugin generates files locally in the working directory instead of writing to the specified S3 paths.

Expected Behavior

The plugin should support S3 paths and write CO2 footprint reports directly to the specified S3 locations, similar to how Nextflow's built-in reporting features (timeline, report, trace, dag) handle S3 paths when params.outdir is an S3 URL.

Environment

  • Plugin Version: 1.0.0
  • Nextflow Version: 25.04.6
  • Cloud Platform: AWS Batch
  • Storage: S3

Configuration Example

// In nextflow.config
plugins {
    id 'nf-co2footprint@1.0.0'
}

params {
    outdir = "s3://my-bucket/pipeline-results"
}

co2summary {
    enabled = true
    file    = "${params.outdir}/pipeline_info/co2footprint_summary_${trace_timestamp}.txt"
}
co2trace {
    enabled = true
    file    = "${params.outdir}/pipeline_info/co2footprint_trace_${trace_timestamp}.txt"
}
co2report {
    enabled = true
    file    = "${params.outdir}/pipeline_info/co2footprint_report_${trace_timestamp}.html"
}

This should write the CO2 footprint files directly to the S3 bucket at the specified paths.

Use case(s):

This feature is essential for cloud-native pipelines where:

  1. All pipeline outputs are stored in S3
  2. Local filesystem storage is limited or temporary
  3. CO2 footprint reports need to be archived alongside other pipeline artifacts

Suggested implementation:

Proposed Implementation

The plugin could leverage Nextflow's built-in file system abstraction (FileHelper.asPath()) to support S3 paths natively, similar to how other Nextflow features handle cloud storage.

Workaround

Currently using a workaround that generates files locally and syncs to S3 in workflow.onComplete, but native S3 support would be more elegant and consistent with Nextflow's cloud-first approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions