-
Notifications
You must be signed in to change notification settings - Fork 11
Description
Describe the feature
Thank you again for this plug-in!
Summary
The nf-co2footprint plugin currently does not support writing CO2 footprint reports directly to S3 paths. When S3 URLs are specified in the file configuration parameters, the plugin appears to write to local filesystem instead of the specified S3 location.
Current Behavior
When configuring the plugin with S3 paths:
co2summary {
enabled = true
file = "s3://my-bucket/pipeline_info/co2footprint_summary_${trace_timestamp}.txt"
}
co2trace {
enabled = true
file = "s3://my-bucket/pipeline_info/co2footprint_trace_${trace_timestamp}.txt"
}
co2report {
enabled = true
file = "s3://my-bucket/pipeline_info/co2footprint_report_${trace_timestamp}.html"
}The plugin generates files locally in the working directory instead of writing to the specified S3 paths.
Expected Behavior
The plugin should support S3 paths and write CO2 footprint reports directly to the specified S3 locations, similar to how Nextflow's built-in reporting features (timeline, report, trace, dag) handle S3 paths when params.outdir is an S3 URL.
Environment
- Plugin Version: 1.0.0
- Nextflow Version: 25.04.6
- Cloud Platform: AWS Batch
- Storage: S3
Configuration Example
// In nextflow.config
plugins {
id 'nf-co2footprint@1.0.0'
}
params {
outdir = "s3://my-bucket/pipeline-results"
}
co2summary {
enabled = true
file = "${params.outdir}/pipeline_info/co2footprint_summary_${trace_timestamp}.txt"
}
co2trace {
enabled = true
file = "${params.outdir}/pipeline_info/co2footprint_trace_${trace_timestamp}.txt"
}
co2report {
enabled = true
file = "${params.outdir}/pipeline_info/co2footprint_report_${trace_timestamp}.html"
}This should write the CO2 footprint files directly to the S3 bucket at the specified paths.
Use case(s):
This feature is essential for cloud-native pipelines where:
- All pipeline outputs are stored in S3
- Local filesystem storage is limited or temporary
- CO2 footprint reports need to be archived alongside other pipeline artifacts
Suggested implementation:
Proposed Implementation
The plugin could leverage Nextflow's built-in file system abstraction (FileHelper.asPath()) to support S3 paths natively, similar to how other Nextflow features handle cloud storage.
Workaround
Currently using a workaround that generates files locally and syncs to S3 in workflow.onComplete, but native S3 support would be more elegant and consistent with Nextflow's cloud-first approach.