Make pyproject.toml single source of truth for dependencies with version pinning

[Copilot]

Unpinned dependencies in requirements.txt caused production docker builds to fail unpredictably when upstream packages released breaking changes. Dependencies were scattered across setup.py, requirements.txt, requirements_dev.txt, environment.yml, and environment_dev.yml with no clear ownership.

Changes

Dependency consolidation

• Migrated all dependencies to pyproject.toml following PEP 621 standards
• Added version constraints for all packages (e.g., numpy>=1.24.0,<3.0) to prevent breaking upgrades
• Organized into logical groups: core, performance, geo, io, analysis, dev, docs
• Simplified setup.py to only handle Cython extensions; metadata now read from pyproject.toml

Automation

• Created scripts/generate_requirements.py to generate requirements.txt and environment.yml from pyproject.toml
• Added pre-commit hook to regenerate files when pyproject.toml changes
• Added CI workflow to validate generated files stay in sync
• Configured Dependabot for monthly dependency monitoring

Generated files (now auto-generated, do not edit manually)

• requirements.txt, requirements_dev.txt
• environment.yml, environment_dev.yml

Usage

Update dependencies:

# Edit pyproject.toml
vim pyproject.toml

# Regenerate all requirement files
python scripts/generate_requirements.py

# Commit both
git add pyproject.toml requirements*.txt environment*.yml

Install with optional features:

pip install .[all]          # All optional dependencies
pip install .[performance]  # Just dask, pyfftw
pip install -e .[dev]       # Development mode

See doc/source/developer_guide/dependency_management.rst for detailed documentation.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>no versions specified in requirements.txt cause unpredictable failures due to dependency upgrades.</issue_title>
<issue_description>I had made some modifications to our operational pysteps, tested the new version and this worked fine in our staging environment. Our production team created a new docker image on the production environment, but here the new prod runs crashed quite dramatically due to an error in a very freshly upgraded dependency.

This can happen at unpredictable moments because the requirements.txt file does not pin specific versions.

I have pinned our docker dependencies to a version that I know works. However, @jbelien pointed out to me that it would make more sense to simply specify the versions in the mainline pysteps requirements.txt instead.

I propose therefore to pin the versions of dependencies for reproducability.

To avoid having to update this manually every time packages are updated, one can run dependabot.
</issue_description>

<agent_instructions>Make pyproject.toml the single source of truth for dependencies. Add documentation on how to generate the requirements yml and txt files for backwards compatibility.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@ladc Additionally, dependencies are defined all over the place: - environment.yml (and the corresponding _dev file) - requirements.txt (and the corresponding _dev file) - setup.py - pyproject.toml

Which is the source of truth?</comment_new>

• Fixes no versions specified in requirements.txt cause unpredictable failures due to dependency upgrades. #526

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.14%. Comparing base (fe3a98a) to head (bd40a86).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #527   +/-   ##
=======================================
  Coverage   84.14%   84.14%           
=======================================
  Files         168      168           
  Lines       14507    14507           
=======================================
  Hits        12207    12207           
  Misses       2300     2300           

Flag	Coverage Δ
unit_tests	84.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.