Atomic, automated upstream merges

[brynpickering]

We should track upstream and attempt a merge on every change, to reduce merge conflicts and make all the latest changes available in our child projects as quickly as possible. This could be an entirely automated process that falls back on human intervention when:

1. there are merge conflicts
2. there are test failures
3. the objective function value of test runs deviate from a tolerance we define (we would need to cache these values to compare against)

The CI could be automatically skipped when a PR is open awaiting human intervention, to avoid a backlog of PRs being generated.

Child projects can then pull in changes at whatever cadence they choose but they have the capacity to pull in latest pypsa/pypsa-eur changes as quickly as possible.

[tgilon]

100% aligned with this need, as we need to stay in sync with PyPSA/PyPSA-Eur for the Open-TYNDP. I've already discussed this extensively with @siddharth-krishna . The soft-fork bot could do the job once it is ready. In the meantime, I'm merging upstream in OET/PyPSA-Eur every Monday (unless I have too much work, as happened last week). This is not ideal, as it requires human intervention. Most of the time, though, it's just one click, as merge conflicts are rare.

[siddharth-krishna]

@brynpickering yep, we created such a bot for GitLab: https://github.com/open-energy-transition/soft-fork-bot/

We need someone to spend a bit of time adding support for GitHub, if someone has capacity let me know and I can walk them through the current state of things. I would estimate a few days of work to port it and test it.

The bot was designed to run on schedule, every week (configurable). Perhaps there's a way to use webhooks to get notified of every new commit on upstream/main if necessary, but running it once a day is probably enough?

[euronion]

There's a range of other bots and GH actions, maybe one of them would be helpful (instead):

• https://github.com/marketplace/actions/sync-your-forks
• https://github.com/marketplace/actions/autoupdate-fork-action
• https://github.com/marketplace/actions/fork-sync
• https://github.com/wei/pull

[brynpickering]

Some ideas of things we'd benefit from (from offline discussion):

• release note / commit message summary with highlights for key features that are most important to devs and to users
• a CI check that validates results derived from running a set of rules in the current commit compared to the previous commit (to see whether the changes being merged in materially affect results)