add timestep retry mechanism

[BenWibking]

This is an old idea, but was specifically inspired by the implementation in Castro (https://amrex-astro.github.io/Castro/docs/timestepping.html#retry-mechanism) and Quokka (https://quokka-astro.github.io/quokka/instability/):

• If the post-update state does not satisfy the CFL-stable timestep that was computed pre-update, then go back and re-do the hyperbolic update with a new timestep $\Delta t/2$ where $\Delta t$ was the just-used value of the timestep. Repeat recursively if necessary, up to a maximum number of retries.

This may be implemented in Parthenon codes by:

I don't think the default driver is set up for that at all. But maybe you modify the downstream's EvolutionDriver subclass to tweak Execute and maybe SetGlobalTimeStep? There you have the power to just tweak tm.dt and re-run Step()

[pgrete]

I think the basics would technically be very straightforward to implement by

• adding an input/runtime parameter that enables this feature
• adding a cons_orig field
• adding a task that always backups cons to cons_orig
• "hack" PreStepUserMeshWork (or similar) to directly update dt and revert t (both should be possible as directly chaning tm.dt was originally done for the RKL2 integrator)
• adding a task that conditionally restore cons_orig
• make the ConsToPrim checks a softfail

*BUT * the details might become ugly

• we need to ensure that all reductions or data that is accumulated for an hst dump (or more generally stored in Params) is also backed up and restored
• ensure that no mesh refinement is triggered on a failed step
• output logic keeps working (or also ensure that no outputs are triggered)

Some of those piece might be fairly cleanly to fix/add upstream in Parthenon.