8377180: Shenandoah: make escalation from degen to full more conservative

[kdnilsen]

This PR merges into Traditional Shenandoah an optimization that was applied to Generational Shenandoah in September 2025 and has proven itself effective in that context. See #27456

The change is to not immediately escalate Degenerated GC to Full GC following "bad progress" by Degen GC. The most common reason for bad degenerated progress is that most of the potential garbage was created following the start of the concurrent GC cycle which degenerated. This "floating" garbage cannot be reclaimed by the degenerated cycle because of the SATB protocol.

Pursuing at least one additional concurrent cycle, even if that subsequent concurrent GC degenerates, is usually a much better approach than forcing immediate escalation to Full GC. While escalation to full GC will resolve the out-of-memory condition, it imposes unnecessarily long stop-the-world pauses:

   Existing behavior not to scale:
       <- [       Conc GC       ] -> <- [ Degen    ] -> <- [            Full GC             ]
          [    experiences      ]       [  with    ]
          [     allocation      ]       [   bad    ]
          [      failure        ]       [ progress ]
                                        [ STW pause is 100 ms degen + 3s full GC            ]

   Revised behavior not to scale:
       <- [       Conc GC       ] -> <- [ Degen    ] -> <- [       Conc GC       ] -> [ Degen    ]
          [    experiences      ]       [  with    ]       [    experiences      ]    [  with    ]
          [     allocation      ]       [   bad    ]       [     allocation      ]    [  good    ]
          [      failure        ]       [ progress ]       [      failure        ]    [ progress ]
                                        [  100 ms  ]                                  [  100 ms  ] 
                                        [   pause  ]                                  [   pause  ]

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8377180: Shenandoah: make escalation from degen to full more conservative (Enhancement - P4)

Reviewers

• William Kemper (@earthling-amzn - Reviewer)
• Xiaolong Peng (@pengxiaolong - Committer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/29574/head:pull/29574
$ git checkout pull/29574

Update a local copy of the PR:
$ git checkout pull/29574
$ git pull https://git.openjdk.org/jdk.git pull/29574/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 29574

View PR using the GUI difftool:
$ git pr show -t 29574

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/29574.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back kdnilsen! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@kdnilsen This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8377180: Shenandoah: make escalation from degen to full more conservative

Reviewed-by: wkemper, xpeng

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 34 new commits pushed to the master branch:

• 986d377: 8376533: Remove test dependencies on ReferenceQueue$Lock in preparation for JDK-8376477
• cd5256d: 8374170: I/O Poller updates
• 9f13ec1: 8377365: [BACKOUT] Mixed jstack cannot find function in vDSO
• ... and 31 more: https://git.openjdk.org/jdk/compare/d49e29aa8c88a0f966446de4288f32a529f0dd52...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@kdnilsen The following labels will be automatically applied to this pull request:

• hotspot-gc
• shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 03: Full - Incremental (cb40a211)
• 02: Full - Incremental (d8d5423f)
• 01: Full - Incremental (0f094a68)
• 00: Full (8c8aee6f)

[earthling-amzn]

Looks good, suggest one small change.

[earthling-amzn]

Shall we rename GENERATIONAL_CONSECUTIVE_BAD_DEGEN_PROGRESS_THRESHOLD as well? Perhaps CONSECUTIVE_BAD_DEGEN_PROGRESS_THRESHOLD?

[kdnilsen]

Sure.

[pengxiaolong]

Traditional Shenandoah will have same upgrade policy as Genshen with the change.
Looks good to me, thanks!