Skip to content

review: test: Fixed nondeterministic failures in SubstitutionTest.testCreateTypeFromTemplate()#6501

Merged
monperrus merged 2 commits intoINRIA:masterfrom
yonghanlin:fix/substitutiontest-nondeterminism
Nov 8, 2025
Merged

review: test: Fixed nondeterministic failures in SubstitutionTest.testCreateTypeFromTemplate()#6501
monperrus merged 2 commits intoINRIA:masterfrom
yonghanlin:fix/substitutiontest-nondeterminism

Conversation

@yonghanlin
Copy link
Contributor

@yonghanlin yonghanlin commented Oct 28, 2025

What does this PR do?

This PR draws inspiration from the earlier PR #6084, which has not seen recent activity. The fix has been re-implemented on the current codebase and updated accordingly. In particular, this PR fixes a nondeterministic test failure in SubstitutionTest.testCreateTypeFromTemplate() when running with NonDex.

Problem

NonDex exposes that the generation of enum constants inside createTypeFromTemplate does not guarantee a fixed iteration order. During type construction, enum values are collected and inserted using data structures whose traversal order is unspecified (e.g., HashMap). As a result, the produced enum may list its values as GOOD, BETTER in one run and BETTER, GOOD in another. The test assumed a strict positional order of enum constants, even though the underlying creation process does not enforce such ordering. This causes failures under certain NonDex seeds.

Reproduce Test

To reproduce the failure, run NonDex on . module using the following commands:

mvn -pl . edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=spoon.test.template.SubstitutionTest#testCreateTypeFromTemplate

The Fix

The assertion in testCreateTypeFromTemplate is updated to extract the enum value names, sort them, and then assert against the canonical list ["BETTER", "GOOD"]. This verifies correctness by content, not by insertion order.

@yonghanlin yonghanlin changed the title fix: Prevent nondeterministic failures in SubstitutionTest.testCreateTypeFromTemplate() test: Fixed nondeterministic failures in SubstitutionTest.testCreateTypeFromTemplate() Oct 29, 2025
@yonghanlin yonghanlin changed the title test: Fixed nondeterministic failures in SubstitutionTest.testCreateTypeFromTemplate() review: test: Fixed nondeterministic failures in SubstitutionTest.testCreateTypeFromTemplate() Oct 29, 2025
@monperrus monperrus merged commit 94624c4 into INRIA:master Nov 8, 2025
13 checks passed
@monperrus
Copy link
Collaborator

Thanks a lot @yonghanlin

We like short PRs to improve tests.

Tip: avoid opening many PRs at once, it's easily overwhelming for maintainers. Prefer opening them one by one, one after the other, once the previous one has been merged.

@yonghanlin
Copy link
Contributor Author

@monperrus Thank you for the review and suggestion!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants