Add counter validation against CUTEst native reporting #463
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Potential fix for code scanning alert no. 8: Workflow does not contain permissions
tmigot
requested changes
Jul 14, 2025
Member
tmigot
left a comment
tmigot
left a comment
There was a problem hiding this comment.
Thanks ! I made a couple of comments.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #463 +/- ##
===========================================
- Coverage 89.11% 73.95% -15.16%
===========================================
Files 5 7 +2
Lines 790 1678 +888
===========================================
+ Hits 704 1241 +537
- Misses 86 437 +351 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Tangi Migot <tangi.migot@gmail.com>
Co-authored-by: Tangi Migot <tangi.migot@gmail.com>
Co-authored-by: Tangi Migot <tangi.migot@gmail.com>
tmigot
requested changes
Jul 27, 2025
Comment on lines
+160
to
+174
| # CUTEst's constrained problem implementation has specific counting behavior | ||
| @test julia_increments.neval_obj == 1 | ||
| @test cutest_increments.neval_obj == 0 # CUTEst doesn't count obj in constrained setup | ||
|
|
||
| @test julia_increments.neval_cons == 1 | ||
| @test cutest_increments.neval_cons == 2 # CUTEst counts 1 extra constraint call | ||
|
|
||
| @test julia_increments.neval_jac == 1 | ||
| @test cutest_increments.neval_jac == 2 # CUTEst counts 1 extra jacobian call | ||
|
|
||
| @test julia_increments.neval_hess == 1 | ||
| @test cutest_increments.neval_hess == 2 # CUTEst counts 1 extra hessian call | ||
|
|
||
| @test julia_increments.neval_jprod == 1 | ||
| @test cutest_increments.neval_jprod == 2 # CUTEst counts 1 extra jprod call |
Contributor
Author
There was a problem hiding this comment.
The differences between the NLPModels.jl counters and CUTEst’s native counters come from how each system tracks function evaluations internally.
- NLPModels.jl increments its counters every time a Julia wrapper function is called, so it reflects exactly what the user code requests.
- CUTEst, on the other hand, sometimes counts extra internal calls (for things like Hessian, Jacobian, or constraint evaluations) or may skip counting certain evaluations, depending on whether the problem is constrained or unconstrained and how its Fortran backend is implemented.
- For example, in constrained problems, CUTEst might not increment the objective counter, or it might count an extra call for Jacobian or constraint evaluations due to its internal logic. These differences are expected and are documented in the test file.
Co-authored-by: Tangi Migot <tangi.migot@gmail.com>
test_counters.jl """
amontoison
reviewed
Aug 28, 2025
test/test_counters.jl
Outdated
| neval_hprod = Int(calls[4]), | ||
| neval_cons = Int(calls[5]), | ||
| neval_jac = Int(calls[6]), | ||
| neval_jprod = Int(calls[7]) |
Member
There was a problem hiding this comment.
@arnavk23 A counter for jtprod is missing.
Otherwise LGTM 👍
Member
There was a problem hiding this comment.
Sorry for the long delay @arnavk23, I was working on many projects recently.
amontoison
reviewed
Sep 7, 2025
amontoison
approved these changes
Sep 7, 2025
Member
|
Thank you @arnavk23 ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements comprehensive validation of evaluation counters by comparing CUTEst.jl's internal counter tracking with CUTEst's native reporting functions.
test/test_counters.jl: Complete test suite for counter validationget_cutest_counters(): Utility function that extracts counters from CUTEst's native reporting functionstest_counter_validation(): Main test function with separate validation for unconstrained and constrained problemsCloses #278
Closes #465