Meta: Generate per-library coverage reports #26541
Conversation
This will allow us to have custom behavior for coverage builds.
Using `find` to do the heavy lifting makes the execution time go from 44s to 3s on my computer.
This avoids contaminations from previous runs.
This will allow everything defined in headers to be instrumented. As an example, the function `AK::any_of` isn't instantiated inside AK, that means that no coverage is reported if we don't consider the instantiation in the test binary itself.
This allows us to see the coverage of each library without transitive usages. As an example, Vector::clone() is never tested directly in Tests/AK, and it's coverage depends on an external test or library. This means that its coverage is fragile as the code using it in tests might be dropped in the future.
|
The approach seems fine. Is the plan to upstream the patch, and wait for it to become available? I can see the appeal of this, but personally, I think I find a global report more interesting. IMHO testing libm with js tests is kind of fun and a benefit of NIH :) (I agree that it's a bit of a risk, but looking at the global report would also flag is something becomes untested.) No harm in having both though, I suppose :) |
If I achieve to upstream the patch, I will probably also add it to our local patches so we can start to have these reports before the next LLVM release.
Transitive testing is ok for some stuff, but I would consider it bad for base libraries like |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! |
LucasChollet
added
the
⛔️ pr-is-blocked
2 participants
Having per-library reports remove the bias due to the usage of a library outside its own tests. This is of course most visible with
AKandLibCore.Here is AK within the global report:
And here when only considering
Tests/AK:Of course, "standalone" library doesn't have that issue and for example
LibEDIDreports the same results in both reports.This is a draft because in order to generate the per-library reports, I had to patch
llvm-covto teach it to only include files matching a given regex in the source folder (the-include-filename-regexoption). Here is the patch:llvm-cov-include-filename.patch
@nico What do you think of this approach? (And if you think I should upstream the patch, can you give it a quick look?)