Knowing which tests will run in AssemblyInitialize

[nohwnd]

Summary

I am doing setup that is expensive and some parts are necessary only for some of the tests in the assembly. In my case some I am building a solution that holds all compatibility projects for vstest, and would like to build them only when any "compatibility" tests is running.

Is there a way to know in assembly initialize which tests will run (even roughly - if they passed filtering is good enough, they can end up being skipped)?

This would be especially useful when running single test in VS.

[Youssef1313]

There is none today. We can add it to TestContext, but I don't like having a single TestContext class that is getting bloated with information that are not relevant to every step in the test lifecycle. I would prefer us to design and implement more fine grained classes (AssemblyInitializeTestContext, etc). We can keep the existing TestContext for compatibility. We will need a concrete design here for how we will design those different test contexts, esp when we also want to expose TestContext.Current.

[Evangelink]

I have seen this pattern a few times and it feels to me the AssemblyInitialize is not the right answer for that. Instead, I would expect to have a service holding the state and being called either by the tests or TestClassInit for the relevant tests.

Ideally this is something where the fixture concept of NUnit and xUnit (and we had implemented in TATF) is useful because you could request a singleton of a service on the test classes that need that and check the global state without having to handle the synchronization yourself.

[nohwnd]

👍

For me some of the value lies in being able to do the work in the right place (initialization for the whole assembly), and hopefully being able to report that (from the assembly initialize), rather than having some Lazy object that will start setting up as soon as someone asks it.

I don't want to start tests, reach first group of tests, that will kick off build, and then reach the compatibility tests, and kick of another really expensive build in the middle of the test run.

I would much rather get all info, make a plan, do the setup, report how long it took to setup the whole assembly, and then progress to run tests from all categories.

[Youssef1313]

I have seen this pattern a few times and it feels to me the AssemblyInitialize is not the right answer for that. Instead, I would expect to have a service holding the state and being called either by the tests or TestClassInit for the relevant tests.

Ideally this is something where the fixture concept of NUnit and xUnit (and we had implemented in TATF) is useful because you could request a singleton of a service on the test classes that need that and check the global state without having to handle the synchronization yourself.

That also seems like a good idea to isolate the information from TestContext.