Batches with large number of targets slows down test runs

[lyndsey-ferguson]

From @asos-chrisblackmore

Hey @lyndsey-ferguson - jus an FYI that although removing invocation_based_tests does solve the problem, supplying a batch_count drastically slows down the test run. It batches per target - so with 10 targets, and a batch count of 5, would result in 50 batches. There seems to be a little overhead with batches, and when you consider 1000's of tests, it does become an issue in terms of speed/performance.

[lyndsey-ferguson]

Here's my initial thoughts on this.

Each target is already considered a batch, as multi_scan needs to separate them out to properly create a junit xml file for tracking failed tests within the target.

When you add the additional :batch_count parameter, it is going to split each of those batches up into 5 batches. It seems that this is working as expected as, yes, it will take longer. You could use the parallelize_test_runs option to send each batch to a different simulator to try and speed things up.

Do you have thoughts on how this would be done differently?