Parallelisation of the mutations.

Coordinator
Apr 23, 2012 at 8:31 AM

Thought I might broach the subject of adding parallel mutating of the tests.  Obviously, the larger the unit under test gets, the more mutations we are likely to have an extremely long running test suite.

What should be our strategy for this parallel computing:

  • by turtle type
  • inside the type, so that multiple Arithmetic operations happen at once
  • both

To do so would require copying the files under test to a new directory (due to the renaming/munging of the files).  Clean-up is obviously required.

Coordinator
Apr 23, 2012 at 8:36 AM

First comment: the test suite is run with the "stop on error" option set, so we only have to wait for the first failure in each case.

That said, for the extremely simplistic SimpleCalculatorTests, it takes a few minutes to run the mutation tests, so parallelisation may well be worth looking at.

Coordinator
Apr 23, 2012 at 9:30 AM
Edited Apr 23, 2012 at 9:35 AM

I've just committed a simple intra-turtle parallelisation (using Parallel.ForEach). Comments welcome.

Seems to be almost a four-fold improvement in performance on my box (four cores with hyperthreading).

Coordinator
Apr 23, 2012 at 9:49 AM
Edited Apr 23, 2012 at 9:49 AM

Ok, running this on my machine (quad core with hyperthreading), these are the results:

Mutation Test Single Threaded (secs) Multi Threaded (secs)
Add 29 10.5
Divide 61 16
MixedAdd 118 21
MultiAdd 81 17
StaticAdd 28 9

This was pretty consistent across multiple runs.

Coordinator
Apr 24, 2012 at 1:59 PM

Yes, broadly in line with my observations. So definitely worth doing.