ninjaturtles Discussions Rss Feed

New Post: Error Messages

davidmus — Tue, 18 Dec 2012 10:12:04 GMT

Sorry for the delay replying. I haven't had much time for this project in recent weeks as you may have noticed. Yes, this is a pretty generic exception handling message. I am modifying it to at least output the exception details - this is a genuine exception case, rather than an identifiable problem in the command line parameters.

David

New Post: Error Messages

WSerapis — Thu, 22 Nov 2012 06:17:37 GMT

Is there any chance we'll get more descriptive error messages?

"An exception was thrown setting up the mutation tests. Please check yourcommand line parameters and try again." could be improved to give some clearer feedback as to what part of the command line args failed

Cheers

New Post: Beta?

freaky — Thu, 06 Sep 2012 07:55:42 GMT

At this point in time, I think it's pretty close to a production ready piece of kit. If the project I was working on right now had better than the 17% coverage that it does have, I'd be introducing this quite happily...... alas! Still, there are parts of the code that I've managed to get some good coverage around. I'll kick it into action shortly.

New Post: Console runner?

freaky — Thu, 06 Sep 2012 07:53:13 GMT

I had meant to reply to this earlier. Obviously a good idea, especially for experimentation. I have at this point seen what you've done with the console runner, and it's an awsome job. Keep em coming.

New Post: Console runner?

davidmus — Fri, 10 Aug 2012 07:44:16 GMT

The recent discussion in the issue tracker has got me thinking - how about a console runner for NinjaTurtles? This would cover the following use cases:

1. Running all tests marked with [MutationTest] in an assembly.

2. Automatically running mutation tests for a specified class and method (or just class). In case of the class, specify something like a BindingFlags enum value to identify which methods.

An advantage is it gives us a single operation with which to scope the creation of an output file, unlike the current NUnit integrated mode, where each method's mutation tests must either create a new file or merge into an existing one - no easily identifiable start point to clear this down.

Also, this might sit better in a CI/automated build environment.

Thoughts anyone?

New Post: Beta?

davidmus — Mon, 25 Jun 2012 15:32:43 GMT

Are we ready to call this a beta yet? It does what I think it should at this stage. It's pretty much eat what you cook, and I'd like to start encouraging its use, so the beta label may help with that. What does everyone think?

New Post: Parallelisation of the mutations.

davidmus — Tue, 24 Apr 2012 13:59:09 GMT

Yes, broadly in line with my observations. So definitely worth doing.

New Post: Parallelisation of the mutations.

freaky — Mon, 23 Apr 2012 09:49:21 GMT

Ok, running this on my machine (quad core with hyperthreading), these are the results:

Mutation Test	Single Threaded (secs)	Multi Threaded (secs)
Add	29	10.5
Divide	61	16
MixedAdd	118	21
MultiAdd	81	17
StaticAdd	28	9

This was pretty consistent across multiple runs.

New Post: Parallelisation of the mutations.

davidmus — Mon, 23 Apr 2012 09:30:41 GMT

I've just committed a simple intra-turtle parallelisation (using Parallel.ForEach). Comments welcome.

Seems to be almost a four-fold improvement in performance on my box (four cores with hyperthreading).

New Post: Parallelisation of the mutations.

davidmus — Mon, 23 Apr 2012 08:36:15 GMT

First comment: the test suite is run with the "stop on error" option set, so we only have to wait for the first failure in each case.

That said, for the extremely simplistic SimpleCalculatorTests, it takes a few minutes to run the mutation tests, so parallelisation may well be worth looking at.

New Post: Parallelisation of the mutations.

freaky — Mon, 23 Apr 2012 08:31:03 GMT

Thought I might broach the subject of adding parallel mutating of the tests. Obviously, the larger the unit under test gets, the more mutations we are likely to have an extremely long running test suite.

What should be our strategy for this parallel computing:

by turtle type
inside the type, so that multiple Arithmetic operations happen at once
both

To do so would require copying the files under test to a new directory (due to the renaming/munging of the files). Clean-up is obviously required.