Considering applying NinjaTurtles to our greenfield .net 4.5 project

Nov 19, 2014 at 3:50 PM
Edited Nov 19, 2014 at 3:51 PM
Hey guys, I'm considering adding NinjaTurtles mutation testing to our greenfield project and wanted to ask you about some questions I have. I've been trying to find some of these answers as I go, and if we do end up using NinjaTurtles I'm sure I'll be sending pull requests sometime in the future. I appreciate that the project seems to be actively developed and is pushing towards a v1 release.

So, some of the questions/concerns I have deal with the code specifically.
  1. Our project has a lot of WPF and caliburn.micro view/viewmodel code. There's a lot of interactions in there between different methods, event handlers and property getters. A lot of our public methods call into more complex private methods, and its the private methods that would really benefit from mutation testing. I can mutation test the private method by writing a builder MutationTestBuilder<MyViewModel>.For("MyEventHandler") but that seems to violate our code standard of blackbox testing.
    Could we set up a MutationTest that identifies private methods called into by the method under test, and mutates those as well?
  2. We use closures quite often in our code, as event handlers or actions pushed onto stacks for later execution. Forgive me if I'm wrong, but I've been scanning the code and can't see whether the mutation tests identify these anonymous methods and mutate them as well. It seems like it would reside in MethodTurtleBase.Mutate, and I see where that has code to address yield return generators. Should I open an issue to address this?
  3. Similar to #2, we use async/await a lot as well. Does NinjaTurtles have facility to mutate these methods yet? In the compiled .dll they become state machines in specially named inner classes.
  4. Finally, I want to ask about best practices for integrating this into our build process. My thinking has been to add TestMethods with [TestCategory("Mutation")] for each complex method in our code, and run all of these on our Jenkins build server. We would add as many as we can, trying to get as close as possible to 100% mutation coverage, and if any of them failed this would break the build. Some of my colleagues have expressed concern over this, saying 100% mutation coverage may not be a reasonable goal. Another strategy we may look into is writing all these tests and running them separately from our normal unit tests, and not failing the Jenkins build if these fail. That way we can have an indicator of our mutation coverage, but not a mandate of 100% coverage.
    What are your opinions on this? Do any of you have experience using mutation testing in a team project? What controls did you put in place?
Thank you very much for your help.
Nov 20, 2014 at 9:36 AM
Hi Gordon,

I'm not sure we're actively developing NinjaTurtles, but it does hold a special place in our hearts, so we'll maintain it. I'd love to do more to it, but time is a rare commodity these days.

Having said that, I'll try and answer your questions as best I can.
  1. I forget exactly how it works, but method call return types are mutated IIRC. I can't quite remember if the private methods are mutated with it, or not, but I have a vague recollection that they may not be. There are ways around that of course, and perhaps none of them are particularly palatable. a) you could make private methods internal, and friend assembly your test assembly, allowing you to test the methods directly. b) create a test helper that reflects out the the method and calls it.
  2. I'll need to dig into the closures a bit. They get compiled out to a separate method in IL, so if private methods aren't covered then they won't be either, but the other way round works to your advantage.
  3. Same here.
  4. And this is the long answer: By all means aim for 100%, but it's unlikely you'll ever meet it. Some of the most mundane things can really wreck it for unit testing (think any IO, where to write proper unit tests requires that you create proxies for the class - trivial with Resharper, but the proxy still won't be covered without you doing actual IO, which you want to avoid). I wouldn't try making this run as part of your regular CI build. Use it as more of a nightly/weekly build event and something that is run by the developers locally, to quality check their unit test. It'll just take too long to run all the mutations on anything but the most trivial of solutions. Go with your alternative (run regularly as a separate build, but don't over do it, and don't add it to your CI build). Like all things, it's a tool that needs to be use pragmatically, and as part of a wider development process. If you do code reviews once features are ready, then you can make it a mutation run (and mutation pass) a requirement for acceptance of the feature. Like all other things (code coverage comes to mind here), if you ignore it, it will fall away rapidly. You might have a separate nightly build, but if you don't action based on it, you will find it hard to catch up.
davidmus might be able to comment a bit more on your first 3 points if he recalls. Other than that, I'll have a look around when I'm adding your extension methods in and come back with a little more clarity for you.

As an added aside, if you plan on using anything like NCrunch during your development (and I would heartily recommend it), make sure that you disable it for any mutation category tests. Mutation testing is quite heavy and running it every time you type a character in your solution can bog you down pretty quickly.
Nov 20, 2014 at 3:20 PM
Yeah I definitely get that time is a luxury. I see your last commit was in May, which honestly puts you guys as the most active .net mutation testing framework I can find on Google. I love the simple setup for including it as tests.

Our codebase so far is fortunately small enough that we could aim for a very high percentage of coverage. We've been working on it in my small team for about 9 months so it's not too built out yet. If we can't get 100%, then it may be beneficial to shoot for something like a "mutation coverage report" which we run after every sprint. My thinking would be to run mutation tests over every method (public and private) in our codebase, and generate a report of which methods are covered by tests, which methods have surviving mutations, and which methods we've killed all mutations. Then it would be more like a coverage tool. This would of course mean adding new features to NinjaTurtles, but hopefully if we decide that route is worth it we could convince my manager to let us spend 9-to-5 time working on it.

Does that sound like something consistent with your vision of how NinjaTurtles would be used?
Nov 20, 2014 at 4:05 PM
Edited Nov 20, 2014 at 4:31 PM
GordonBurgett wrote:
Does that sound like something consistent with your vision of how NinjaTurtles would be used?
Pretty much bang on!

I've done a bit of snooping, and I think you should be able to test your private methods. (as long as you're using the string variant of For). The MethodResolver certainly finds private methods, so in theory, just name it as your mutation target and it should work. David also added some code to allow for testing private classes as well, which I had forgotten about.

That covers off point 1 I think, leaving 2 & 3 unanswered. Regarding point 3, there was no specific support for async/await when it was written, but as long as the compiler names the state machines consistently, you could use your decompiled knowledge to create the appropriate test. Once again, not the most palatable of results, but would do as a stop gap.
May 10, 2015 at 5:18 PM
Gordon: I encountered the same issue as you did - couln't find a well maintained production-ready mutation analysis tool. So I wrote my own, based on NinjaTurtles. It's called Splinter :)
If still interested in this topic, please contact me.