1
Vote

Exception in console runner without any information

description

Hi,

I am currently evaluating NinjaTurtles in order to introduce mutation testing in our company, but I cannot get the console runner working. I've got following Projects (solution attached):

NinjaTurtlesTest (Unit Tests)
NinjaturtlesTest.AppUnderTest (Library which should be tested; Contains class NinjaturtlesTest.AppUnderTest.Sut)

When I run following command

NinjaTurtles.Console run -c NinjaTurtlesTest.AppUnderTest.Sut -o Output.html -f HTML NinjaTurtlesTest.dll

I get the message "Unknown type 'NinjaTurtlesTest.AppUnderTest.Sut'".

When I experimentally move class Sut to the test project, I get following output:

_Running mutation tests for NinjaTurtlesTest.Sut.Add(Int32, Int32)
Running mutation tests for NinjaTurtlesTest.Sut.Subtract(Int32, Int32)
Running mutation tests for NinjaTurtlesTest.Sut.IsEqual(Object, Object)
Running mutation tests for NinjaTurtlesTest.Sut.IsGreaterOrEqual(Int32, Int32)
Running mutation tests for NinjaTurtlesTest.Sut..ctor

An exception was thrown setting up the mutation tests. Please check your
command line parameters and try again._

I also don't get any more information if I use the -v switch.
However I need to get the console runner running, because it seems not very comfortable having to write one additional test for each method. Beside this, NinjaTurtles does not work with ReSharper TestRunner (and does not even throw an exception but pretends successfully running all tests). And I'm also just able to execute tests using MS TestRunner when I set up a completely new project. When I try to integrate it in an existing one, I get a Win32Exception with message "System cannot find the file specified" most of the time, even if I install NUnit.Runners and NUnitTestAdapter.

Do you know a solution for my set of problems? And is there a way to get more information out of the console runner?
Do you intend to continue developing NinjaTurtles and get it stable?

Kind Regards
Yo

file attachments

comments

theYoRecords wrote May 19, 2015 at 2:41 PM

Hi,

I realized that this behaviour only occurs using version 0.8.0.2 from the downloads section. Using the latest source (version 0.9.0.0) it works without a problem.

freaky wrote May 19, 2015 at 3:10 PM

Hi Yo,

I've just published a new version to NuGet.org. This is based on the latest source, so hopefully, should fix your problem, and in the event that it doesn't, it should provide better error messages.

Cheers,
Paul

theYoRecords wrote May 19, 2015 at 3:57 PM

Hi Paul,
thanks for the quick reply. I think that should be v0.9.0.0 which I downloaded anyway. So if there are no new changes since i downloaded it one hour ago, I think I found a new bug:

When my test assembly and the tested assembly are both referencing a third assembly, Mono.Cecil fails to resolve it (until I put this assembly in the NinjaTurtles directory).

Exception:
Mono.Cecil.AssemblyResolutionException occurred
HResult=-2147024894
Message=Failed to resolve assembly: 'Third.Assembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'
Source=Mono.Cecil
StackTrace:
   at Mono.Cecil.BaseAssemblyResolver.Resolve(AssemblyNameReference name, ReaderParameters parameters)
   at Mono.Cecil.BaseAssemblyResolver.Resolve(AssemblyNameReference name)
   at Mono.Cecil.DefaultAssemblyResolver.Resolve(AssemblyNameReference name)
   at NinjaTurtles.MutationTest.AddMethodsForInterfaces(MethodDefinition targetMethod, List`1 matchingMethods, TypeDefinition type) in c:\Users\theYoRecords\Documents\Visual Studio 2013\Projects\NinjaTurtlesTest\ninjaturtles_803fd8ebfc97\NinjaTurtles\MutationTest.cs:line 226
InnerException:

This error could be resolved by loading, analyzing and running assemblies in a separate AppDomain with the target directory set as base. Or I can also think of a workaround like registering for AssemblyResolve event and checking of the requested assembly has already been loaded (and if so, return it). But I think the clean solution would be the first one, which might also prevent future problems.

freaky wrote May 19, 2015 at 4:44 PM

Would you be able to email me a solution that exhibits this behaviour? pms1969 At gmail.com

theYoRecords wrote May 20, 2015 at 11:17 AM

Hi,
I did not yet manage to reproduce the exact behaviour, but can provide you a sample project (attached), which produces a ReflectionTypeLoadException which has the same reason. I don't know what is the exact difference to my other project, but both errors could be solved by using a separate AppDomain in order to load assemblies using Assembly.Load instead of Assembly.LoadFile. Using LoadFrom or LoadFile is no good practice for loading assemblies in this case. Some time ago I also had to learn this the hard way.