It's always interesting to see how the different unit testing frameworks approach automated testing, each framework has it's own style, approach and feature set with everyone having their own view on which framework everyone should use. One of the more (you could say most) popular .Net framework is NUnit and over the past few months Charlie has been working on v2.5 so I thought it was about time I took a look.

Parameterised Tests (Documentation can be found here Parameterized Tests)

Similar to XUnit and MbUnit, NUnit 2.5 includes the concept of parameterised tests. RowTest was included in 2.4.7, however additional attributes have now been included.

ValuesAttribute

The first new attribute allows for combinational style testing, this is where two sets of inputs are combined to create each possible test combination. In certain situations useful - always good to have in the toolbox, however generally I wouldn't need to use this on a day-to-day basis. MbUnit has had this for a while, however I prefer the syntax of this approach, which I think is a similar syntax to MbUnit V3.

To implement this, you add the ValuesAttribute, which takes an object array for the test values, before the method parameter.

[Test]
public void ValuesAttributeTest([Values(1,2,3, 20)] int v, [Values(9,8,7)] int v2)
{
    Assert.IsTrue(v < v2);
}

When NUnit loads this up, they create a combination of the test values, as a result we have 12 tests created. If we had another parameter of the test, for example with 3 possible values, then we would have another possible combination and we would have 36 tests. As a result, you can very quickly build up a good selection of test cases.

RangeAttribute

Similar to the ValuesAttribute, however instead of you defining the values, you define the range of possible values which NUnit then creates tests for. You define the range (min, max) and NUnit does the rest. The values are then combined in a combinational style.

[Test]
public void RangeAttributeTest([Range(1,100)] int v, [Range(1,100)] int v2)
{
    Assert.IsTrue(v < v2);
}

The test creates produces then looks something like this:

RandomAttribute

Similar to the two previous attributes, Random produces a set of random input. You can either give it a count of the inputs to use, as I have done with the first attribute which produces 10 random values, while in the second parameter I provide the min and max range for the random values together with the amount of data to produce.

[Test]
public void RandomAttributeTest([Random(10)] int v, [Random(1, 50, 10)] int v2)
{
    Assert.IsTrue(v < v2);
}

However, I'm not sure how useful testing against random data would actually be?

TestCaseAttribute

This attribute is similar to RowTest, however as part of the TestCase you can set your own TestName which I think is actually really useful.

[TestCase("Test1", TestName = "This is a great!", Description = "My first test")]
[TestCase("Test2")]
[TestCase("")]
public void TestCaseAttributeTest(string test)
{
    Console.WriteLine(test);
    Assert.IsNotEmpty(test);
}

The TestCase is then reflected In the GUI and report. The first two tests use the test method name plus their parameter inputs, while the third uses the TestName property we set on the TestCaseAttribute.

Also as part of the TestCaseAtribute, you are able to provide the Result.  Generally, the expected result is passed in as a parameter and verified within the test itself. With this approach you have the result as part of the TestCase itself, with the test returning it's result which NUnit then compares to decide if the test has passed. This does make more sense as the parameters of the test as then really test inputs with the test output being verified as an output, a small difference however it is more logical.

[TestCase("Test1", TestName = "This is a great!", Description = "My first test", Result = 5)]
public int TestCaseAttributeTest(string test)
{
     Console.WriteLine(test);
     return test.Length;
}

This approach to testing has a lot of potential use cases and opportunities as it really increases test case (code) reuse and test maintenance.

TestCaseFactoryAttribute

Similar to MbUnit's FactoryAttribute, you define a class which yield returns each TestCaseData object. It has the advantage of using a fluent interface to define all the different properties, such as results and the test name.  While I have hardcoded the 4 test cases here, there is nothing stopping you dynamically creating these.

[Test,Factory(typeof(MyFactoryClass))]
public int FactoryAttributeTest(int a, int b)
{
    Console.WriteLine(a / b);
    return a/b;
}

public class MyFactoryClass
{
    [TestCaseFactory(typeof(int), typeof(int))]
    public static IEnumerable TestCases
    {
        get
        {
            yield return new TestCaseData(12, 3).Returns(4);
            yield return new TestCaseData(12, 2).Returns(6);
            yield return new TestCaseData(12, 4).Returns(3);
            yield return new TestCaseData(0, 0)
                              .Throws(typeof(DivideByZeroException))
                              .WithName("DivideByZero")
                              .WithDescription("An exception is expected");
        }
    }
}

The TestCases then look like this:

New Assertions

Assert.Throws<T>

Simple, straight forward. Does the method call throw can exception?

[Test]
public void ThrowException()
{
    Assert.Throws<Exception>(throwEx);
}

public void throwEx()
{
    throw new Exception("Test");
}

Assert.DoesNotThrow

Simpler approach, Assert that the method does not throw any exceptions when it is called.

[Test]
public void doesNotThrowExTest()
{
    Assert.DoesNotThrow(doesNotThrowEx);
}

public void doesNotThrowEx()
{}

Others

RequiredAddinAttribute

This is extremely useful if you are writing your own NUnit addins, as it is very important that the addin is available before you attempt to execute your tests. After adding attributes about the addins you require,

[assembly: RequiredAddin("MyTestFixtureAddin")]
[assembly: RequiredAddin("MyTestAddin")]
[assembly: RequiredAddin("MyDecoratorAddin")]

If they are not available and you execute the test, it will fail but will actually tell you the reason why.

PlatformAttribute

Finally, there is a new PlatformAttribute. This attribute defines which platforms the test is allowed to run on, really useful if you have issues with a certain target platform, or if you have a set of tests only targeting a platform, such as mono.

[Test]
[Platform(Exclude="Win98,WinME", Reason = "This only works on Windows 2000 and above")]
public void OnlyWorksWith2000OrHigher()
{
    Assert.IsTrue(Environment.OSVersion.Version.Major > 5);
}

[Test]
[Platform(Include = "Win98,WinME")]
public void BugOnWin98()
{
    Assert.IsTrue(Environment.OSVersion.Version.Major > 5);
}

The list of possible platforms can be found here: http://nunit.org/index.php?p=platform&r=2.5

When the test is executed on an invalid platform, the test is skipped.

This post has summarised all of the interesting bits of the new Alpha release. In future releases, I expect to see some changes going forward before the final 2.5 release. It's nice to see NUnit taking the different approaches from the different frameworks and incorporating them with their own approach, hopefully v3 will have some really interesting concepts. The main feature missing at the moment are data driven test attributes, such as pulling test cases from CSV or SQL Server, but a framework has an additional extension point to make this attribute easier to write.  After this I think most of the missing features would be more suitable in NUnit 3.0.

Release Notes: http://nunit.com/index.php?p=releaseNotes&r=2.5
Download: http://nunit.com/index.php?p=download

Labels: NUnit, TDD, Testing

During the development lifecycle, there are many different automated tests which should be wrote, each style of test has different priorities and different levels of maintenance required. For example, TDD helps with designing the code your just about to write, they are small and focused. As such, when your code changes, your tests change too. Where as acceptance tests have a much longer life-time, there role is to ensure the application works and nothing is regressed as developers continue to make changes. These tests need to be maintainable enough to cope with application changes, readable enough to identity the story \ feature they are testing with reusable\flexible sections to help readability and maintainability.

In an attempt for my tests to meet these goals, recently I've been using delegates in certain scenarios to improve my test code. Delegates are great for reusing sections of code, it allows you to have static sections of code but be flexible to cope with minor changes and different scenarios the tests might cover.

For example, below is an acceptance test which must connect to a server based on a project file which has been saved to disk.  Remember, acceptance tests should cover the system in an end-to-end fashion.

In order to test the application, the code needs to do a lot.  It needs to create and configure a project, save it to disk, create the service and finally it gets to the point of what it is testing - connecting to the service. This code, while it is flexible it is not very maintainable. If we need to change something related to the project or how we connect to the service we would have to change a lot of code to take this into account.

public class Project_Execute_Tests_Standard
{
    [Fact]
    public void Project_Can_Login_And_Can_Connect()
    {
        string path = System.IO.Path.GetRandomFileName();

        Service service = null;
        try
        {
            Project project = new Project();
            project.Server = "Test";
            project.Username = "Admin";
            project.Password = "Password";
            project.Save(path);

            service = new Service();
            service.Start();

            service.LoadProject(path);

            bool connect = service.Connect();
            Assert.True(connect);
        }
        finally
        {
            if(service != null && service.Started)
                service.Stop();
            File.Delete(path);
        }
    }
}

The first logical step would be to use helper methods.  We can extract the creation of the project and gaining access to the service into separate methods which we can reuse throughout our test code. However, these methods aren't very flexible, if we need to add some more configuration to the project we would have to create a different method and as such losing some of our maintainability.

[Fact]
public void Project_Can_Login_And_Can_Connect()
{
    string path = string.Empty;
    Service service = null;
    try
    {
        path = CreateProject();

        service = GetService(path);
        bool connect = service.Connect();
        Assert.True(connect);
    }
    finally
    {
        if (service != null && service.Started)
            service.Stop();
        File.Delete(path);
    }
}

This is why I like delegates. You have the advantage of helper methods, but the flexibility of having the code inline. Below I've created a helper class, this abstracts away from my actual implementation and manages the state, such as the file path for the project. Within the delegate for the project, I'm setting all the details based on the requirements of the test and different configurations, however the rest is abstracted away.

public class Project_Execute_Tests_Delegate
{
    [Fact]
    public void Project_Can_Login_And_Can_Connect()
    {
        ServiceTester tester = new ServiceTester();
        tester.CreateProject(delegate(Project project)
                                        {
                                            project.Server = "Test";
                                            project.Username = "Admin";
                                            project.Password = "Password";
                                        });

        bool connect = tester.ConnectToService();
        Assert.True(connect);
    }
}

While I was writing this example, DevExpress popped up and told me it can shorten my delegate to a lambda expression. As a result, the delegate could be this.

tester.CreateProject(p =>
                            {
                                p.Server = "Test";
                                p.Username = "Admin";
                                p.Password = "Password";
                            });

Within my ServiceTester helper class, my CreateProject method looks like this:

public delegate void ProjectDelegate (Project project);
public void CreateProject(ProjectDelegate ProjectSettings)
{
     ProjectPath = System.IO.Path.GetRandomFileName();
     Project project = new Project();

     ProjectSettings(project);
     project.Save(ProjectPath);
}

Given the right scenario, I think this could really improve your test code.

Download complete code sample: http://blog.benhall.me.uk/Code/Test/TestCodeDelegates.txt

Labels: TDD, Testing

While executing my integration tests today, I was greeted with the excellent Assertion.Failed dialog which is displayed when using System.Diagnostics.  This dialog caused a problem on the build server as it was waiting for someone to click Ignore - that someone never came!

If you haven't saw the Debug.Assert code before, it basically looks like this:

[Test]
public void DebugAssert()
{
    System.Diagnostics.Debug.Assert(false);
}

When the Assert fails, the dialog is displayed.

The solution, add an App.Config file to your test suite.  Within the XML will clear the trace listeners, now when assert is called nothing happens because nothing is listening for it and the code continues. While executing unit tests this is fine - we just don't want the dialog being displayed.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <system.diagnostics>
        <trace>
            <listeners>
                <clear/>
            </listeners>
        </trace>
        <assert assertuienabled="false" />
    </system.diagnostics>
</configuration>

If you was interested in the debug.assert messages, you could have hooked up a different listener to write out the message to the console or a file.  This is what TestDriven.NET does, if an assertion is hit it writes the message to the output window.

Labels: TDD, Testing

Recently, Ayende released Rhino Mocks 3.5 Beta which includes a new Arrange, Act and Assert syntax style. This is a new syntax for creating your stubs and mocks. To take a deeper look into this new syntax, I'm going to migrate an old example I created for my article Beginning to Mock with Rhino Mocks and MbUnit - Part 2 which is a good read if you are new to the world of mock objects. As a quick refresher, the test I created was ensuring that the PaymentProcessor correctly communicated with the PaymentProcessingObject (in the system, this actually talked to PayPal).

[Test]
public void TakePaymentViaPaymentProcessorUsingMockService()
{
    MockRepository mocks = new MockRepository();
    IPaymentProcessing mockProxy = mocks.CreateMock<IPaymentProcessing>();

    using (mocks.Record())
    {
        Expect.Call(mockProxy.TakePayment(1, 1, 10.0)).IgnoreArguments()
                             .Return(true);
    }

    using (mocks.Playback())
    {
        PaymentProcessor pp = new PaymentProcessor(mockProxy);
        bool result = pp.TakePayment(1, 1, 10.0);
        Assert.IsTrue(result);
    }
}

The code this tested was:

public interface IPaymentProcessing
{ bool TakePayment(int paymentId, int customerId, double amount); }

internal class PaymentProcessor
{
    internal IPaymentProcessing wsProxy;

    public PaymentProcessor(IPaymentProcessing proxy)
    {
        wsProxy = proxy;
    }

    public bool TakePayment(int paymentId, int customerId, double amount)
    {
        return wsProxy.TakePayment(paymentId, customerId, amount);
    }
}

This is a very generic example, but demonstrates Rhino Mocks using the Record (Setup the mock object) and Playback (Test and use) approach to creating mock objects. Now we have our classic test, let's look at the changes for 3.5

CreateMock is Obsolete

The first change is that CreateMock has been marked as obsolete. Instead, we should be using StrictMock, this makes a lot more sense as it better defines the type of mock being created.

IPaymentProcessing mockProxy = mocks.StrictMock<IPaymentProcessing>();

But this still uses the same Record\Playback model.

AAA Syntax - Arrange, Act, Assert

This is where Rhino Mocks 3.5 is really interesting - for me at least.  We can now express our mock object is a much cleaner fashion, taking advantage of .Net 3.5 extension methods and lambda questions. Below, is the same test as above but using the Mock object.

[Test]
public void GenerateMock_TakePaymentViaPaymentProcessorUsingMockService()
{
    IPaymentProcessing mockProxy = MockRepository.GenerateMock<IPaymentProcessing>();  #1

    mockProxy.Expect(x => x.TakePayment(1, 1, 10.0))                                                                #2
                    .Constraints(Is.Equal(1), Is.Equal(1), Is.Equal(10.0)) 
                    .Return(true);

    PaymentProcessor pp = new PaymentProcessor(mockProxy);
    bool result = pp.TakePayment(1, 1, 10.0);
    Assert.IsTrue(result);

    mockProxy.VerifyAllExpectations();                                                                                       #3
}

#1  Here we tell Rhino Mocks to create us a mock object of type IPaymentProcessing.
#2  Next we define our mock. Here, we are saying we expect TakePayment to be called, we then add some constraints about what the parameters passed in much be, finally defining that true be returned when this is called.
#3  Finally, we verify the exceptions we set in #2 where correct.

I find this new approach to be much easier to read, explain and write. The first time I tried this, I actually mistaken Constraints for Return and as such the following exception was thrown.

mockProxy.Expect(x => x.TakePayment(1, 1, 10.0)).Constraints(Is.Equal(true));  = failed: System.InvalidOperationException : The number of constraints is not the same as the number of the method's parameters!

Just be aware of these new constraints on the parameters. 

But, not only can we use this for creating mocks, but we can also create Stubs.

[Test]
public void GenerateStub_TakePaymentViaPaymentProcessorUsingMockService()
{
    IPaymentProcessing stubProxy = MockRepository.GenerateStub<IPaymentProcessing>();  #1
    stubProxy.Stub(action => action.TakePayment(1, 1, 10.0)).Return(true);                           #2

    PaymentProcessor pp = new PaymentProcessor(stubProxy);
    bool result = pp.TakePayment(1, 1, 10.0);
    Assert.IsTrue(result);
}

#1  Generate the stub
#2  Define the stub 

How cool is that!! Two lines of code to create our stub!

[Test]
public void GenerateStub_AssertWasCalled_TakePaymentViaPaymentProcessorUsingMockService()
{
    IPaymentProcessing stubProxy = MockRepository.GenerateStub<IPaymentProcessing>();
    stubProxy.Stub(action => action.TakePayment(1, 1, 10.0)).Return(true);

    PaymentProcessor pp = new PaymentProcessor(stubProxy);
    bool result = pp.TakePayment(1, 1, 10.0);
    Assert.IsTrue(result);

    stubProxy.AssertWasCalled(x => x.TakePayment(1, 1, 10.00));                                                  #1
}

#1  With stubs, we can also verify that the method was called. Notice this AssertWasCalled is an extension method Rhino Mocks has added to the interface.  gain, this is helping with the readability and demonstrates an excellent use of Extension Methods.

If AssertWasCalled failed, then the following exception would be thrown and the test would fail.

Expected that IPaymentProcessing.TakePayment(1, 1, 10); would be called, but is was it was not found on the actual calls made on the mocked object  

I think the new AAA syntax is really cool. I used it to explain mock objects for my NxtGenUG Coventry session and upgrading my mock objects was a simple task and made the tests much easier to read. Really looking forward to the final release.

Download Examples Used: http://blog.benhall.me.uk/Code/RhinoMocks/RhinoMocks35Beta.zip

More examples of new syntax: http://www.ayende.com/Blog/archive/2008/05/16/Rhino-Mocks--Arrange-Act-Assert-Syntax.aspx

Download: Rhino Mocks 3.5

Labels: Rhino Mocks, TDD, Testing

I've been waiting for today for too long! Peli has announced that Pex 0.5 has been released and is available to download today! In this post, I am just going to have a very quick look at the release.  Expect a lot more posts around this framework, best subscribe now so you don't miss any.

Pex is a Microsoft research project which generates tests that cover all possible inputs. It does this by a mixture of static\dynamic analysis while taking advantage of the CLR Profiler to analyse your .Net code to find all the possible paths. Pex can then creates a series of test inputs for a test YOU wrote, you write the test which Pex then uses to find all the possible routes. You can use these tests in order to find all the different possible combinations for your application, find the areas where their are problems and have Pex suggest possible fixes.

Copied from site:

Pex comes with a Visual Studio Add-in which only works with Visual Studio 2008 Professional (or higher).
Pex also works from the command line without any Visual Studio on the machine.

Downloads

v0.5.30516.0, 05/21/2008, download pex.30516.0.msi (x86), release notes

The Installer:

Very simple msi - just click next a few times..

Taking a look at the samples

The team has spent a long term putting together some documentation and samples on how to use Pex which is great (especially for a research project). The samples are installed as part of the main framework and are in a zip file linked off the Microsoft Pex start menu item. Within the zip there is a VS2008 solution, with a project Samples.Pex which appears to contain samples for a lot of the functionality (without researching more I can't tell what).

Within the samples, the first test I picked at random is the LuhnAlgorithmTestClass. This was of interest because I had a similar test scenario for Red Gate SQL Data Generator as that has a Credit Card Number Generator for the Luhn Algorithm.

The test itself, based on MSTEST, looks very standard. At the class level, we add a PexClass attribute and give the type of class we are testing against. The MaxRuns defines how many runs will be tried during the exploration.

[TestClass]
[PexClass(typeof(LuhnAlgorithm), MaxRuns = 100)]
public partial class LuhnAlgorithmTestClass

The test itself, we add a PexMethod and most importantly a parameter for the input to use. Pex will then use this test to generate more exploratory tests and use the parameter to inject the value to test against. 

[PexMethod]
public void CharactersAreNotValid(string input)
{ 
   string number = input + 'a'; 
   bool result = LuhnAlgorithm.Validate(number); 
   Assert.IsFalse(result);
}

Pex includes a Visual Studio plugin, I can write click and go Pex It!

This should run all of my unit tests, but after executing Pex It! I got an error ClrMonitorFail (-667) - "Exit code returned when the ExtendedReflection profiler encountered a fatal error." within the Pex Results dialog. I've logged this with the team, if I get a solution I will post it online.

Luckily, Pex comes with a console application so the post isn't over! The command line is simple Pex with the assembly containing the Pex tests. The following command executes Pex for me:

D:\Users\Ben Hall\Desktop\samples\Pex\Pex\bin\Debug>pex Samples.Pex.dll 

After executing this, we get a lot of information wrote to the console application and a nice HTML report outputted.  The console information looks like this:

Microsoft Pex v0.5.30516.0 -- http://research.microsoft.com/pex
Copyright (c) Microsoft Corporation 2007-2008. All rights reserved.

instrumenting... launched Pex 0.5.30516.0 x86 Edition on .net v2.0.50727
[reports] report path: reports\Samples.Pex.80522.232227.pex
00:00:00.0> starting execution
  00:00:00.6> reflecting tests
  00:00:04.9> Samples.Pex
    00:00:05.0> LuhnAlgorithmTestClass
      00:00:06.5> CharactersAreNotValid(String)
        [test] (run 1) CharactersAreNotValidString_20080522_232253_000
        [test] (run 4) CharactersAreNotValidString_20080522_232256_001
        [test] (run 6) CharactersAreNotValidString_20080522_232256_002
        [execution] imprecision at Samples.Pex.Implementations.LuhnAlgorithm.Validate, offset 0x55
        [execution]     9 runs (88.89 % unique paths), 9/14 blocks covered
        [test] (run 10) CharactersAreNotValidString_20080522_232256_003
        [test] (run 27) CharactersAreNotValidString_20080522_232257_004
        [test] (run 28) CharactersAreNotValidString_20080522_232257_005
        [test] (run 41) CharactersAreNotValidString_20080522_232258_006
        [test] (run 45) CharactersAreNotValidString_20080522_232259_007
        [test] (run 87) CharactersAreNotValidString_20080522_232301_008
        <boundary> maxruns - 100
        <boundary> MaxRuns, 1 times
        [coverage] 11/14 block (78.57 %)
<snip>
00:07:43.6> [finished] 269 generated tests (51 failures), 00:07:43.6011870
    -- 0 critical errors, 0 errors, 4 warnings
[reports] generating reports...
[reports] html report: reports\samples.pex.80522.232227.pex\pex.html 
  EXPLORATION SUCCESS

This is showing that Pex is working against the LuhnAlgorithmTestClass.CharactersAreNotValid(string input) test (as shown above), and is generating multiple different tests based on that initial test.

With the report looking like this:

The report contains a lot of information about the execution. Clicking the name of the test class takes you to some more information about what was executed. 

Clicking the parameter values link will display all of the values used for the different tests.  Each parameter sends the code down a different route.

Another link which interested me is the Coverage link for the LuhnAlgorithmTestClass. Clicking on this, you get a really nice Code Coverage report page.

Putting this a different way, by writing a single test with a parameter, Pex can create us a series of different tests to execute all of the possible routes in the method. This is great! As a developer, I can focus on creating a test which can exercise the method, but don't have to worry about all of those edge cases as Pex will help find those.

Hello World Pex'ed

Hopefully you are still with me, now I just wanted to create a very quick Hello World application.  The class I have developed is this HelloWorld class, it has three different paths different different values for each.

public class HelloWorld
{
    public string GetHelloWorld(int ID)
    {
        if (ID == 1)
            return "Hi";

        if (ID == 2)
            return "Hello";

        if ((ID != 1) || (ID != 2) && (ID % 2 == 0))
            return "Hello World";

        return string.Empty;
    }
}

The related test is this HelloWorld(int id).  Pex will input the required parameters as a argument, and then execute the test. We want to ensure Hello World is returned.

[TestFixture]
[PexClass(typeof(HelloWorld))]
public class HelloWorldTests
{
    [PexMethod]
    public void HelloWorld(int id)
    {
        HelloWorld h = new HelloWorld();
        string helloWorld = h.GetHelloWorld(id);
        Assert.AreEqual("Hello World", helloWorld);
    }
}

On the HTML report, it displays three different parameters which caused it to go down different routes.  For 0, it worked as expected. For 1 and 2, the test failed.

To prove this, I wrote a classic test which worked successfully. 

[Test]
public void HelloWorld()
{
    HelloWorld h = new HelloWorld();
    string helloWorld = h.GetHelloWorld(0);
    Assert.AreEqual("Hello World", helloWorld);
}

What has Pex done for us? Well, it has identified one working parameter and two parameters which will cause the test to fail.

Summary

This is a huge framework and I have only just scratched a very small surface area. I haven't even touched the more advance parts (I will need to read the documentation for that).  I will be posting more, writing more and generally seeing what this framework is actually all about. Still not 100% convinced about this framework, I'm saving that until I see more.  Not being able to use it against a commercial application might limit my venture.

Labels: Pex, TDD, Testing

An article I have wrote about Extending NUnit has just been published on the Simple Talk website. In the article I discuss how to extend the framework to support your own custom attributes, the article includes sample code about how I built attributes such as SqlServerDataSource and ExecuteSql for use when writing your test code.

You can read the article here : Testing Times Ahead- Extending NUnit

-------------------------------------------------

Simple-Talk is an online technical journal and community hub for working SQL Server and .NET developers and administrators, as well as enthusiasts and those taking their first steps with these Microsoft technologies. Simple-Talk is a growing site, currently with 150,000 subscribers, who each receive a twice-monthly newsletter providing highlights from the Simple-Talk website along with updates and announcements concerning Red Gate tools

Labels: NUnit, Red Gate, TDD, Testing

I've been wanting to write about this for a while however haven't really had the chance.  While prepping for a NxtGenUG session I came across the TestSuite attribute.  By using this, you can build up your tests to execute dynamically and provide the parameters when the tests are loaded into the runner and framework.  This allows for some great flexibility and extendibility, especially if you don't know all of the possible test values at design-time and want values to be easily added (adding a row to the database, or dropping another file into a directory).

To take advantage of this, you need to use the TestSuiteFixtureAttribute at the top of your class.

[TestSuiteFixture]
public class TestSuiteTests

You can then create the test methods you want to be executed in order to test your system.  Two important points, they must take a parameter of type Object and return an Object - the rest is up to you. In the first test, I just write out the ToString value to the console,  in the second I check to see if the value is between 0 and 5.

public Object MethodUnderTest(object testData)
{
    Console.WriteLine("MethodUnderTest Executed");
    StringAssert.Contains(testData.ToString(), "Parameter");
    return null;
}

public Object AnotherMethodUnderTest(object testData)
{
    Console.WriteLine("AnotherTest");
    string testDataString = testData.ToString();

    int testExecution = Convert.ToInt32(testDataString[testDataString.Length -1].ToString());
    Assert.Between(testExecution, 0, 5);

    return null;
}

The final stage is to create the method which will programmatically create all of your tests and test parameters.

The method below is marked with the TestSuiteAttribute so the framework knows this can create tests.  Inside the method, we create a new TestSuite object with a name to identify it, we can then add new items into the collection which are then returned to the framework and executed.  The parameters to add are:

1. Name of test
2. Method to call (the test itself) as a TestDelegate object
3. Any parameters for the test

[TestSuite]
public ITestSuite Suite()
{
    TestSuite suite = new TestSuite("NewSimpleSuite");

    for (int i = 0; i < 10; i++)
    {
        suite.Add("MethodUnderTest" + i, new TestDelegate(this.MethodUnderTest), "Parameter" + i);
        suite.Add("AnotherMethodUnderTest" + i, new TestDelegate(this.AnotherMethodUnderTest), "Parameter" + i);
    }

    return suite;
}

The above code creates 20 tests, 10 for each test method. In the UI, the tests are all loaded correctly, as you can see each one is named correctly based on our code.

Very powerful attribute! But use the power wisely...

Download code sample here - http://blog.benhall.me.uk/Code/MbUnit/TestSuiteTests.cs.txt

Labels: MbUnit, TDD, Testing

After using WaTiN, I have been thinking about UI Testing for WinForms, if it's possible and if it's even worth it. On the MbUnit mailing list I posted some syntax for an approach to WinForms and I had some good ideas, I brought up the subject again at Alt.Net.UK and while people have had success using WaTiN, they didn't seem that interested in WinForm testing. I know others had been talking about WPF Testing during the day and problems with it.

As it happens, I read on Jeremy Miller's blog that Thoughtworks have released 'Project White' which is a UI Testing framework for WPF, WinForms, Win32 and SWT (Java) and works based on Microsoft's UIAutomation library and windows messages. Sounds promising so I decided to take a closer look, this post just discusses me playing around with the framework and a simple form to get an understanding of how it works.

Firstly, I created a standard Windows Forms application with just a single form. First test - does it display?

The form looks like this:

Using White and MbUnit, the test looks like this:

private const string path = @"..\..\..\White_HelloWorld\bin\Debug\White_HelloWorld.exe"; #1 [Test] public void ApplicationLaunch_NoArgs_Form1Displayed() { Application application = Application.Launch(path); #2 Window window = application.GetWindow("Form1", InitializeOption.NoCache); #3 Assert.IsNotNull(window); Assert.IsTrue(window.DisplayState == DisplayState.Restored); #4 application.Kill(); #5 }

#1 We need to define the path to our executable. This is fine if you know your always going to be building into the same folder (both test and live assemblies), bit difficult when you have separate output directories. #2 I then use White to execute the exe #3 Once the application has launched, we get the form displayed as an object. This works based on the form's title - in this case, Form1 #4 I then check the Window state to see if it has been displayed #5 Finally, I close the application.

That's a very basic test. Let's add some functionality and explore the framework in more depth. What happens if the framework cannot find the form?

[Test] [ExpectedException(typeof(Core.UIItems.UIActionException))] public void ApplicationLaunch_NoArgs_Form2NotDisplayed() { Application application = Application.Launch(path); Window window = application.GetWindow("Form2", InitializeOption.NoCache); application.Kill(); }

White will attempt to find a window called Form2, if the timeout expires it throws the UIActionException. This is the same if it cannot find a control on the form.

To make this more interesting, I created an additional form with some buttons and labels.

The first button has a simple action, when you click it the text of the button changes to be Hello World!!. We can then create a test for this as follows:

[Test] public void ButtonClickable_btnClick1_ChangesText() { Application application = Application.Launch(path); Window window = application.GetWindow("White Hello World", InitializeOption.NoCache);

Button button = window.Get<Button>("btnClick1"); #1 //Moves the mouse to click the button

button.Click(); #2 Assert.AreEqual("Hello World!!", button.Name); #3 }

#1 Using the Get method, we can give it the type and name of the control we want to access. #2 We can then call the Click method which will move the mouse cursor over to the button and click it. #3 We can then verify that the action was correctly performed, in this case the Name has changed.

Sometimes, we want to be more flexible than referring to the object by it's name. As such, we can use the SearchCriteria object which allows us to access the control in different ways, for example by it's text:

SearchCriteria searchCriteria = SearchCriteria.ByText("Click Me!"); Button button = window.Get<Button>(searchCriteria);

One problem I did encounter was with unhandled exceptions. With one of my buttons, when clicked it will cause an exception to be thrown.

The test looks like this:

[Test] //Doesn't work? No way to get the exception...I guess the test would fail so you would reproduce manually. public void ClickButton_ThrowsException() { Application application = Application.Launch(path); Window window = application.GetWindow("White Hello World", InitializeOption.NoCache);

Button button = window.Get<Button>("throws"); button.Click(); Assert.AreEqual("Hello World!!", button.Name); }

The test fails because of the assertion, sadly it doesn't report back saying that an exception was thrown which I would have liked.

failed: Equal assertion failed: [[Hello World!!]]!=[[Throws Exception]]

Another problem which I wanted to see if the framework would handle was with message boxes. When clicking btnMsg, a message box is displayed on screen. Using the framework, we can use the MessageBox() method, giving it the title of the message box in order to get a Window object (would have preferred this to be a MessageBox object). We can then treat it just like a normal window which is nice.

Button button = window.Get<Button>("btnMsg"); button.Click(); Window messageBox = window.MessageBox(""); messageBox.Close();

One last simple test I wanted to perform was how to verify that when a button is pressed, that a label is updated correctly.

[Test] public void ClickButton_btnLabelText_ChangesLabel() { Application application = Application.Launch(path); Window window = application.GetWindow("White Hello World", InitializeOption.NoCache);

Button button = window.Get<Button>("btnLabelText"); button.Click(); Label label = window.Get<Label>("lblText"); #1 Assert.AreEqual("Updated", label.Text); application.Kill(); }

#1 We can access a label in the same way we access another other control, such as a button. No problems at all. There are other objects for difference controls available.

One thing about all of these tests is that they aren't very readable, we have a lot of noise about getting access to the objects which is making it harder to read what the test is actually testing. One quick way to improve readability is to move the launch and kill calls into Setup and Teardown methods.

private Application _app = null; [SetUp] public void Setup() { _app = Application.Launch(path); }

[TearDown] public void Teardown() { _app.Kill(); }

But it can be cleaned up more, one technique I've discussed before with WaTiN is to create a wrapper around the UI and test against that to make our tests more readable and less fragile.

I've taken the code from the last test and refactored it into a wrapper. The wrapper looks like this:

public class Test2 #1 { private Application _host = null; private Window _main = null; public Test2(Application host) #2 { _host = host; _main = _host.GetWindow("White Hello World", InitializeOption.NoCache); }

public Window Main { get { return _main; } }

public Button btnLabelText #3 { get { return Main.Get<Button>("btnLabelText"); } #4 }

public Label lblText { get { return Main.Get<Label>("lblText"); } } }

#1 This is the same name as the Form in the code under test for readability #2 Pass in the application so we can gain access to the running exe #3 This could be called anything for readability #4 Use the same code to access the button and return it to the calling test.

Using this wrapper and the Setup/Teardown, the same test as before would be this:

[Test] public void ClickButton_btnLabelText_ChangesLabel_ImprovedSyntax() { Test2 form = new Test2(_app); form.btnLabelText.Click();

Assert.AreEqual("Updated", form.lblText); }

This could be reduced more by moving the form creation into Setup as well. By using this simply step, we have made the tests more much readable. The other advantage is that if the name of a button changes, we only have to change the wrapper and not all of the dependent tests.

One final problem you might encounter is finding out what each control on the form is actually called. The .Net 3.0\Windows SDK includes a tool called UISpy. This allows you to see all the properties on a running application, as such all the information required for use with White.

Another good tool is HawkEye (Updated Link).

In summary, I'm really impressed with the framework. There are a few missing features, but hopefully they can be added over time, the fact that the framework does multiple different UI technologies is great, means you don't have to worry about what to use to write your UI tests or if your UI deals with the different technologies. Can't wait to use it on a real project...

Keep an eye out for a post how to use this with WPF and more advanced scenarios.

Project White Homepage - http://www.codeplex.com/white

Code Sample - http://blog.benhall.me.uk/Code/ProjectWhite/White_HelloWorld.zip

Labels: Project White, TDD, Testing

So far, I have discussed XUnit and the XUnit Extensions but in this post I wanted to discuss how to create your own extensions.  Building your own custom extensions is a very useful as it can make your tests much more readable and maintainable. Especially if you have a lot of tasks which need to be performed before or after your tests are executed.

XUnit provides five main base classes which you can build your extensions upon. 

FactAttribute

The Fact attribute is the main attribute used in XUnit to identify a method to execute and return the result of.  By creating a subclass of FactAttribute we can modify how the test runner should treat the method and allow for customisation. For example, the Theory attribute, which allows for data driven tests, is based on this attribute. The theory attribute customizes how the framework should execute the tests as it creates a separate test to execute based on each DataAttribute added to the test method - this is how TD.net can identify separate passing/failing tests.

If you want to create your own subclass of FactAttribute then the most common scenario will be to subclass from the FactAttribute class, and then override the EnumerateTestCommands method.  This is used to enumerate over the test commands system the test object and it returns an instance of ITestCommand per execution of a test method. By overriding this method, you take control of the test methods to be executed.

ITestCommand is the interface of test execution to be returned, which will be executed by the runner.  The interface has three items, a name property for the name of the test, a parameters property for all the properties of the test and a Execute method which accepts a testClass object.  The Execute method is called in order for the method to be executed. If you just want the standard test execution, then the base concrete class is TestCommand.

Let's have a look at a real example.  Within the Samples project (downloadable from codeplex site) there is a RepeatAttribute class.  The attribute on a method would look like this:

[RepeatTest(5)]
public void RepeatingTestMethod()

The implementation of the attribute overrides the EnumerableTestCommands method, and for the repeatCount (set in constructor) it returns a new TestCommand object to be executed.

protected override IEnumerable<ITestCommand> EnumerateTestCommands(MethodInfo method)
{
    for (int index = 0; index < repeatCount; index++)
        yield return new TestCommand(method);
}

In summary, EnumerateTestCommands defines what tests to execute. ITestCommand allows you to define what happens then they are executed.

BeforeAfterTestAttribute

The BeforeAfterTestAttribute is a very interesting attribute which allows for a lot of interesting attributes.  This attribute allows us to hook into the execution of the tests before and after they have been executed.  Its very simply to implement aswell, BeforeAfterTestAttribute is an abstract class with two virtual methods - before and after.  Implement either or both in your attribute, add it to your test/fact and it will be executed.

Again, in the samples solution there is an example of how to use this. It includes a TracingSplicerAttribute which outputs the name.  The test looks like this:

[Fact, TracingSplicer]
public void TestThis()
{
    Console.WriteLine("I'm inside the test!");
}

The attribute code is this:

public class TracingSplicerAttribute : BeforeAfterTestAttribute
{
    public override void Before(MethodInfo methodUnderTest)
    {
        Console.WriteLine("Before : {0}.{1}", methodUnderTest.DeclaringType.FullName, methodUnderTest.Name);
    }

    public override void After(MethodInfo methodUnderTest)
    {
        Console.WriteLine("After : {0}.{1}", methodUnderTest.DeclaringType.FullName, methodUnderTest.Name);
    }
}

When executing the test, the output is:

Before : Example.TestThis
I'm inside the test!
After : Example.TestThis

This method allows for implementation of Rollback and RestoreDatabase (MbUnit) as they are required to be executed before and after the test but don't interact with the actual execution.

DataAttribute

The DataAttribute allows us to set data providers for use with the Theory attribute. DataAttribute is an abstract class which has a single method, IEnumerable <object []> GetData(MethodInfo methodUnderTest);.  This means that any data you want to use within your tests has to be returned as an object[] containing your different test values. The best implementation of this to look at is the OleDbDataAttribute.

public override IEnumerable< object[] > GetData(MethodInfo methodUnderTest)
{
    DataTable table = new DataTable();

    using (OleDbDataAdapter adapter = new OleDbDataAdapter(selectStatement, connectionString))
        adapter.Fill(table);

    foreach (DataRow row in table.Rows)
        yield return row.ItemArray;
}

This executes the database query and pulls everything into a DataTable in memory.  It then uses the ItemArray method which returns a object[] containing all the columns.  This together with the yield statement means every row is a separate test data set.  Remember the TheoryAttribute then takes each object[] returned and creates a TestCommand object which is a separate test execution.  The end result is a test execution object for each data item which the runner can then execute independently.

If you have a special testing scenario which requires data from a certain source then using this attribute could make your tests more streamlined and a lot easier to maintain without much effort in implementing the attribute.

TraitAttribute

This attribute allows us to decorate the test method with name/value pairs.  Within the EnumerateTestCommands method, you can access these attributes on the MethodInfo by using the GetCustomAttributes() method.  The samples provide us with a CategoryTraitAttribute

[Fact, Category("Slow Test")]
public void LongTest()

With the implementation just looking like this, where it hard codes the key "Category" as we know what the key will be. 

public class CategoryAttribute : TraitAttribute
{
    public CategoryAttribute(string category)
        : base("Category", category) {}
}

We can then use GetCustomAttributes to get type of CategoryAttribute attributes.

RunWithAttribute

This attribute enables you to specify a different framework to execute the tests against, this is the base attribute for the RunWithNUnitAttribute.  If you are interested in this I recommend you download the xunitext source code (available from CodePlex) and look how the NUnit Attribute was implemented.

However, if you are really interested in creating an custom extension, I really recommend you download the source code to XUnit and XUnit Extensions to see how the existing attributes have been created.

Labels: TDD, Testing, XUnit

Just an early heads up, I will be doing a Test Driven Development talk at NxtGenUG Coventry on Monday May 19th.

My plan for the talk is to start with nothing, and build up a application using a test driven development approach.  Unlikely it will be a feature complete SubText level application, but my aim is to demonstrate how to get started with TDD with some real application features, how to take advantage of mocking and of course some of the excellent MbUnit features...

Talk Abstract

Red, Green, Refactor!

Starting to unit test your first project is difficult, where to start? What to test? How do you even get started? In this session, Ben starts from scratch and implements an ASP.net 2.0 application using test driven development techniques. The application will have to deal with real world situations such as databases, web services and even some users! Ben will demonstrate how to design the application for testability and how unit testing and mock frameworks can make your life easier and your tests less fragile. At the end, will the tests go green?

Register:  http://www.nxtgenug.net/ViewEvent.aspx?EventID=112

Labels: MbUnit, NxtGenUG, TDD

Continuing in my writing about MbUnit, this post will focus on the TypeFixtureAttribute. The TypeFixture allows a set of tests to focus on testing a certain type of class (generally an interface) and to provide objects of that type to be tested against.

This method is really useful if you have a collection of objects which all have similar behaviour which requires testing. Instead of having to duplicate your test code, you can define the types to be tested with instances of the tests being passed into the test being executed.

In my fake system being tested, I have a collection of objects which all implement a INotNull interface. I need to test to make sure that the implementation of INotNull works correctly, that not nulls are returned when GetNotNullValues() is called. My 'Production' code is this:

public interface INotNull { IEnumerable GetNotNullValues(); }

public class TypeA : INotNull { public override string ToString() { return "TypeA"; }

public IEnumerable GetNotNullValues() { yield return "TypeA"; } }

public class TypeB : INotNull { public override string ToString() { return "TypeB"; }

public IEnumerable GetNotNullValues() { yield return 1; yield return 2; yield return 3; yield return 4; } }

public class TypeC : INotNull { public override string ToString() { return "TypeC"; }

public IEnumerable GetNotNullValues() { yield return new object(); yield return null; yield return new object(); } }

The code looks fine, apart from TypeC which returns a null. In a standard unit testing framework, I would have to create multiple test fixtures for each type in the system and duplicate all of the tests. TypeFixture makes our life easier by allowing us to define a single fixture and test all the objects we want. Let's have a look at the test code.

[TypeFixture(typeof(INotNull), "Tests the INotNull interface.")] #1 public class INotNullTests

Instead of defining a TestFixture at the top of our class, we define a TypeFixture and give it the typeof object we will be testing. We then need to tell it all the objects to use.

[Provider(typeof(TypeA))] #1 public TypeA ProvideTypeA() { return new TypeA(); #2 }

#1 We use the ProviderAttribute and provide the type of object we are returning as a parameter.

#2 We then create a new instance of the object and return it to the caller.

We can use the Provider attribute on as many methods as required to cover all of our objects being tested. The other two objects being tested are TypeB and TypeC.

[Provider(typeof(TypeB))] public TypeB ProvideTypeB() { return new TypeB(); }

[Provider(typeof(TypeC))] public TypeC ProvideTypeC() { return new TypeC(); }

Now we have that in place, we need to write out tests. The test is a standard test, however accepts the type (as defined in the TypeFixture) as a parameter.

[Test] public void TestINotNullValuesNotNull(INotNull instance) { Console.WriteLine(instance.ToString());

foreach (object o in instance.GetNotNullValues()) { Assert.IsNotNull(o, "Instance: {0}. Object was null: {1}", instance, o); } }

We can then use that object within our test to verify it works correctly. This test method will be called once per object defined by a provider attribute. In this case, three tests are executed with TypeC test failing.

Adding more tests to the fixture is very quick and easy and with the TypeFixture will result in it being used to test multiple objects. Adding another implementation of the interface is easy as we define another method allowing it to be picked up by all the existing tests.

Labels: MbUnit, TDD, Testing

In this post, I want to talk about a little know feature of MbUnit which is the CombinationalTestAttribute.  This was originally described on Peli's blog, but I thought I would revisit the subject.

The combinational test attribute allows us to do pair-wise testing.  This is where we take two inputs (A and B) and test them against each other ({A1, B1}, {A1, B2}, {A1, B3}).

When implementing this system, the OUT (Object Under Test) needs to implement an interface.  In this case a ICount

public interface ICount
{
    int Count(string s);
}

We then implement a X counter which will the number of X's which appear in a string.

public class XCounter : ICount
{
    public int Count(string s)
    {
        int count = 0;
        foreach (char c in s)
            if (c == 'x')
                count++;
        return count;
    }
}

For the purposes of demo'ing, a BadCounter is also implemented.

public class BadCounter : ICount
{
    public int Count(string s)
    {
        return 2;
    }
}

So that would be our production code which we want to test.  Using just standard TestFixture and Test attributes, our tests would have to look something like this:

[TestFixture]
public class ICountTests_Orginal
{
    [Test]
    public void BadCounter_Count_NoX()
    {
        ICount counter = new BadCounter();
        int result = counter.Count("");
        Assert.AreEqual(0, result);
    }

    [Test]
    public void BadCounter_Count_TwoX()
    {
        ICount counter = new BadCounter();
        int result = counter.Count("XX");
        Assert.AreEqual(2, result);
    }

    [Test]
    public void BadCounter_Count_XSpaceYSpaceZ()
    {
        ICount counter = new BadCounter();
        int result = counter.Count("X Y Z");
        Assert.AreEqual(1, result);
    }

    [Test]
    public void XCounter_Count_NoX()
    {
        ICount counter = new XCounter();
        int result = counter.Count("");
        Assert.AreEqual(0, result);
    }

    [Test]
    public void XCounter_Count_TwoX()
    {
        ICount counter = new XCounter();
        int result = counter.Count("XX");
        Assert.AreEqual(2, result);
    }

    [Test]
    public void XCounter_Count_XSpaceYSpaceZ()
    {
        ICount counter = new XCounter();
        int result = counter.Count("X Y Z");
        Assert.AreEqual(1, result);
    }
}

This is only testing the two objects (imagine 100s) with three test cases (imagine 1000s).  This would very quickly become unmanageable and a maintenance nightmare. We can make this much more effective using Combinational.

First part of configuring this is to setup a Factory for the interface.  This will produce all the objects we want to be tested.

[Factory(typeof(ICount))]
public IEnumerable Counters()
{
    yield return new BadCounter();
    yield return new XCounter();
}

Next, we create a factory of test cases.

[Factory(typeof(CombinationTestItem))]
public IEnumerable Strings()
{
    yield return new CombinationTestItem("", 0);
    yield return new CombinationTestItem("aaa", 0);
    yield return new CombinationTestItem("x", 1);
    yield return new CombinationTestItem("xa", 1);
    yield return new CombinationTestItem("xax", 2);
    yield return new CombinationTestItem("X x", 2);
    yield return new CombinationTestItem("X X", 2);
}

I've created a helper class within my code called CombinationTestItem.  This is used just to store the input and expected output so I can access them within my test. This could store as much or as little as you want, only restriction is that the Factory must return IEnumerable.

public class CombinationTestItem
{
    public CombinationTestItem(string value, int xcount) { Input = value; ExpectedOutput = xcount; }
    public string Input;
    public int ExpectedOutput;
}

The last part is to actually create the test.

[CombinatorialTest]                       #1
public void CountIsExact([UsingFactories("Counters")] ICount counter,       #2
                         [UsingFactories("Strings")] CombinationTestItem test)            #3
{
    Assert.AreEqual(test.ExpectedOutput, counter.Count(test.Input));            #4
}

#1 Instead of using [Test] we use [CombinatorialTest]

#2 The first parameter of the test is the Factory of Counters we created.  We use the [UsingFactories] attribute to help the framework understand which factory to pull the data from

#3 Like with #2, we do the same for the test data factory.

#4  Finally, we call the count method on the counter object passed into the test with the value we set on our CombinationTestItem object.  We then verify that it matches the related ExpectedOutput.

When it comes to executing the test cases the framework creates a test for each ICount with each CombinationTestItem test case. The result is 14 test cases (Two ICount objects * 7 CombinationTestItem test cases).

Adding additional objects or test cases is simply a case of adding an additional line of code.

My own point about this attribute is that I have never found a real world situation where I have required this.  In theory (like above) it sounds great, but I'm but sure if it would actually solve any of my integration testing problems - I can't imagine it would solve any unit testing problems. If this does solve a problem for you, it would be great to hear about it.

Labels: MbUnit, TDD, Testing

Following on from my previous post on XUnit I decided to look at the Extensions library associated with the XUnit.net framework.  The XUnit Extensions project is available from http://www.codeplex.com/xunitext and is a separate assembly containing additional attributes (and I guess additional assertions later) and contains some really cool features.

To use the extensions, you need to reference both xunit.dll and xunitext.dll. Currently, there are six main extension attributes included within the assembly.

Theory - Support for Data Driven Tests

The first extensions I am going to look at are a set of extensions categorised as Theories. These theories allow external data to be used as the test data when executing. The extensions assembly has support for the following data providers:

1. ExcelData
2. InlineData
3. OleDbData
4. PropertyData
5. SqlServerData

To start with, I will look at the most basic - InlineData.  The InlineDataAttribute allows us to specify data to use within the test, executing the test each time for each attribute - similar to MbUnit's RowTest.  Below is a test which verifies that the length of a string is as expected.

[Theory]
[InlineData("hello", 5)]
[InlineData("hello world", 11)]
[InlineData("failing", 0)]
public void Theory_InLine(string value, int expectedLength)
{
    Console.WriteLine("String:" + value + " With expected length of " + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

Using TestDriven.NET, each attribute causes a separate test to be executed with the output shown below.

String:hello With expected length of 5
String:hello world With expected length of 11
String:failing With expected length of 0
TestCase 'IntroToXUnitExtensions.Ext.Theory_InLine("failing", 0)'
failed: Assert.Equal() Failure
Expected: 0
Actual:   7
    E:\Users\Ben Hall\Documents\Visual Studio 2008\Projects\IntroToXUnit\IntroToXUnitExtensions\Class1.cs(20,0): at IntroToXUnitExtensions.Ext.Theory_InLine(String value, Int32 expectedLength)

2 passed, 1 failed, 0 skipped, took 1.96 seconds

This is great for reducing the amount of duplicate test code and really helps readability as if you want to add additional test cases you just include another attribute.  Sadly, the ReSharper plugin just thinks this is a single test - instead of three separate tests like TD.net does.  Having it just as a single test makes it difficult to identify the failing test.

Moving on, next is the PropertyData attribute.  Property Data allows you to specify a property which returns an IEnumerable<object[]>.  Below is my property which returns the same data as in the above test. I've hard coded the values, but this could some from another source to meet your own requirements.

public static IEnumerable<object[]> MyTestData
{
  get
  {
       yield return new object[] { "hello", 5 };
       yield return new object[] { "hello world", 11 };
       yield return new object[] { "failing", 0 };
  }
}

With the test, you define a PropertyDataAttribute passing in the name of the Property as a string argument.  The test itself then should accept the same parameters as being returned within the object array (in this case, a string and int).  The code to implement this is as below.

[Theory]
[PropertyData("MyTestData")]
public static void Theory_Property(string value, int expectedLength)
{
    Console.WriteLine("String:" + value + " With expected length of " + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

When it hits the failing test, TD.net outputs the following.  I think this might be a bug as the output isn't as clean as Inline. ReSharper still threats this as a single test.

String:hello With expected length of 5
String:hello world With expected length of 11
String:failing With expected length of 0
TestCase 'M:IntroToXUnitExtensions.Ext.Theory_Property(System.String,System.Int32)'
failed: Exception has been thrown by the target of an invocation.
    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Xunit.EqualException: Assert.Equal() Failure
    Expected: 0
    Actual:   7
    E:\Users\Ben Hall\Documents\Visual Studio 2008\Projects\IntroToXUnit\IntroToXUnitExtensions\Class1.cs(39,0): at IntroToXUnitExtensions.Ext.Theory_Property(String value, Int32 expectedLength)
       --- End of inner exception stack trace ---
.......

Some 'limitations', not sure if limitations is the right word - more rules of usage.  The property must be within the same class as the test being executed and the property must return IEnumerable<object[]<object []>>.  However the advantage of this is what we can reuse the property in multiple different tests instead of redefining the test cases for each test like with Inline.

The next set of attributes pull data from an external location.  The first is ExcelData, this allows us to use a Excel spreadsheet as the source of the data for the tests. By using an excel spreadsheet, we can easily edit and maintain the test data separate to the actual code but it also removes any dependency on having a database setup to hold out test data.  The test with the attribute would look something like below.  You specify the Excel file to use and then a query which will be used to pull the data, the TestData name is actually a Named Range in Excel so the test knows what data to use.

[Theory, ExcelData(@"ExcelTestData.xls", "select * from TestData")]
public void Theory_Excel(string value, int expectedLength)
{
    Console.WriteLine("String:" + value + " With expected length of " + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

However, the first time I executed this test I got an exception which was something like:

"The Microsoft Jet database engine could not find the object..."

The reason for this was because I hadn't included the ExcelTestData.xls file into the project and set Copy to Output to Copy always.

The file ExcelTestData contains the following, note you must include a header (otherwise your first test case is ignored):

To set the data as a named range, right the block of data (including header), right click and select Name a range

If you want to edit or delete an existing name, then you need to use the Name Manager under the Formulas tab

Important note - it must be a Excel 2003 format - not the 2007 format.

Next is the SqlServerDataAttribute. This allows us to use data within a Sql Server Database and use a SELECT statement to return the test data we want to use within our test code.  This is great when combined with a tool such as Red Gate's SQL Data Generator.

The SqlServerDataAttribute accepts either Server, Database, SelectStatement or Server,Database,Username,Password,SelectStatement.

[Theory]
[SqlServerData("(local)", "IntroToXUnit", "SELECT teststring, expectedLength FROM TestDataTable")]
public void Theory_SqlServer(string value, int expectedLength)
{
    Console.WriteLine("String:" + value + " With expected length of " + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

One important note, the amount of columns returned from the select statement must match the parameters of the test.

Finally, OleDBDataAttribute works in a similar fashion to SqlServerDataAttribute however allows you to connect to other data sources apart from SqlServer.  The attribute is:

[OleDbDataAttribute(string connectionString, string SelectStatement);

AutoRollback

Similar to MbUnit's Rollback feature, the extensions library has an AutoRollback attribute. By adding this attribute to your test, the entire test will be wrapped within a transaction so once the test has finished (passed or failed) the transaction will be rolled back meaning any changes will be undone.  For example, if we delete the entire customers table, once the test has finished the changes will be rolled back - leaving the database in the same state as before the test.  This is very important for test isolation as we want each test to be independent of any other tests, when it comes to interacting with the database we especially don't want any tests making permanent changes to the database which is used by other tests.

[Fact]
[AutoRollback]
public void AutoRollback()
{
    string connectionString = "Data Source=BIGBLUE;Initial Catalog=Northwind;Integrated Security=True";

    using (SqlConnection connection = new SqlConnection(connectionString))
    {
        connection.Open();
        //SqlCommand command = new SqlCommand("DELETE FROM Customers", connection);
        //command.ExecuteNonQuery();

        SqlCommand command2 = new SqlCommand("SELECT CustomerID FROM Customers", connection);
        SqlDataReader reader = command2.ExecuteReader();
        try
        {
            while (reader.Read())
            {
                Console.WriteLine(String.Format("{0}", reader[0]));
            }
        }
        finally
        {
            // Always call Close when done reading.
            reader.Close();
        }
    }
}

The Rollback attribute also includes additional properties, IsolationLevel, TransactionScopeOption and a Timeout. 

FreezeClock

Unit testing anything to do with the system clock is always difficult!  The problem is that you don't know what time your tests will be executed, and you definitely don't want to limit them to only running at one extract moment.  In my article Beginning to Mock with Rhino Mocks and MbUnit - Part 1: ASP Alliance I spoke about mocking the date time to get around this.

When I first heard about this I was intrigued how they managed to freeze the clock, turns out it wasn't so special - but still useful.  What the extensions framework provides is a Clock class which is a wrapper around the .Net DateTime class.  It properties three properties to access the current DateTime - Now, Today, UtcNow.

In your production code, you use this Clock class instead of DateTime to offer all of your date functionality. When dealing with dates, you would do something like below (of course, instead of outputting to the console you would do something real like query the database).

public class ProductionCode
{
    public void ProcessSomethingInvolvingTime()
    {
        Thread.Sleep(1500);
        Console.WriteLine(Clock.Now);
    }
}

To test this we would then add the FreezeClock attribute, which automatically freezes the Clock object for us. We then use the same Clock.Now method to access the now frozen time.

[Fact, FreezeClock]
public void FrozenProductionCode()
{
    DateTime expected = Clock.Now;
    Console.WriteLine(expected);

    ProductionCode pc = new ProductionCode();
    pc.ProcessSomethingInvolvingTime();
}

Outputting both to the console, we verify that the two dates are actually the same - impossible with the standard DateTime.

07/01/2008 00:48:26
07/01/2008 00:48:26

1 passed, 0 failed, 0 skipped, took 3.56 seconds.

However, I really really don't like the fact that I would have to use the Clock class and reference xunitext within my production code.  Just gives me a nasty feeling and I can't imagine this is how Brad and James wanted it to be used.  Instead, I would prefer to use a type of Dependency Injection (well Date Injection), passing in a DateTime object into the method under test.

public void ProcessSomethingInvolvingTimeDI(DateTime now)
{
    Thread.Sleep(1500);
    Console.WriteLine(now);
}

Our test now looks like this.  Instead of letting the method obtain the time itself we give it the DateTime object (which we got from Clock.Now) to use.

[Fact, FreezeClock]
public void FrozenProductionCodeDI()
{
    DateTime expected = Clock.Now;
    Console.WriteLine(expected);

    ProductionCode pc = new ProductionCode();
    pc.ProcessSomethingInvolvingTimeDI(expected);
}

When we run the test, the date time is still the same and still frozen. However, if we are doing this why not just use the DateTime object itself?

[Fact]
public void DateTimeProductionCodeDI()
{
    DateTime expected = DateTime.Now;
    Console.WriteLine(expected);

    ProductionCode pc = new ProductionCode();
    pc.ProcessSomethingInvolvingTimeDI(expected);
}

I think its something useful, but not sure if I like it.

AssumeIdentity

Assume Identify allows us to change the Current Principal (the user currently executing the assembly).  This is used for various things such as Role management and within CAS (Code Access Security).

[Fact]
public void AssumeIdentityWithoutAttribute()
{
    Console.WriteLine("Username of Thread:" + Thread.CurrentPrincipal.Identity.Name);
}

What is outputted is nothing.

"Username of Thread: "

If we use the AssumeIdentityAttribute, we can pass in a RoleName for which the user executing the test will be a remember of.

[Fact, AssumeIdentity("RoleName")]
public void AssumeIdentityWithAttribute()
{
    Console.WriteLine("Username of Thread:" + Thread.CurrentPrincipal.Identity.Name);
    Assert.True(Thread.CurrentPrincipal.IsInRole("RoleName"));
}

If we run this test, what is outputted is the name of XUnit and IsInRole returns true.

"Username of Thread:xUnit"

If a method we was testing involved CAS as below, the user exectuing the test would have to be a remember of that role - not always the case for build servers.

[PrincipalPermission(SecurityAction.Demand, Role = "Administrators")]
public void AdminOnly()
{
    Console.WriteLine("Executing....");
}

With AssumeIdentity, the test can correctly execute the method.

[Fact, AssumeIdentity("Administrators")]
public void AssumeIdentityCAS()
{
    ProductionCode p =new ProductionCode();
    p.AdminOnly();
}

Trace

Adding Trace attribute to your tests causes the test name to be wrote out to the Trace listener before and after the test is being executed. Useful for debugging if your tests have a habit of stopped mid-execution.

RunWithNUnit

Finally, the RunWithNUnit attribute. This method actually lives in the xunitext.nunit.dll assembly.  To check that I wasn't going made, I created a separate project referencing NUnit 2.4, XUnit and XUnitExt.NUnit.  I then use the XUnit console to make sure I'm executing under XUnit.

The code should look something like this:

[XunitExt.NUnit.RunWithNUnit]
[NUnit.Framework.TestFixture]
public class OldNUnitTests
{
     [NUnit.Framework.Test]
     public void RunWithNUnit()
     {
          NUnit.Framework.Assert.AreEqual("Running with NUnit", "Running with NUnit");
     }
}

However, I couldn't get this to work correctly as I kept receiving an "error: Exception has been thrown by the target of an invocation. in TargetInvocationException" exception.  I posted on the forum, but at the time of posting no reply.

RunWithNUnit throws exception

In summary, I hope you have found this post useful when getting started using XUnit.net and their extensions. Any feedback/comments then please leave a comment on the post.

Labels: TDD, Testing, XUnit