Category: XUnit

xUnit Browser Attribute – Run Cross-Browser WatiN tests effortlessly

Recently I wanted to run a web UI test across multiple different browsers using WatiN. WatiN supports FireFox, IE and Chrome so that wasn’t a problem, however in order to test the application on difference browsers I would have to duplicate the logic multiple times. This harms readability and maintainability of your tests and is just pain wrong. As such, I created a very similar xUnit Browser Attribute, code can be found on my GitHub repository along with the binaries.

The concept is simple, have the instance of the browser created for you outside of your test and provided the instance as a argument during execution. By using attributes, you can define multiple different browsers, each one will generate a separate test. The advantage of having the tests in this fashion means that if a test one particular browser fails then you can quickly see the problem and the results form the other browsers. If they where just a single test, if one part of  test failed, you wouldn’t be able to see the results for the other part. This can make identify where the problems lies much more difficult. An example of the attribute and a test is below. 

public class Example
{
    [Theory]
    [Browser("IE")]
    [Browser("FireFox")]
    [Browser("Chrome")]
    public void Should_Click_Search_On_Google_And_Return_Results_For_AspNet(Browser browser)
    {
        browser.GoTo("http://www.google.co.uk");
        browser.TextField(Find.ByName("q")).Value = "asp.net";
        browser.Button(Find.ByName("btnG")).Click();
        Assert.True(browser.ContainsText("The Official Microsoft ASP.NET Site"));
    }
}

This example is made possible by taking advantage of the xUnit Extensibility model and being able to create objects dynamically in C#. The attribute code is below, as WatiN has a clean namespace structure creating an object based off the string of the name is easy. 

public class BrowserAttribute : DataAttribute
{
    public string Browser { get; set; }
    public BrowserAttribute(string browser)
    {
        Browser = browser;
    }
 
    public override IEnumerable GetData(MethodInfo methodUnderTest, Type[] parameterTypes)
    {
        Type type = Assembly.GetAssembly(typeof(Browser)).GetType("WatiN.Core." + Browser);
        return new[] { new[] {Activator.CreateInstance(type) }};
    }
}

With this attribute, you can now quickly and easily define the browsers you want the test to run against, without having to duplicate the test itself.

xUnit.GWT – Given When Then (GWT) extension for xUnit.net

One of the most important considerations when writing automated tests, especially customer acceptance tests is readability. I have been looking at how you could potentially create more readable tests targeted against C# applications. While I have spoke before about IronRuby and RSpecCucumber as customer acceptance tests for C# applications, this combination still has some performance issues together with an initial ‘fear factor’ for some users.

One of the reasons why I created this framework was based on previous experiences of introducing new concepts. Attempting to introduce too many new concepts to a team unfamiliar with the idea can be a huge challenge. Team members focus too much on the amount of information they have to learn instead of focusing on the concept and how it can be applied.

In an attempt to combat this, I have created a prototype which approaches the topic of creating customer acceptance tests in a similar fashion to cucumber, however in the familiarity of C#. While you could successfully write your tests like this, my aim is to use this as a gateway to move people onto other frameworks such as Cucumber via IronRuby.

The code for this can be found on my GitHub repository. For those of you following me on twitter, you will be aware that this code has been available for a while – I’ve only just managed to find time the time to write a post about the framework. If you want to stay up to date, I recommend to stay up to date is follow me on twitter, GitHub and of course my blog.

Back on to the prototype. It is called xUnit.GWT and builds on top of the amazing extensibility model xUnit has in place.

Example

[Feature("As a user... I want to... So that I...")]
public class This_is_my_story : This_is_my_story_Base
{
        [Scenario]
        public void when_transferring_between_two_accounts()
        {
            Given("both accounts have positive balances");           
            When("transfer money from a to b");
            Then("a should have the total balance of b", () => a.Balance.ShouldBe(3));
            Then("balance of b should be 0", () => b.Balance.ShouldBe(0));
        }
}

The above code is an example of how a test would look with this prototype. This can be executed using TestDriven.NET, ReSharper and the xUnit.Console itself.

First, lets focus on the class. The class has an attribute called Feature. This should define the feature which the various different scenarios relate to. The scenarios are methods within the class.  These methods has been flagged with the attribute Scenario so xUnit can detect and execute them.

Each scenario is then broken down into three different stages. Each of the Given, When, Then statements are in fact method calls with two overloads. One just accepts a string, while the other accepts a string and a delegate.

The reason we can simply pass in a string, is because the associated delegate has already been set within a base class.  An example base class is shown here.

public class This_is_my_story_Base : Story
{
    protected Account a;
    protected Account b;

    public This_is_my_story_Base()
    {
        SetGiven("both accounts have positive balances", () =>
                                                             {
                                                                 a = new Account();
                                                                 a.Balance = 1;
                                                                 b = new Account();
                                                                 b.Balance = 2;
                                                             });

        SetWhen("transfer money from a to b", () =>
                                                  {
                                                      a.Balance += b.Balance;
                                                      b.Balance = 0;
                                                  });
    }
}

The use of SetGiven or SetWhen allows us to store re-usable blocks of code. Using the string as a key, in our subclass we can use the same key to indicate which block of code should be executed. The aim is that these blocks of code should be reusable enough to allow different scenarios to make use of them and as such improve the readability of our tests.

While playing around, I found this sometimes caused too much overhead which is why you can simply pass in the delegate to execute within the method call. Using the explicit approach when defining your Then’Assertions can actually improve the readability.

With this structure in place, we can now start becoming more flexible. One thing I found when working on test harnesses at Red Gate is that you want to write the boiler plate test while it is fresh in your head, even if the actual implementation is not in place. Having these in place can also provide a good indication of the state of the project. If the test is still pending – it isn’t done.

To cope, you can pass in a delegate called Pending. When you execute the scenario the test will be skipped and not executed.

[Scenario]
public void test_marked_as_pending()
{
    Given("one pending block", Pending);
    Then("it should be marked as pending in the report");
}

The same applies in case you haven’t provided an implementation for anyone of the blocks you are attempting for use.

[Scenario]
public void pending_block()
{
    Given(“a block which has been defined”);
    When(“no action assigned”);
    Then(“it should be marked as pending”);
}

Yet, this is not the whole story. Another problem I commonly experienced was with the NUnit report embedded into CCNet. Even if you have take the effort to make your namespaces, classes and methods make sense I still found it difficult to read.

Taking advantage of the more readable tests, I have created a custom XSLT which can be applied to the xUnit report.

With a bit of styling and bold text in the right place, the report can be extremely readable. When the above examples are executed and the transform has been applied, the report looks like this.

Screenshot

You can easily see which scenarios are passing, failing or pending while also taking advantage of the readable blocks of code to provide more context about what was executed.

This is just a prototype. It is only one possible syntax which you could use. As such, I would be really interested to hear your feedback on this. The framework is fully usable, the binaries can be downloaded here. If your interested in the actual implementation, all the code is available on GitHub. If you have any questions, then please contact me via twitter.

xUnit TeamCity Integration

Last night I was doing some experiments with xUnit and TeamCity. TeamCity has a nice UI for reporting tests, however by default it only supports NUnit. Luckily, extending this to support other tests is easy as it just listens for MSBuild outputs in the right format (##teamcity[testStarted name=’Test1′]).

This worked and my tests where appearing in the report.

My TeamCity Runner

On the other side of the ocean, Brad was actually working on extending the existing xUnit MSBuild runner for TeamCity integration.

The advantage of Brad’s runner is that you don’t need to edit your MSBuild script, as long as your using the xunit msbuild task it will know when TeamCity executed the build and log to the correct place. Otherwise, it will log to the standard MSBuild output – very cool!

image

If you want to use this today, download the source code from CodePlex and build it yourself (Get3rdParty.cmd is a great script to run before you build), but I suggest you wait for the next release (1.0.0.4), where support will be included.

Documentation on how to use the MSBuild task can be found here, as I mentioned, using it via TeamCity is no difference.

Technorati Tags: ,

Creating your own XUnit Extension

So far, I have discussed XUnit and the XUnit Extensions but in this post I wanted to discuss how to create your own extensions.  Building your own custom extensions is a very useful as it can make your tests much more readable and maintainable. Especially if you have a lot of tasks which need to be performed before or after your tests are executed.

XUnit provides five main base classes which you can build your extensions upon. 

FactAttribute

The Fact attribute is the main attribute used in XUnit to identify a method to execute and return the result of.  By creating a subclass of FactAttribute we can modify how the test runner should treat the method and allow for customisation. For example, the Theory attribute, which allows for data driven tests, is based on this attribute. The theory attribute customizes how the framework should execute the tests as it creates a separate test to execute based on each DataAttribute added to the test method – this is how TD.net can identify separate passing/failing tests.

If you want to create your own subclass of FactAttribute then the most common scenario will be to subclass from the FactAttribute class, and then override the EnumerateTestCommands method.  This is used to enumerate over the test commands system the test object and it returns an instance of ITestCommand per execution of a test method. By overriding this method, you take control of the test methods to be executed.

ITestCommand is the interface of test execution to be returned, which will be executed by the runner.  The interface has three items, a name property for the name of the test, a parameters property for all the properties of the test and a Execute method which accepts a testClass object.  The Execute method is called in order for the method to be executed. If you just want the standard test execution, then the base concrete class is TestCommand.

Let’s have a look at a real example.  Within the Samples project (downloadable from codeplex site) there is a RepeatAttribute class.  The attribute on a method would look like this:

[RepeatTest(5)]
public void RepeatingTestMethod()

The implementation of the attribute overrides the EnumerableTestCommands method, and for the repeatCount (set in constructor) it returns a new TestCommand object to be executed.

protected override IEnumerable EnumerateTestCommands(MethodInfo method)
{
    for (int index = 0; index < repeatCount; index++)
        yield return new TestCommand(method);
}

In summary, EnumerateTestCommands defines what tests to execute. ITestCommand allows you to define what happens then they are executed.

BeforeAfterTestAttribute

The BeforeAfterTestAttribute is a very interesting attribute which allows for a lot of interesting attributes.  This attribute allows us to hook into the execution of the tests before and after they have been executed.  Its very simply to implement aswell, BeforeAfterTestAttribute is an abstract class with two virtual methods – before and after.  Implement either or both in your attribute, add it to your test/fact and it will be executed.

Again, in the samples solution there is an example of how to use this. It includes a TracingSplicerAttribute which outputs the name.  The test looks like this:

[Fact, TracingSplicer]
public void TestThis()
{
    Console.WriteLine(“I’m inside the test!”);
}

The attribute code is this:

public class TracingSplicerAttribute : BeforeAfterTestAttribute
{
    public override void Before(MethodInfo methodUnderTest)
    {
        Console.WriteLine(“Before : {0}.{1}”, methodUnderTest.DeclaringType.FullName, methodUnderTest.Name);
    }

    public override void After(MethodInfo methodUnderTest)
    {
        Console.WriteLine(“After : {0}.{1}”, methodUnderTest.DeclaringType.FullName, methodUnderTest.Name);
    }
}

When executing the test, the output is:

Before : Example.TestThis
I’m inside the test!
After : Example.TestThis

This method allows for implementation of Rollback and RestoreDatabase (MbUnit) as they are required to be executed before and after the test but don’t interact with the actual execution.

DataAttribute

The DataAttribute allows us to set data providers for use with the Theory attribute. DataAttribute is an abstract class which has a single method, IEnumerable GetData(MethodInfo methodUnderTest);.  This means that any data you want to use within your tests has to be returned as an object[] containing your different test values. The best implementation of this to look at is the OleDbDataAttribute.

public override IEnumerable< object[] > GetData(MethodInfo methodUnderTest)
{
    DataTable table = new DataTable();

    using (OleDbDataAdapter adapter = new OleDbDataAdapter(selectStatement, connectionString))
        adapter.Fill(table);

    foreach (DataRow row in table.Rows)
        yield return row.ItemArray;
}

This executes the database query and pulls everything into a DataTable in memory.  It then uses the ItemArray method which returns a object[] containing all the columns.  This together with the yield statement means every row is a separate test data set.  Remember the TheoryAttribute then takes each object[] returned and creates a TestCommand object which is a separate test execution.  The end result is a test execution object for each data item which the runner can then execute independently.

If you have a special testing scenario which requires data from a certain source then using this attribute could make your tests more streamlined and a lot easier to maintain without much effort in implementing the attribute.

TraitAttribute

This attribute allows us to decorate the test method with name/value pairs.  Within the EnumerateTestCommands method, you can access these attributes on the MethodInfo by using the GetCustomAttributes() method.  The samples provide us with a CategoryTraitAttribute

[Fact, Category(“Slow Test”)]
public void LongTest()

With the implementation just looking like this, where it hard codes the key “Category” as we know what the key will be. 

public class CategoryAttribute : TraitAttribute
{
    public CategoryAttribute(string category)
        : base(“Category”, category) {}
}

We can then use GetCustomAttributes to get type of CategoryAttribute attributes.

RunWithAttribute

This attribute enables you to specify a different framework to execute the tests against, this is the base attribute for the RunWithNUnitAttribute.  If you are interested in this I recommend you download the xunitext source code (available from CodePlex) and look how the NUnit Attribute was implemented.

However, if you are really interested in creating an custom extension, I really recommend you download the source code to XUnit and XUnit Extensions to see how the existing attributes have been created.

Technorati Tags: , ,

Introduction to XUnit.net Extensions

Following on from my previous post on XUnit I decided to look at the Extensions library associated with the XUnit.net framework.  The XUnit Extensions project is available from http://www.codeplex.com/xunitext and is a separate assembly containing additional attributes (and I guess additional assertions later) and contains some really cool features.

To use the extensions, you need to reference both xunit.dll and xunitext.dll. Currently, there are six main extension attributes included within the assembly.

Theory – Support for Data Driven Tests

The first extensions I am going to look at are a set of extensions categorised as Theories. These theories allow external data to be used as the test data when executing. The extensions assembly has support for the following data providers:

  1. ExcelData
  2. InlineData
  3. OleDbData
  4. PropertyData
  5. SqlServerData

To start with, I will look at the most basic – InlineData.  The InlineDataAttribute allows us to specify data to use within the test, executing the test each time for each attribute – similar to MbUnit’s RowTest.  Below is a test which verifies that the length of a string is as expected.

[Theory]
[InlineData(“hello”, 5)]
[InlineData(“hello world”, 11)]
[InlineData(“failing”, 0)]
public void Theory_InLine(string value, int expectedLength)
{
    Console.WriteLine(“String:” + value + ” With expected length of ” + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

Using TestDriven.NET, each attribute causes a separate test to be executed with the output shown below.

String:hello With expected length of 5
String:hello world With expected length of 11
String:failing With expected length of 0
TestCase ‘IntroToXUnitExtensions.Ext.Theory_InLine(“failing”, 0)’
failed: Assert.Equal() Failure
Expected: 0
Actual:   7
    E:UsersBen HallDocumentsVisual Studio 2008ProjectsIntroToXUnitIntroToXUnitExtensionsClass1.cs(20,0): at IntroToXUnitExtensions.Ext.Theory_InLine(String value, Int32 expectedLength)

2 passed, 1 failed, 0 skipped, took 1.96 seconds

This is great for reducing the amount of duplicate test code and really helps readability as if you want to add additional test cases you just include another attribute.  Sadly, the ReSharper plugin just thinks this is a single test – instead of three separate tests like TD.net does.  Having it just as a single test makes it difficult to identify the failing test.

image

Moving on, next is the PropertyData attribute.  Property Data allows you to specify a property which returns an IEnumerable.  Below is my property which returns the same data as in the above test. I’ve hard coded the values, but this could some from another source to meet your own requirements.

public static IEnumerable MyTestData
{
  get
  {
       yield return new object[] { “hello”, 5 };
       yield return new object[] { “hello world”, 11 };
       yield return new object[] { “failing”, 0 };
  }
}

With the test, you define a PropertyDataAttribute passing in the name of the Property as a string argument.  The test itself then should accept the same parameters as being returned within the object array (in this case, a string and int).  The code to implement this is as below.

[Theory]
[PropertyData(“MyTestData”)]
public static void Theory_Property(string value, int expectedLength)
{
    Console.WriteLine(“String:” + value + ” With expected length of ” + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

When it hits the failing test, TD.net outputs the following.  I think this might be a bug as the output isn’t as clean as Inline. ReSharper still threats this as a single test.

String:hello With expected length of 5
String:hello world With expected length of 11
String:failing With expected length of 0
TestCase ‘M:IntroToXUnitExtensions.Ext.Theory_Property(System.String,System.Int32)’
failed: Exception has been thrown by the target of an invocation.
    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> Xunit.EqualException: Assert.Equal() Failure
    Expected: 0
    Actual:   7
    E:UsersBen HallDocumentsVisual Studio 2008ProjectsIntroToXUnitIntroToXUnitExtensionsClass1.cs(39,0): at IntroToXUnitExtensions.Ext.Theory_Property(String value, Int32 expectedLength)
       — End of inner exception stack trace —
…….

Some ‘limitations’, not sure if limitations is the right word – more rules of usage.  The property must be within the same class as the test being executed and the property must return IEnumerable>.  However the advantage of this is what we can reuse the property in multiple different tests instead of redefining the test cases for each test like with Inline.

The next set of attributes pull data from an external location.  The first is ExcelData, this allows us to use a Excel spreadsheet as the source of the data for the tests. By using an excel spreadsheet, we can easily edit and maintain the test data separate to the actual code but it also removes any dependency on having a database setup to hold out test data.  The test with the attribute would look something like below.  You specify the Excel file to use and then a query which will be used to pull the data, the TestData name is actually a Named Range in Excel so the test knows what data to use.

[Theory, ExcelData(@”ExcelTestData.xls”, “select * from TestData”)]
public void Theory_Excel(string value, int expectedLength)
{
    Console.WriteLine(“String:” + value + ” With expected length of ” + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

However, the first time I executed this test I got an exception which was something like:

“The Microsoft Jet database engine could not find the object…”

The reason for this was because I hadn’t included the ExcelTestData.xls file into the project and set Copy to Output to Copy always.

The file ExcelTestData contains the following, note you must include a header (otherwise your first test case is ignored):

image

To set the data as a named range, right the block of data (including header), right click and select Name a range

image

If you want to edit or delete an existing name, then you need to use the Name Manager under the Formulas tab

image

Important note – it must be a Excel 2003 format – not the 2007 format.

Next is the SqlServerDataAttribute. This allows us to use data within a Sql Server Database and use a SELECT statement to return the test data we want to use within our test code.  This is great when combined with a tool such as Red Gate’s SQL Data Generator.

The SqlServerDataAttribute accepts either Server, Database, SelectStatement or Server,Database,Username,Password,SelectStatement.

[Theory]
[SqlServerData(“(local)”, “IntroToXUnit”, “SELECT teststring, expectedLength FROM TestDataTable”)]
public void Theory_SqlServer(string value, int expectedLength)
{
    Console.WriteLine(“String:” + value + ” With expected length of ” + expectedLength);
    Assert.Equal(expectedLength, value.Length);
}

One important note, the amount of columns returned from the select statement must match the parameters of the test.

Finally, OleDBDataAttribute works in a similar fashion to SqlServerDataAttribute however allows you to connect to other data sources apart from SqlServer.  The attribute is:

[OleDbDataAttribute(string connectionString, string SelectStatement);

AutoRollback

Similar to MbUnit’s Rollback feature, the extensions library has an AutoRollback attribute. By adding this attribute to your test, the entire test will be wrapped within a transaction so once the test has finished (passed or failed) the transaction will be rolled back meaning any changes will be undone.  For example, if we delete the entire customers table, once the test has finished the changes will be rolled back – leaving the database in the same state as before the test.  This is very important for test isolation as we want each test to be independent of any other tests, when it comes to interacting with the database we especially don’t want any tests making permanent changes to the database which is used by other tests.

[Fact]
[AutoRollback]
public void AutoRollback()
{
    string connectionString = “Data Source=BIGBLUE;Initial Catalog=Northwind;Integrated Security=True”;

    using (SqlConnection connection = new SqlConnection(connectionString))
    {
        connection.Open();
        //SqlCommand command = new SqlCommand(“DELETE FROM Customers”, connection);
        //command.ExecuteNonQuery();

        SqlCommand command2 = new SqlCommand(“SELECT CustomerID FROM Customers”, connection);
        SqlDataReader reader = command2.ExecuteReader();
        try
        {
            while (reader.Read())
            {
                Console.WriteLine(String.Format(“{0}”, reader[0]));
            }
        }
        finally
        {
            // Always call Close when done reading.
            reader.Close();
        }
    }
}

The Rollback attribute also includes additional properties, IsolationLevel, TransactionScopeOption and a Timeout. 

FreezeClock

Unit testing anything to do with the system clock is always difficult!  The problem is that you don’t know what time your tests will be executed, and you definitely don’t want to limit them to only running at one extract moment.  In my article Beginning to Mock with Rhino Mocks and MbUnit – Part 1: ASP Alliance I spoke about mocking the date time to get around this.

When I first heard about this I was intrigued how they managed to freeze the clock, turns out it wasn’t so special – but still useful.  What the extensions framework provides is a Clock class which is a wrapper around the .Net DateTime class.  It properties three properties to access the current DateTime – Now, Today, UtcNow.

In your production code, you use this Clock class instead of DateTime to offer all of your date functionality. When dealing with dates, you would do something like below (of course, instead of outputting to the console you would do something real like query the database).

public class ProductionCode
{
    public void ProcessSomethingInvolvingTime()
    {
        Thread.Sleep(1500);
        Console.WriteLine(Clock.Now);
    }
}

To test this we would then add the FreezeClock attribute, which automatically freezes the Clock object for us. We then use the same Clock.Now method to access the now frozen time.

[Fact, FreezeClock]
public void FrozenProductionCode()
{
    DateTime expected = Clock.Now;
    Console.WriteLine(expected);

    ProductionCode pc = new ProductionCode();
    pc.ProcessSomethingInvolvingTime();
}

Outputting both to the console, we verify that the two dates are actually the same – impossible with the standard DateTime.

07/01/2008 00:48:26
07/01/2008 00:48:26

1 passed, 0 failed, 0 skipped, took 3.56 seconds.

However, I really really don’t like the fact that I would have to use the Clock class and reference xunitext within my production code.  Just gives me a nasty feeling and I can’t imagine this is how Brad and James wanted it to be used.  Instead, I would prefer to use a type of Dependency Injection (well Date Injection), passing in a DateTime object into the method under test.

public void ProcessSomethingInvolvingTimeDI(DateTime now)
{
    Thread.Sleep(1500);
    Console.WriteLine(now);
}

Our test now looks like this.  Instead of letting the method obtain the time itself we give it the DateTime object (which we got from Clock.Now) to use.

[Fact, FreezeClock]
public void FrozenProductionCodeDI()
{
    DateTime expected = Clock.Now;
    Console.WriteLine(expected);

    ProductionCode pc = new ProductionCode();
    pc.ProcessSomethingInvolvingTimeDI(expected);
}

When we run the test, the date time is still the same and still frozen. However, if we are doing this why not just use the DateTime object itself?

[Fact]
public void DateTimeProductionCodeDI()
{
    DateTime expected = DateTime.Now;
    Console.WriteLine(expected);

    ProductionCode pc = new ProductionCode();
    pc.ProcessSomethingInvolvingTimeDI(expected);
}

I think its something useful, but not sure if I like it.

AssumeIdentity

Assume Identify allows us to change the Current Principal (the user currently executing the assembly).  This is used for various things such as Role management and within CAS (Code Access Security).

[Fact]
public void AssumeIdentityWithoutAttribute()
{
    Console.WriteLine(“Username of Thread:” + Thread.CurrentPrincipal.Identity.Name);
}

What is outputted is nothing.

“Username of Thread: “

If we use the AssumeIdentityAttribute, we can pass in a RoleName for which the user executing the test will be a remember of.

[Fact, AssumeIdentity(“RoleName”)]
public void AssumeIdentityWithAttribute()
{
    Console.WriteLine(“Username of Thread:” + Thread.CurrentPrincipal.Identity.Name);
    Assert.True(Thread.CurrentPrincipal.IsInRole(“RoleName”));
}

If we run this test, what is outputted is the name of XUnit and IsInRole returns true.

“Username of Thread:xUnit”

If a method we was testing involved CAS as below, the user exectuing the test would have to be a remember of that role – not always the case for build servers.

[PrincipalPermission(SecurityAction.Demand, Role = “Administrators”)]
public void AdminOnly()
{
    Console.WriteLine(“Executing….”);
}

With AssumeIdentity, the test can correctly execute the method.

[Fact, AssumeIdentity(“Administrators”)]
public void AssumeIdentityCAS()
{
    ProductionCode p =new ProductionCode();
    p.AdminOnly();
}

Trace

Adding Trace attribute to your tests causes the test name to be wrote out to the Trace listener before and after the test is being executed. Useful for debugging if your tests have a habit of stopped mid-execution.

RunWithNUnit

Finally, the RunWithNUnit attribute. This method actually lives in the xunitext.nunit.dll assembly.  To check that I wasn’t going made, I created a separate project referencing NUnit 2.4, XUnit and XUnitExt.NUnit.  I then use the XUnit console to make sure I’m executing under XUnit.

The code should look something like this:

[XunitExt.NUnit.RunWithNUnit]
[NUnit.Framework.TestFixture]
public class OldNUnitTests
{
     [NUnit.Framework.Test]
     public void RunWithNUnit()
     {
          NUnit.Framework.Assert.AreEqual(“Running with NUnit”, “Running with NUnit”);
     }
}

However, I couldn’t get this to work correctly as I kept receiving an “error: Exception has been thrown by the target of an invocation. in TargetInvocationException” exception.  I posted on the forum, but at the time of posting no reply.

RunWithNUnit throws exception

In summary, I hope you have found this post useful when getting started using XUnit.net and their extensions. Any feedback/comments then please leave a comment on the post.

Technorati Tags: , ,

Introduction to XUnit

While MbUnit is my unit testing framework of choice, I’ve heard some good reports about XUnit so I decided to take a closer look.  XUnit is a unit testing framework for the .Net platform, created by Brad Wilson and James Newkirk (One of the original developers of NUnit, both currently working for Microsoft) and is available to download from http://www.codeplex.com/xunit.  The framework itself is built using .Net Framework 2.0, doesn’t require any installation (XCopy) which makes it great for storing in source control and includes a TestDriven.NET runner, ReSharper runner and a console runner for executing the tests.

This framework takes a major departure from the other frameworks currently around. The major change is the naming of the attributes which you add to your test methods. Using MbUnitNUnit, your test fixture would look something like:

[TestFixture]
public class MbUnitAccountTests
{
    [TestFixtureSetUp]
    public void TestFixtureSetup()
    {
        Console.WriteLine(“We are currently in the constructor”);
    }

    [Test]
    public void MyFirstTest()
    {
        Console.WriteLine(“We are currently in my first test – well done!”);
        Assert.IsTrue(true);
    }

    [Test]
    public void MyFirstFailingTest()
    {
        Console.WriteLine(“We are currently in my first failing test – not so well done!”);
        Assert.IsTrue(false);
    }

    [TestFixtureTearDown]
    public void TestFixtureTeardown()
    {
        Console.WriteLine(“We are currently disposing.”);
    }
}

However, using XUnit your tests would look something like this:

public class AccountTests : IDisposable   #1
{
    public AccountTests()                          #2
    {
        Console.WriteLine(“We are currently in the constructor”);
    }

    [Fact]                                                   #3
    public void MyFirstTest()
    {
        Console.WriteLine(“We are currently in my first test – well done!”);
        Assert.True(true);
    }

    [Fact]
    public void MyFirstFailingTest()
    {
        Console.WriteLine(“We are currently in my first failing test – not so well done!”);
        Assert.True(false);
    }

    public void Dispose()                           #4
    {
        Console.WriteLine(“We are currently disposing.”);
    }
}

#1 – The first change is that we don’t need to add a TestFixture attribute to our class. Also, XUnit doesn’t have any Test or TestFixture Setup and Teardown attributes, however it does allow you to take advantage of the constructor and the Dispose method (if it implements IDisposable) so you can configure anything before the tests start executing.  You can’t have methods executing before and after each test separately.  The aim of this is to improve test isolation and to make the tests more readable.

#2 – As mentioned above, we use the constructor for the test class to setup anything

#3 – Before, we would say that a method would be a Test.  With XUnit they have taken a different approach and says that TDD is about example driven development and not actually testing (which I agree with to a point) as we use the methods to express intent of the class. By using FactAttribute, we are saying that this is true for this example.

#4 – As mentioned, we use Dispose to clean up anything once the tests have executed.

If we execute the above test cases using the ReSharper plugin, the output would be like below. If you look closely at the output, you can see we are constructing and disposing the test object for EACH TEST. MbUnitNUnit creates the object once and calls the test methods on the same object. Having an object per test improves the test isolation.

image

XUnit does however support a type of Test Setup – just not as we know it. In the code below, we create SomeFixture with a method.  We then use the IUseFixture interface to inject the fixture into the test class. Within the SetFixture method (which is on the interface), we can call any method on the SomeFixture object.

public class SomeFixture
{
    public void SomeMethod()
    {
        Console.WriteLine(“SomeFixture.SomeMethod()”);
    }
}

public class AccountTests : IUseFixture, IDisposable
{
    public void SetFixture(SomeFixture data)
    {
        Console.Write(“SetFixture? t “);
        data.SomeMethod();
    }
//…Your Tests
}

Now if we execute the tests, you can see the method is being called and outputted without us writing any additional code.

image

Moving on, the framework comes a good selection of assertions.  The assertions included are:

  • Contains and Contains
  • DoesNotContain and DoesNotContain
  • Empty
  • Equal
  • False
  • InRange
  • IsNotType and IsNotType
  • IsType
  • NotEmpty
  • NotEqual
  • NotInRange
  • NotNull
  • NotSame
  • Null
  • Same
  • Throws and Throws
  • True

The framework takes full advantage of generics.  Most of them are pretty standard, but two I want to pay special attention to is InRange and Throws.

Firstly, InRange is an interesting method and something which I have really wanted in the past.  The method takes three arguments, the actual value being tested and then the min/max value which the value should be in.  This checks that the actual value is between the two values.  This is useful when you know it should be in an range but you don’t care what the value actually is.

The first is a very simply test which tests to see if myValue is inbetween the min and max values.

[Fact]
public void InRangeInt()
{
    int myValue = 100;
    int min = 1;
    int max = 10;
    Assert.InRange(myValue, min, max);
}

This fails (of course…). The output from TD.net is as follows:

TestCase ‘IntroToXUnit.Tests.AccountTests.InRangeInt’ failed: Assert.InRange() Failure
Range:  (1 – 10)
Actual: 100
    E:UsersBen HallDocumentsVisual Studio 2008ProjectsIntroToXUnitIntroToXUnitClass1.cs(49,0): at IntroToXUnit.Tests.AccountTests.InRangeInt()

This also works with other objects, for example, below does the same thing but with dates.

[Fact]
public void InRangeDateTime()
{
    Assert.InRange(new DateTime(2009, 1, 1), new DateTime(2000, 1, 1), new DateTime(2008, 1, 6));
}

As such, the failing message is this:

TestCase ‘IntroToXUnit.Tests.AccountTests.InRangeDateTime’
failed: Assert.InRange() Failure
Range:  (01/01/2000 00:00:00 – 06/01/2008 00:00:00)
Actual: 01/01/2009 00:00:00
    E:UsersBen HallDocumentsVisual Studio 2008ProjectsIntroToXUnitIntroToXUnitClass1.cs(55,0): at IntroToXUnit.Tests.AccountTests.InRangeDateTime()

The method also has a parameter which accepts a IComparer so you can customize how the checking is done.

The Throws is another departure from traditional unit testing frameworks.  Instead of saying you expect a particular exception to be thrown from somewhere in the test case, you actually define the line of code that you expect to throw the exception – again, improving readability as we know the exact call to expect the exception to be raised from.  As an example, we are testing the following method. 

public void Withdraw(decimal amount)
{
    if(amount <= 0)
        throw new ArgumentException();

    Balance += amount;
}

Using XUnit, the test we would write would look something like below.  When we call a.Withdraw(0), we wrap the call inside a delegate and pass it as a parameter to the Assert.Throws class.  We then define the type of exception we expect to be thrown as the type for the method, in this case ArgumentException.

[Fact]
public void Withdraw_Amount0_ThrowsArgumentException()
{
    Account a = new Account();
    a.Balance = 1000;

    Assert.Throws(delegate { a.Withdraw(0);});
}

This passes successfully.  If we call a different method (one which doesn’t throw the exception).

[Fact]
public void Withdraw_Amount0_ThrowsArgumentException_Fails()
{
    Account a = new Account();
    a.Balance = 1000;

    Assert.Throws(delegate { a.WithdrawWithOutException(0); });
}

Then the test fails with the following error:

TestCase ‘IntroToXUnit.Tests.AccountTests.Withdraw_Amount0_ThrowsArgumentException_Fails’
failed: Assert.Throws() Failure
Expected: System.ArgumentException
Actual:   (No exception was thrown)
    E:UsersBen HallDocumentsVisual Studio 2008ProjectsIntroToXUnitIntroToXUnitClass1.cs(76,0): at IntroToXUnit.Tests.AccountTests.Withdraw_Amount0_ThrowsArgumentException_Fails()

The final thing I want to talk about is other common scenarios, such as Ignore and Timeout for tests.  Both Skip and Timeout are parameters on the Fact attribute. If you want a test to be skipped, then you set the Skip attribute and give a reason.

[Fact(Skip = “Takes too long”)]
public void SkippedTest()
{
    //Never executed.
}

When the test is executed, the following will be outputted.

TestCase ‘IntroToXUnit.Tests.AccountTests.SkippedTest’ not executed: Takes too long

0 passed, 0 failed, 1 skipped, took 2.18 seconds.

Timeout is set in a similar fashion, but you give it the timeout in milliseconds.  Below, we want the test to timeout after 1 seconds.

[Fact(Timeout = 1000)]
public void TimeoutTest()
{
    Thread.Sleep(5000);
}

Once the timeout is reached, it fails and the output is as follows.

TestCase ‘IntroToXUnit.Tests.AccountTests.TimeoutTest’ failed: Test execution time exceeded: 1000ms

0 passed, 1 failed, 0 skipped, took 4.15 seconds.

In summary, that has been a quick summary of XUnit and what it can offer. The framework has some really interesting concepts, but I don’t know if its ready for a full production test suite just yet.  It will be interesting to see what Brad and James does once 1.0 is released.

Technorati Tags: , ,