GUI Automation – A waste of time? (Potentially first of many posts)

Alan Page, a Microsoft Tester, recently posted about GUI Automation and I just wanted to provide my view. Alan’s main comment was:

“For 95% of all software applications, automating the GUI is a waste of time.”

Bold statement, but one I have to agree with – to a point.

Personally, I think we should automate UI’s, however, we need to be very clear and careful about what we automate and the tools we use. The most important part of UI testing is how you structure your tests and know your aim of the tests.

Generally, there are two types of automated testing tools. The first is the ‘record and playback’ approach, using tools such as QTP (HP QuickTest Professional). These tools have been around for a long time, and from my point of view has generally given automated GUI testing a bad name.

QTP records all the users interaction with the application (Web or Desktop) and scripts it out to scripting language built on top of VBScript file for playback.

Sadly the application doesn’t support sharing of steps which means you have to repeat the same actions within each test. If this same repeated action changes, you need to change delete and recreate all of your existing tests – something which is really frustrating when you are just about to release but need to change the name of a button.  This causes a lot of waste.

The other approach is more programmatic, where you program against an object model which interacts with the SUT (System Under Test). For example, with Watin, the html ‘’ turns into Ie.Link object. I’ve spoken before about WatiN and Project White, two of the talks I suggested for DeveloperDay are based on these topics.

As mentioned in the blog posts, these frameworks allow you to interact in a more domain driven way. They allow you to write the test in a reusable way using your application terminology making them much more readable and maintainable. However, from my experience its still relatively slow to start creating the tests.  Once they are created, it fine, but identitying the GUI controls and how to interact with them can take a while.

Are these tests worth writing? Or are they just a waste?

That really depends on your aim. Both approaches will get your good test coverage. The programmatic approach will produce much more maintainable tests which results in less waste.

One approach which I have recently been thinking about is having a set of core smoke tests, created using WatinWhite, focusing on core functionality of the application. These tests are just to ensure that the happy scenario of the application works, if someone picks up the latest build will they be able to use it? Or when it launches, is the first thing a user sees an unhandled exception?

For example, with Windows Live Writer, the test would create a new post, write some text, click publish and then verify the blog was posted. If that breaks, I really want to know, I want to know as soon as possible. I guess this is the 5% of UI tests.

The rest of the GUI (95% general waste) is automated using a similar approach to an APIbusiness logic. You are still automating the UI, but you are not interacting directly with the UI, this is covered by manual test cases, exploratory testing and generally making sure the application is useable – something which you can only do manually. I want to cover this in more details in a later post, but the CompositeWPF (Prism) project from Patterns and Practices is a great start.

Together, this should provide you with confidence that the application works as you expect and want in the most maintainable way.  If you can’t test the 95% using the correct patterns, then you might need to look at using WatinWhite for more than just core functionality.

As I said, this is just one approach, which has advantages and disadvantages.

UPDATE: I think I might have been wrong with my percentages, in fact ignore the percentages. There should be a bigger ratio of tests using WhiteWatin, but they should only focus on core functionality.

My main point was that these tests shouldn’t attempt to cover all possible inputs and outputs, their aim should be to give you confidence in your application and that it will work on a given platform. In order to have this confidence, you might want more tests, which is fine as long as you are structuring your tests correctly and using frameworks such as Watin.

Technorati Tags: , ,

Taking a look at NUnit 2.5 Alpha 3

image It’s always interesting to see how the different unit testing frameworks approach automated testing, each framework has it’s own style, approach and feature set with everyone having their own view on which framework everyone should use. One of the more (you could say most) popular .Net framework is NUnit and over the past few months Charlie has been working on v2.5 so I thought it was about time I took a look.

Parameterised Tests (Documentation can be found here Parameterized Tests)

Similar to XUnit and MbUnit, NUnit 2.5 includes the concept of parameterised tests. RowTest was included in 2.4.7, however additional attributes have now been included.

ValuesAttribute

The first new attribute allows for combinational style testing, this is where two sets of inputs are combined to create each possible test combination. In certain situations useful – always good to have in the toolbox, however generally I wouldn’t need to use this on a day-to-day basis. MbUnit has had this for a while, however I prefer the syntax of this approach, which I think is a similar syntax to MbUnit V3.

To implement this, you add the ValuesAttribute, which takes an object array for the test values, before the method parameter.

[Test]
public void ValuesAttributeTest([Values(1,2,3, 20)] int v, [Values(9,8,7)] int v2)
{
    Assert.IsTrue(v < v2);
}

When NUnit loads this up, they create a combination of the test values, as a result we have 12 tests created. If we had another parameter of the test, for example with 3 possible values, then we would have another possible combination and we would have 36 tests. As a result, you can very quickly build up a good selection of test cases.

image

RangeAttribute

Similar to the ValuesAttribute, however instead of you defining the values, you define the range of possible values which NUnit then creates tests for. You define the range (min, max) and NUnit does the rest. The values are then combined in a combinational style.

[Test]
public void RangeAttributeTest([Range(1,100)] int v, [Range(1,100)] int v2)
{
    Assert.IsTrue(v < v2);
}

The test creates produces then looks something like this:

image

RandomAttribute

Similar to the two previous attributes, Random produces a set of random input. You can either give it a count of the inputs to use, as I have done with the first attribute which produces 10 random values, while in the second parameter I provide the min and max range for the random values together with the amount of data to produce.

[Test]
public void RandomAttributeTest([Random(10)] int v, [Random(1, 50, 10)] int v2)
{
    Assert.IsTrue(v < v2);
}

However, I’m not sure how useful testing against random data would actually be?

TestCaseAttribute

This attribute is similar to RowTest, however as part of the TestCase you can set your own TestName which I think is actually really useful.

[TestCase(“Test1”, TestName = “This is a great!”, Description = “My first test”)]
[TestCase(“Test2”)]
[TestCase(“”)]
public void TestCaseAttributeTest(string test)
{
    Console.WriteLine(test);
    Assert.IsNotEmpty(test);
}

The TestCase is then reflected In the GUI and report. The first two tests use the test method name plus their parameter inputs, while the third uses the TestName property we set on the TestCaseAttribute.

image 

Also as part of the TestCaseAtribute, you are able to provide the Result.  Generally, the expected result is passed in as a parameter and verified within the test itself. With this approach you have the result as part of the TestCase itself, with the test returning it’s result which NUnit then compares to decide if the test has passed. This does make more sense as the parameters of the test as then really test inputs with the test output being verified as an output, a small difference however it is more logical.

[TestCase(“Test1”, TestName = “This is a great!”, Description = “My first test”, Result = 5)]
public int TestCaseAttributeTest(string test)
{
     Console.WriteLine(test);
     return test.Length;
}

This approach to testing has a lot of potential use cases and opportunities as it really increases test case (code) reuse and test maintenance.

TestCaseFactoryAttribute

Similar to MbUnit’s FactoryAttribute, you define a class which yield returns each TestCaseData object. It has the advantage of using a fluent interface to define all the different properties, such as results and the test name.  While I have hardcoded the 4 test cases here, there is nothing stopping you dynamically creating these.

[Test,Factory(typeof(MyFactoryClass))]
public int FactoryAttributeTest(int a, int b)
{
    Console.WriteLine(a / b);
    return a/b;
}

public class MyFactoryClass
{
    [TestCaseFactory(typeof(int), typeof(int))]
    public static IEnumerable TestCases
    {
        get
        {
            yield return new TestCaseData(12, 3).Returns(4);
            yield return new TestCaseData(12, 2).Returns(6);
            yield return new TestCaseData(12, 4).Returns(3);
            yield return new TestCaseData(0, 0)
                              .Throws(typeof(DivideByZeroException))
                              .WithName(“DivideByZero”)
                              .WithDescription(“An exception is expected”);
        }
    }
}

The TestCases then look like this:

image 

New Assertions

Assert.Throws

Simple, straight forward. Does the method call throw can exception?

[Test]
public void ThrowException()
{
    Assert.Throws(throwEx);
}

public void throwEx()
{
    throw new Exception(“Test”);
}

Assert.DoesNotThrow

Simpler approach, Assert that the method does not throw any exceptions when it is called.

[Test]
public void doesNotThrowExTest()
{
    Assert.DoesNotThrow(doesNotThrowEx);
}

public void doesNotThrowEx()
{}

Others

RequiredAddinAttribute

This is extremely useful if you are writing your own NUnit addins, as it is very important that the addin is available before you attempt to execute your tests. After adding attributes about the addins you require,

[assembly: RequiredAddin(“MyTestFixtureAddin”)]
[assembly: RequiredAddin(“MyTestAddin”)]
[assembly: RequiredAddin(“MyDecoratorAddin”)]

If they are not available and you execute the test, it will fail but will actually tell you the reason why.

image

PlatformAttribute

Finally, there is a new PlatformAttribute. This attribute defines which platforms the test is allowed to run on, really useful if you have issues with a certain target platform, or if you have a set of tests only targeting a platform, such as mono.

[Test]
[Platform(Exclude=”Win98,WinME”, Reason = “This only works on Windows 2000 and above”)]
public void OnlyWorksWith2000OrHigher()
{
    Assert.IsTrue(Environment.OSVersion.Version.Major > 5);
}

[Test]
[Platform(Include = “Win98,WinME”)]
public void BugOnWin98()
{
    Assert.IsTrue(Environment.OSVersion.Version.Major > 5);
}

The list of possible platforms can be found here: http://nunit.org/index.php?p=platform&r=2.5

When the test is executed on an invalid platform, the test is skipped.

image

This post has summarised all of the interesting bits of the new Alpha release. In future releases, I expect to see some changes going forward before the final 2.5 release. It’s nice to see NUnit taking the different approaches from the different frameworks and incorporating them with their own approach, hopefully v3 will have some really interesting concepts. The main feature missing at the moment are data driven test attributes, such as pulling test cases from CSV or SQL Server, but a framework has an additional extension point to make this attribute easier to write.  After this I think most of the missing features would be more suitable in NUnit 3.0.

Release Notes: http://nunit.com/index.php?p=releaseNotes&r=2.5
Download: http://nunit.com/index.php?p=download

Technorati Tags: , ,

Improving test code readability using delegates

During the development lifecycle, there are many different automated tests which should be wrote, each style of test has different priorities and different levels of maintenance required. For example, TDD helps with designing the code your just about to write, they are small and focused. As such, when your code changes, your tests change too. Where as acceptance tests have a much longer life-time, there role is to ensure the application works and nothing is regressed as developers continue to make changes. These tests need to be maintainable enough to cope with application changes, readable enough to identity the story feature they are testing with reusableflexible sections to help readability and maintainability.

In an attempt for my tests to meet these goals, recently I’ve been using delegates in certain scenarios to improve my test code. Delegates are great for reusing sections of code, it allows you to have static sections of code but be flexible to cope with minor changes and different scenarios the tests might cover.

For example, below is an acceptance test which must connect to a server based on a project file which has been saved to disk.  Remember, acceptance tests should cover the system in an end-to-end fashion.

In order to test the application, the code needs to do a lot.  It needs to create and configure a project, save it to disk, create the service and finally it gets to the point of what it is testing – connecting to the service. This code, while it is flexible it is not very maintainable. If we need to change something related to the project or how we connect to the service we would have to change a lot of code to take this into account.

public class Project_Execute_Tests_Standard
{
    [Fact]
    public void Project_Can_Login_And_Can_Connect()
    {
        string path = System.IO.Path.GetRandomFileName();

        Service service = null;
        try
        {
            Project project = new Project();
            project.Server = “Test”;
            project.Username = “Admin”;
            project.Password = “Password”;
            project.Save(path);

            service = new Service();
            service.Start();

            service.LoadProject(path);

            bool connect = service.Connect();
            Assert.True(connect);
        }
        finally
        {
            if(service != null && service.Started)
                service.Stop();
            File.Delete(path);
        }
    }
}

The first logical step would be to use helper methods.  We can extract the creation of the project and gaining access to the service into separate methods which we can reuse throughout our test code. However, these methods aren’t very flexible, if we need to add some more configuration to the project we would have to create a different method and as such losing some of our maintainability.

[Fact]
public void Project_Can_Login_And_Can_Connect()
{
    string path = string.Empty;
    Service service = null;
    try
    {
        path = CreateProject();

        service = GetService(path);
        bool connect = service.Connect();
        Assert.True(connect);
    }
    finally
    {
        if (service != null && service.Started)
            service.Stop();
        File.Delete(path);
    }
}

This is why I like delegates. You have the advantage of helper methods, but the flexibility of having the code inline. Below I’ve created a helper class, this abstracts away from my actual implementation and manages the state, such as the file path for the project. Within the delegate for the project, I’m setting all the details based on the requirements of the test and different configurations, however the rest is abstracted away.

public class Project_Execute_Tests_Delegate
{
    [Fact]
    public void Project_Can_Login_And_Can_Connect()
    {
        ServiceTester tester = new ServiceTester();
        tester.CreateProject(delegate(Project project)
                                        {
                                            project.Server = “Test”;
                                            project.Username = “Admin”;
                                            project.Password = “Password”;
                                        });

        bool connect = tester.ConnectToService();
        Assert.True(connect);
    }
}

While I was writing this example, DevExpress popped up and told me it can shorten my delegate to a lambda expression. As a result, the delegate could be this.

tester.CreateProject(p =>
                            {
                                p.Server = “Test”;
                                p.Username = “Admin”;
                                p.Password = “Password”;
                            });

Within my ServiceTester helper class, my CreateProject method looks like this:

public delegate void ProjectDelegate (Project project);
public void CreateProject(ProjectDelegate ProjectSettings)
{
     ProjectPath = System.IO.Path.GetRandomFileName();
     Project project = new Project();

     ProjectSettings(project);
     project.Save(ProjectPath);
}

Given the right scenario, I think this could really improve your test code.

Download complete code sample: http://blog.benhall.me.uk/Code/Test/TestCodeDelegates.txt

Technorati Tags: ,

Community call to action – Where are all the testers?

As many of you are aware, I currently work at Red Gate software as a Test Engineer. I have a strong interest in both code and test side and happy to discuss either at length. I joined Red Gate as a tester because I strongly believe Red Gate are doing something amazing, producing great useable software and I felt I could make a difference (for the better) as a tester. However, after six months of being a tester I have to ask the question – in the community, where are all of the testers? Developers are easy to find, they have massive conferences (PDC, TechEd) down to small user groups (NxtGenUG), I have been a member of NxtGenUG for almost 2 years since they first started in Coventry and I attended TechEd Europe but where are the testers at these types of events? Or am I just missing something?

I know recruiters have been asking this question for a while, but from a community point of view – where are all the testers? Where are the conversations happening? There must be a conversation happening about how we can improve software testing, how testers fit into the project structure and take advantage of new development technologies. Alt.Net has really promoted thinking differently about software development, but how many testers are involved in that kind of discussion? People such as Roy Osherove provides his experience and is very passionate about unit testing, but who is leading the way when it comes to functionalacceptanceintegration testing? Where are people discussing how we can take advantage of “design for testability” from the testers point of view – not just developers?

Visual Studio 10 (Rosario) is apparently very focused on testers and how the visual studio ecosystem (VSTS) can support testers equally as well as software developers, but without a good public conversation happening how is it going to happen and not just let it fall by the waste side leaving them to focus on other existing parts. In November 2005, The Register wrote about Microsoft wanting to turn testers into ‘Rock Stars’, but since then nothing has happened.  A Microsoft Tester Centre opened on MSDN in November 2007 (two years after The Register article) but its extremely light on content!  It’s not even a feature developer centre? In fact, the link is under “Servers and Enterprise Development” within the Developers Centre! How can Microsoft expect to make a major stand and push testing if they don’t even have the testing centre under the right category?

Internally, Microsoft seem to have a really good setup with Software Design Engineer in Test (SDET) who I would class as ‘Test Developers’, but at heart still testers (also in terms of this post), while some of those such as Steve Rowe are blogging and talking about testing the reach externally still doesn’t seem to be enough.

This is my call to action, If there are any testers (or Test Developers) reading this, we should talk!! Especially those in the UK, I would love to hear from you! We should have a chat about the testing community and testing in general.

Please either leave a comment or email Blog {at} BenHall {dot} me {dot} uk. On the other side – developers – what do you think?

Technorati Tags: ,

Disable System.Diagnostics.Debug.Assert dialogs

While executing my integration tests today, I was greeted with the excellent Assertion.Failed dialog which is displayed when using System.Diagnostics.  This dialog caused a problem on the build server as it was waiting for someone to click Ignore – that someone never came!

image

If you haven’t saw the Debug.Assert code before, it basically looks like this:

[Test]
public void DebugAssert()
{
System.Diagnostics.Debug.Assert(false);
}

When the Assert fails, the dialog is displayed.

The solution, add an App.Config file to your test suite.  Within the XML will clear the trace listeners, now when assert is called nothing happens because nothing is listening for it and the code continues. While executing unit tests this is fine – we just don’t want the dialog being displayed.

If you was interested in the debug.assert messages, you could have hooked up a different listener to write out the message to the console or a file.  This is what TestDriven.NET does, if an assertion is hit it writes the message to the output window.

Technorati Tags: ,

Rhino Mocks 3.5 – Goodbye Record and Playback!

Recently, Ayende released Rhino Mocks 3.5 Beta which includes a new Arrange, Act and Assert syntax style. This is a new syntax for creating your stubs and mocks. To take a deeper look into this new syntax, I’m going to migrate an old example I created for my article Beginning to Mock with Rhino Mocks and MbUnit – Part 2 which is a good read if you are new to the world of mock objects. As a quick refresher, the test I created was ensuring that the PaymentProcessor correctly communicated with the PaymentProcessingObject (in the system, this actually talked to PayPal).

[Test]
public void TakePaymentViaPaymentProcessorUsingMockService()
{
    MockRepository mocks = new MockRepository();
    IPaymentProcessing mockProxy = mocks.CreateMock();

    using (mocks.Record())
    {
        Expect.Call(mockProxy.TakePayment(1, 1, 10.0)).IgnoreArguments()
                             .Return(true);
    }

    using (mocks.Playback())
    {
        PaymentProcessor pp = new PaymentProcessor(mockProxy);
        bool result = pp.TakePayment(1, 1, 10.0);
        Assert.IsTrue(result);
    }
}

The code this tested was:

public interface IPaymentProcessing
{ bool TakePayment(int paymentId, int customerId, double amount); }

internal class PaymentProcessor
{
    internal IPaymentProcessing wsProxy;

    public PaymentProcessor(IPaymentProcessing proxy)
    {
        wsProxy = proxy;
    }

    public bool TakePayment(int paymentId, int customerId, double amount)
    {
        return wsProxy.TakePayment(paymentId, customerId, amount);
    }
}

This is a very generic example, but demonstrates Rhino Mocks using the Record (Setup the mock object) and Playback (Test and use) approach to creating mock objects. Now we have our classic test, let’s look at the changes for 3.5

CreateMock is Obsolete

The first change is that CreateMock has been marked as obsolete. Instead, we should be using StrictMock, this makes a lot more sense as it better defines the type of mock being created.

IPaymentProcessing mockProxy = mocks.StrictMock();

But this still uses the same RecordPlayback model.

AAA Syntax – Arrange, Act, Assert

This is where Rhino Mocks 3.5 is really interesting – for me at least.  We can now express our mock object is a much cleaner fashion, taking advantage of .Net 3.5 extension methods and lambda questions. Below, is the same test as above but using the Mock object.

[Test]
public void GenerateMock_TakePaymentViaPaymentProcessorUsingMockService()
{
    IPaymentProcessing mockProxy = MockRepository.GenerateMock();  #1

    mockProxy.Expect(x => x.TakePayment(1, 1, 10.0))                                                                #2
                    .Constraints(Is.Equal(1), Is.Equal(1), Is.Equal(10.0)) 
                    .Return(true);

    PaymentProcessor pp = new PaymentProcessor(mockProxy);
    bool result = pp.TakePayment(1, 1, 10.0);
    Assert.IsTrue(result);

    mockProxy.VerifyAllExpectations();                                                                                       #3
}

#1  Here we tell Rhino Mocks to create us a mock object of type IPaymentProcessing.
#2  Next we define our mock. Here, we are saying we expect TakePayment to be called, we then add some constraints about what the parameters passed in much be, finally defining that true be returned when this is called.
#3  Finally, we verify the exceptions we set in #2 where correct.

I find this new approach to be much easier to read, explain and write. The first time I tried this, I actually mistaken Constraints for Return and as such the following exception was thrown.

mockProxy.Expect(x => x.TakePayment(1, 1, 10.0)).Constraints(Is.Equal(true));  = failed: System.InvalidOperationException : The number of constraints is not the same as the number of the method’s parameters!

Just be aware of these new constraints on the parameters. 

But, not only can we use this for creating mocks, but we can also create Stubs.

[Test]
public void GenerateStub_TakePaymentViaPaymentProcessorUsingMockService()
{
    IPaymentProcessing stubProxy = MockRepository.GenerateStub();  #1
    stubProxy.Stub(action => action.TakePayment(1, 1, 10.0)).Return(true);                           #2

    PaymentProcessor pp = new PaymentProcessor(stubProxy);
    bool result = pp.TakePayment(1, 1, 10.0);
    Assert.IsTrue(result);
}

#1  Generate the stub
#2  Define the stub 

How cool is that!! Two lines of code to create our stub!

[Test]
public void GenerateStub_AssertWasCalled_TakePaymentViaPaymentProcessorUsingMockService()
{
    IPaymentProcessing stubProxy = MockRepository.GenerateStub();
    stubProxy.Stub(action => action.TakePayment(1, 1, 10.0)).Return(true);

    PaymentProcessor pp = new PaymentProcessor(stubProxy);
    bool result = pp.TakePayment(1, 1, 10.0);
    Assert.IsTrue(result);

    stubProxy.AssertWasCalled(x => x.TakePayment(1, 1, 10.00));                                                  #1
}

#1  With stubs, we can also verify that the method was called. Notice this AssertWasCalled is an extension method Rhino Mocks has added to the interface.  gain, this is helping with the readability and demonstrates an excellent use of Extension Methods.

If AssertWasCalled failed, then the following exception would be thrown and the test would fail.

Expected that IPaymentProcessing.TakePayment(1, 1, 10); would be called, but is was it was not found on the actual calls made on the mocked object  

I think the new AAA syntax is really cool. I used it to explain mock objects for my NxtGenUG Coventry session and upgrading my mock objects was a simple task and made the tests much easier to read. Really looking forward to the final release.

Download Examples Used: http://blog.benhall.me.uk/Code/RhinoMocks/RhinoMocks35Beta.zip

More examples of new syntax: http://www.ayende.com/Blog/archive/2008/05/16/Rhino-Mocks–Arrange-Act-Assert-Syntax.aspx

Download: Rhino Mocks 3.5

Technorati Tags: , , ,

Microsoft Pex – 0.5 Released

I’ve been waiting for today for too long! Peli has announced that Pex 0.5 has been released and is available to download today! In this post, I am just going to have a very quick look at the release.  Expect a lot more posts around this framework, best subscribe now so you don’t miss any.

Pex is a Microsoft research project which generates tests that cover all possible inputs. It does this by a mixture of staticdynamic analysis while taking advantage of the CLR Profiler to analyse your .Net code to find all the possible paths. Pex can then creates a series of test inputs for a test YOU wrote, you write the test which Pex then uses to find all the possible routes. You can use these tests in order to find all the different possible combinations for your application, find the areas where their are problems and have Pex suggest possible fixes.

Copied from site:

Pex comes with a Visual Studio Add-in which only works with Visual Studio 2008 Professional (or higher).
Pex also works from the
command line without any Visual Studio on the machine.

Downloads

v0.5.30516.0, 05/21/2008, download pex.30516.0.msi (x86), release notes

The Installer:

Very simple msi – just click next a few times..

image image

Taking a look at the samples

The team has spent a long term putting together some documentation and samples on how to use Pex which is great (especially for a research project). The samples are installed as part of the main framework and are in a zip file linked off the Microsoft Pex start menu item. Within the zip there is a VS2008 solution, with a project Samples.Pex which appears to contain samples for a lot of the functionality (without researching more I can’t tell what).

Within the samples, the first test I picked at random is the LuhnAlgorithmTestClass. This was of interest because I had a similar test scenario for Red Gate SQL Data Generator as that has a Credit Card Number Generator for the Luhn Algorithm.

The test itself, based on MSTEST, looks very standard. At the class level, we add a PexClass attribute and give the type of class we are testing against. The MaxRuns defines how many runs will be tried during the exploration.

[TestClass]
[PexClass(typeof(LuhnAlgorithm), MaxRuns = 100)]
public partial class LuhnAlgorithmTestClass

The test itself, we add a PexMethod and most importantly a parameter for the input to use. Pex will then use this test to generate more exploratory tests and use the parameter to inject the value to test against. 

[PexMethod]
public void CharactersAreNotValid(string input)

   string number = input + ‘a’; 
   bool result = LuhnAlgorithm.Validate(number); 
   Assert.IsFalse(result);
}

Pex includes a Visual Studio plugin, I can write click and go Pex It!

image

This should run all of my unit tests, but after executing Pex It! I got an error ClrMonitorFail (-667) – “Exit code returned when the ExtendedReflection profiler encountered a fatal error.” within the Pex Results dialog. I’ve logged this with the team, if I get a solution I will post it online.

image

Luckily, Pex comes with a console application so the post isn’t over! The command line is simple Pex with the assembly containing the Pex tests. The following command executes Pex for me:

D:UsersBen HallDesktopsamplesPexPexbinDebug>pex Samples.Pex.dll 

After executing this, we get a lot of information wrote to the console application and a nice HTML report outputted.  The console information looks like this:

Microsoft Pex v0.5.30516.0 — http://research.microsoft.com/pex
Copyright (c) Microsoft Corporation 2007-2008. All rights reserved.

instrumenting… launched Pex 0.5.30516.0 x86 Edition on .net v2.0.50727
[reports] report path: reportsSamples.Pex.80522.232227.pex
00:00:00.0> starting execution
  00:00:00.6> reflecting tests
  00:00:04.9> Samples.Pex
    00:00:05.0> LuhnAlgorithmTestClass
      00:00:06.5> CharactersAreNotValid(String)
        [test] (run 1) CharactersAreNotValidString_20080522_232253_000
        [test] (run 4) CharactersAreNotValidString_20080522_232256_001
        [test] (run 6) CharactersAreNotValidString_20080522_232256_002
        [execution] imprecision at Samples.Pex.Implementations.LuhnAlgorithm.Validate, offset 0x55
        [execution]     9 runs (88.89 % unique paths), 9/14 blocks covered
        [test] (run 10) CharactersAreNotValidString_20080522_232256_003
        [test] (run 27) CharactersAreNotValidString_20080522_232257_004
        [test] (run 28) CharactersAreNotValidString_20080522_232257_005
        [test] (run 41) CharactersAreNotValidString_20080522_232258_006
        [test] (run 45) CharactersAreNotValidString_20080522_232259_007
        [test] (run 87) CharactersAreNotValidString_20080522_232301_008
        maxruns – 100
        MaxRuns, 1 times
        [coverage] 11/14 block (78.57 %)

00:07:43.6> [finished] 269 generated tests (51 failures), 00:07:43.6011870
    — 0 critical errors, 0 errors, 4 warnings
[reports] generating reports…
[reports] html report: reportssamples.pex.80522.232227.pexpex.html 
  EXPLORATION SUCCESS

This is showing that Pex is working against the LuhnAlgorithmTestClass.CharactersAreNotValid(string input) test (as shown above), and is generating multiple different tests based on that initial test.

With the report looking like this:

image

The report contains a lot of information about the execution. Clicking the name of the test class takes you to some more information about what was executed. 

image

Clicking the parameter values link will display all of the values used for the different tests.  Each parameter sends the code down a different route.

image

Another link which interested me is the Coverage link for the LuhnAlgorithmTestClass. Clicking on this, you get a really nice Code Coverage report page.

WindowClipping (7)

Putting this a different way, by writing a single test with a parameter, Pex can create us a series of different tests to execute all of the possible routes in the method. This is great! As a developer, I can focus on creating a test which can exercise the method, but don’t have to worry about all of those edge cases as Pex will help find those.

Hello World Pex’ed

Hopefully you are still with me, now I just wanted to create a very quick Hello World application.  The class I have developed is this HelloWorld class, it has three different paths different different values for each.

public class HelloWorld
{
    public string GetHelloWorld(int ID)
    {
        if (ID == 1)
            return “Hi”;

        if (ID == 2)
            return “Hello”;

        if ((ID != 1) || (ID != 2) && (ID % 2 == 0))
            return “Hello World”;

        return string.Empty;
    }
}

The related test is this HelloWorld(int id).  Pex will input the required parameters as a argument, and then execute the test. We want to ensure Hello World is returned.

[TestFixture]
[PexClass(typeof(HelloWorld))]
public class HelloWorldTests
{
    [PexMethod]
    public void HelloWorld(int id)
    {
        HelloWorld h = new HelloWorld();
        string helloWorld = h.GetHelloWorld(id);
        Assert.AreEqual(“Hello World”, helloWorld);
    }
}

On the HTML report, it displays three different parameters which caused it to go down different routes.  For 0, it worked as expected. For 1 and 2, the test failed.

image

To prove this, I wrote a classic test which worked successfully. 

[Test]
public void HelloWorld()
{
    HelloWorld h = new HelloWorld();
    string helloWorld = h.GetHelloWorld(0);
    Assert.AreEqual(“Hello World”, helloWorld);
}

What has Pex done for us? Well, it has identified one working parameter and two parameters which will cause the test to fail.

Summary

This is a huge framework and I have only just scratched a very small surface area. I haven’t even touched the more advance parts (I will need to read the documentation for that).  I will be posting more, writing more and generally seeing what this framework is actually all about. Still not 100% convinced about this framework, I’m saving that until I see more.  Not being able to use it against a commercial application might limit my venture.

Technorati Tags: ,,,

Testing Times Ahead: Extending NUnit

An article I have wrote about Extending NUnit has just been published on the Simple Talk website. In the article I discuss how to extend the framework to support your own custom attributes, the article includes sample code about how I built attributes such as SqlServerDataSource and ExecuteSql for use when writing your test code.

You can read the article here : Testing Times Ahead- Extending NUnit

————————————————-

Simple-Talk is an online technical journal and community hub for working SQL Server and .NET developers and administrators, as well as enthusiasts and those taking their first steps with these Microsoft technologies. Simple-Talk is a growing site, currently with 150,000 subscribers, who each receive a twice-monthly newsletter providing highlights from the Simple-Talk website along with updates and announcements concerning Red Gate tools

 

Technorati Tags: , , ,

MbUnit TestSuite Attribute – Creating tests dynamically

I’ve been wanting to write about this for a while however haven’t really had the chance.  While prepping for a NxtGenUG session I came across the TestSuite attribute.  By using this, you can build up your tests to execute dynamically and provide the parameters when the tests are loaded into the runner and framework.  This allows for some great flexibility and extendibility, especially if you don’t know all of the possible test values at design-time and want values to be easily added (adding a row to the database, or dropping another file into a directory).

To take advantage of this, you need to use the TestSuiteFixtureAttribute at the top of your class.

[TestSuiteFixture]
public class TestSuiteTests

You can then create the test methods you want to be executed in order to test your system.  Two important points, they must take a parameter of type Object and return an Object – the rest is up to you. In the first test, I just write out the ToString value to the console,  in the second I check to see if the value is between 0 and 5.

public Object MethodUnderTest(object testData)
{
    Console.WriteLine(“MethodUnderTest Executed”);
    StringAssert.Contains(testData.ToString(), “Parameter”);
    return null;
}

public Object AnotherMethodUnderTest(object testData)
{
    Console.WriteLine(“AnotherTest”);
    string testDataString = testData.ToString();

    int testExecution = Convert.ToInt32(testDataString[testDataString.Length -1].ToString());
    Assert.Between(testExecution, 0, 5);

    return null;
}

The final stage is to create the method which will programmatically create all of your tests and test parameters.

The method below is marked with the TestSuiteAttribute so the framework knows this can create tests.  Inside the method, we create a new TestSuite object with a name to identify it, we can then add new items into the collection which are then returned to the framework and executed.  The parameters to add are:

  1. Name of test
  2. Method to call (the test itself) as a TestDelegate object
  3. Any parameters for the test

[TestSuite]
public ITestSuite Suite()
{
    TestSuite suite = new TestSuite(“NewSimpleSuite”);

    for (int i = 0; i < 10; i++)
    {
        suite.Add(“MethodUnderTest” + i, new TestDelegate(this.MethodUnderTest), “Parameter” + i);
        suite.Add(“AnotherMethodUnderTest” + i, new TestDelegate(this.AnotherMethodUnderTest), “Parameter” + i);
    }

    return suite;
}

The above code creates 20 tests, 10 for each test method. In the UI, the tests are all loaded correctly, as you can see each one is named correctly based on our code.

image

Very powerful attribute! But use the power wisely…

Download code sample here – http://blog.benhall.me.uk/Code/MbUnit/TestSuiteTests.cs.txt

Technorati Tags: ,,

Project White: Automated UI Testing

After using WaTiN, I have been thinking about UI Testing for WinForms, if it’s possible and if it’s even worth it. On the MbUnit mailing list I posted some syntax for an approach to WinForms and I had some good ideas, I brought up the subject again at Alt.Net.UK and while people have had success using WaTiN, they didn’t seem that interested in WinForm testing. I know others had been talking about WPF Testing during the day and problems with it.

As it happens, I read on Jeremy Miller’s blog that Thoughtworks have released ‘Project White’ which is a UI Testing framework for WPF, WinForms, Win32 and SWT (Java) and works based on Microsoft’s UIAutomation library and windows messages. Sounds promising so I decided to take a closer look, this post just discusses me playing around with the framework and a simple form to get an understanding of how it works.

Firstly, I created a standard Windows Forms application with just a single form. First test – does it display?

The form looks like this:

image

Using White and MbUnit, the test looks like this:

private const string path = @”……White_HelloWorldbinDebugWhite_HelloWorld.exe”;
#1 [Test]
public void ApplicationLaunch_NoArgs_Form1Displayed()
{
Application application = Application.Launch(path); #2
Window window = application.GetWindow(“Form1”, InitializeOption.NoCache); #3
Assert.IsNotNull(window);
Assert.IsTrue(window.DisplayState == DisplayState.Restored); #4

application.Kill(); #5
}

#1 We need to define the path to our executable. This is fine if you know your always going to be building into the same folder (both test and live assemblies), bit difficult when you have separate output directories.
#2 I then use White to execute the exe
#3 Once the application has launched, we get the form displayed as an object. This works based on the form’s title – in this case, Form1
#4 I then check the Window state to see if it has been displayed
#5 Finally, I close the application.

That’s a very basic test. Let’s add some functionality and explore the framework in more depth. What happens if the framework cannot find the form?

[Test]
[ExpectedException(typeof(Core.UIItems.UIActionException))]
public void ApplicationLaunch_NoArgs_Form2NotDisplayed()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“Form2”, InitializeOption.NoCache);
application.Kill();
}

White will attempt to find a window called Form2, if the timeout expires it throws the UIActionException. This is the same if it cannot find a control on the form.

To make this more interesting, I created an additional form with some buttons and labels.

image

The first button has a simple action, when you click it the text of the button changes to be Hello World!!. We can then create a test for this as follows:

[Test]
public void ButtonClickable_btnClick1_ChangesText()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“White Hello World”, InitializeOption.NoCache);



Button button = window.Get