DDD7 Session Video – Microsoft Pex – The future on unit testing?

As you may recall, in November I presented at DDD7 in Reading, UK about Pex and the future of unit testing.

Is unit testing about to have a major change? Pex is a project from Microsoft Research which automatically generates a traditional unit testing suite with high code coverage from hand-written parameterised unit tests. In this session, Ben explores the Pex framework, explaining the approach the framework takes and how it computes the test inputs based on your programs execution. Ben demonstrates how to use the framework and how it could potentially change the way we write unit tests.

At the time the session was recorded, with a little help from Liam (thank you!), the session is now available online. I hope you enjoy it!

Slides and code are available to download from here

DDD7 Slides and Code – Pex – The future of unit testing

Pex

Yesterday, Developer Day (DDD) 7 was held at Microsoft UK. I delivered a presentation on Pex, a new project from Microsoft Research. I provided an introduction into Pex, the problem it attempts to solve, how Pex works in the real world and finally where I think the future of unit testing and Pex is heading.

I had a great day and I was very happy with my session – the demo gods where with me. Based on tweets via twitter and other comments, it was generally well received.  Thank you to everyone who attended, when feedback opens on the website please be sure to leave your comments as they are all taken into account.

 

Code Samples: http://blog.benhall.me.uk/downloads/DDD7/Code.zip

Slides: http://blog.benhall.me.uk/downloads/DDD7/Slides.zip

DDD7 also had Channel 9 recording all the sessions, if you couldn’t attend – don’t worry, hopefully the videos will be online in the new year!

Technorati Tags: ,,

DDD7 – Microsoft Pex – The future of unit testing?

I received my email last night confirming that my Microsoft Pex session for DDD7 has been selected by you, the voting ‘potential’ attendees! I’m honoured that the session has been voted for, DDD is an amazing conference (registration is free, opening very soon) and based on the tweets from various people, there should be some great sessions.

If you would like a quick overview of Pex before DDD, then I suggest you read my MSDN Flash article.

What:
DDD 7

When:
Saturday, November 22, 2008 9:00 AM to 5:00 PM

Where:
Microsoft

Thames Valley Park
Reading, Berkshire RG6 1WG England

Technorati Tags: , ,

UK MSDN Flash – Pex: Automated testing for .NET

A quick heads up regarding a short Microsoft Pex article I wrote for the UK MSDN Flash newsletter. If you want a quick introduction, or a link to share with your colleagues then the article can be found here – 01 October 2008- Pex- Automated testing for .NET

On a side note, if you haven’t registered for the newsletter, now is a great time to do soon.  If you sign up before 28th November 2008, you will be entered into a prize draw where you could win a XBox 360 Elite (plus other prizes). Register on the website:

http://msdn.microsoft.com/en-gb/flash/default.aspx

Technorati Tags: ,

How to have Pex generate NUnit, MbUnit and XUnit tests

By default, Pex will generate MSTEST tests, however that doesn’t mean you are constrained to using MSTEST for your project. The Pex Extensions project on CodePlex has a set of extensions which allows Pex to generate code for the three main test frameworks. I’ve just committed some changes to make the code work against the newly released Pex 0.6.  In order to generate tests for a particular framework, you need to follow these instructions.

XUnit

image1) Reference Pex.xUnit.dll

2) In your test project’s AssemblyInfo.cs. Add the attribute – [assembly: Pex.Xunit.PexXunitPackage]

3) Pex will now start generating xUnit tests.

NUnit

image 1) Reference Pex.NUnit.dll

2) In your test project’s AssemblyInfo.cs. Add the attribute – [assembly: Pex.NUnit.PexNUnitPackage]

3) Pex will now start generating NUnit tests.

MbUnit v3

1) Reference Pex.MbUnit.dll

2) In your test project’s AssemblyInfo.cs. Add the attribute – [assembly: Pex.MbUnit.PexMbUnitPackage]

3) Pex will now start generating MbUnit v3 tests. A MbUnit extension is also included within the GallioMbUnit package.

Note: At the moment, you need to download the source code from codeplex and compile the binaries yourself, there is a single solution file which will build everything.

image

You also need to ensure both the extensions and your test project are built against the same version of the unit testing framework, otherwise there will be a type mismatch and Pex won’t be able to generate any tests.

Technorati Tags: , , ,

Microsoft Pex – 0.5 Released

I’ve been waiting for today for too long! Peli has announced that Pex 0.5 has been released and is available to download today! In this post, I am just going to have a very quick look at the release.  Expect a lot more posts around this framework, best subscribe now so you don’t miss any.

Pex is a Microsoft research project which generates tests that cover all possible inputs. It does this by a mixture of staticdynamic analysis while taking advantage of the CLR Profiler to analyse your .Net code to find all the possible paths. Pex can then creates a series of test inputs for a test YOU wrote, you write the test which Pex then uses to find all the possible routes. You can use these tests in order to find all the different possible combinations for your application, find the areas where their are problems and have Pex suggest possible fixes.

Copied from site:

Pex comes with a Visual Studio Add-in which only works with Visual Studio 2008 Professional (or higher).
Pex also works from the
command line without any Visual Studio on the machine.

Downloads

v0.5.30516.0, 05/21/2008, download pex.30516.0.msi (x86), release notes

The Installer:

Very simple msi – just click next a few times..

image image

Taking a look at the samples

The team has spent a long term putting together some documentation and samples on how to use Pex which is great (especially for a research project). The samples are installed as part of the main framework and are in a zip file linked off the Microsoft Pex start menu item. Within the zip there is a VS2008 solution, with a project Samples.Pex which appears to contain samples for a lot of the functionality (without researching more I can’t tell what).

Within the samples, the first test I picked at random is the LuhnAlgorithmTestClass. This was of interest because I had a similar test scenario for Red Gate SQL Data Generator as that has a Credit Card Number Generator for the Luhn Algorithm.

The test itself, based on MSTEST, looks very standard. At the class level, we add a PexClass attribute and give the type of class we are testing against. The MaxRuns defines how many runs will be tried during the exploration.

[TestClass]
[PexClass(typeof(LuhnAlgorithm), MaxRuns = 100)]
public partial class LuhnAlgorithmTestClass

The test itself, we add a PexMethod and most importantly a parameter for the input to use. Pex will then use this test to generate more exploratory tests and use the parameter to inject the value to test against. 

[PexMethod]
public void CharactersAreNotValid(string input)

   string number = input + ‘a’; 
   bool result = LuhnAlgorithm.Validate(number); 
   Assert.IsFalse(result);
}

Pex includes a Visual Studio plugin, I can write click and go Pex It!

image

This should run all of my unit tests, but after executing Pex It! I got an error ClrMonitorFail (-667) – “Exit code returned when the ExtendedReflection profiler encountered a fatal error.” within the Pex Results dialog. I’ve logged this with the team, if I get a solution I will post it online.

image

Luckily, Pex comes with a console application so the post isn’t over! The command line is simple Pex with the assembly containing the Pex tests. The following command executes Pex for me:

D:UsersBen HallDesktopsamplesPexPexbinDebug>pex Samples.Pex.dll 

After executing this, we get a lot of information wrote to the console application and a nice HTML report outputted.  The console information looks like this:

Microsoft Pex v0.5.30516.0 — http://research.microsoft.com/pex
Copyright (c) Microsoft Corporation 2007-2008. All rights reserved.

instrumenting… launched Pex 0.5.30516.0 x86 Edition on .net v2.0.50727
[reports] report path: reportsSamples.Pex.80522.232227.pex
00:00:00.0> starting execution
  00:00:00.6> reflecting tests
  00:00:04.9> Samples.Pex
    00:00:05.0> LuhnAlgorithmTestClass
      00:00:06.5> CharactersAreNotValid(String)
        [test] (run 1) CharactersAreNotValidString_20080522_232253_000
        [test] (run 4) CharactersAreNotValidString_20080522_232256_001
        [test] (run 6) CharactersAreNotValidString_20080522_232256_002
        [execution] imprecision at Samples.Pex.Implementations.LuhnAlgorithm.Validate, offset 0x55
        [execution]     9 runs (88.89 % unique paths), 9/14 blocks covered
        [test] (run 10) CharactersAreNotValidString_20080522_232256_003
        [test] (run 27) CharactersAreNotValidString_20080522_232257_004
        [test] (run 28) CharactersAreNotValidString_20080522_232257_005
        [test] (run 41) CharactersAreNotValidString_20080522_232258_006
        [test] (run 45) CharactersAreNotValidString_20080522_232259_007
        [test] (run 87) CharactersAreNotValidString_20080522_232301_008
        maxruns – 100
        MaxRuns, 1 times
        [coverage] 11/14 block (78.57 %)

00:07:43.6> [finished] 269 generated tests (51 failures), 00:07:43.6011870
    — 0 critical errors, 0 errors, 4 warnings
[reports] generating reports…
[reports] html report: reportssamples.pex.80522.232227.pexpex.html 
  EXPLORATION SUCCESS

This is showing that Pex is working against the LuhnAlgorithmTestClass.CharactersAreNotValid(string input) test (as shown above), and is generating multiple different tests based on that initial test.

With the report looking like this:

image

The report contains a lot of information about the execution. Clicking the name of the test class takes you to some more information about what was executed. 

image

Clicking the parameter values link will display all of the values used for the different tests.  Each parameter sends the code down a different route.

image

Another link which interested me is the Coverage link for the LuhnAlgorithmTestClass. Clicking on this, you get a really nice Code Coverage report page.

WindowClipping (7)

Putting this a different way, by writing a single test with a parameter, Pex can create us a series of different tests to execute all of the possible routes in the method. This is great! As a developer, I can focus on creating a test which can exercise the method, but don’t have to worry about all of those edge cases as Pex will help find those.

Hello World Pex’ed

Hopefully you are still with me, now I just wanted to create a very quick Hello World application.  The class I have developed is this HelloWorld class, it has three different paths different different values for each.

public class HelloWorld
{
    public string GetHelloWorld(int ID)
    {
        if (ID == 1)
            return “Hi”;

        if (ID == 2)
            return “Hello”;

        if ((ID != 1) || (ID != 2) && (ID % 2 == 0))
            return “Hello World”;

        return string.Empty;
    }
}

The related test is this HelloWorld(int id).  Pex will input the required parameters as a argument, and then execute the test. We want to ensure Hello World is returned.

[TestFixture]
[PexClass(typeof(HelloWorld))]
public class HelloWorldTests
{
    [PexMethod]
    public void HelloWorld(int id)
    {
        HelloWorld h = new HelloWorld();
        string helloWorld = h.GetHelloWorld(id);
        Assert.AreEqual(“Hello World”, helloWorld);
    }
}

On the HTML report, it displays three different parameters which caused it to go down different routes.  For 0, it worked as expected. For 1 and 2, the test failed.

image

To prove this, I wrote a classic test which worked successfully. 

[Test]
public void HelloWorld()
{
    HelloWorld h = new HelloWorld();
    string helloWorld = h.GetHelloWorld(0);
    Assert.AreEqual(“Hello World”, helloWorld);
}

What has Pex done for us? Well, it has identified one working parameter and two parameters which will cause the test to fail.

Summary

This is a huge framework and I have only just scratched a very small surface area. I haven’t even touched the more advance parts (I will need to read the documentation for that).  I will be posting more, writing more and generally seeing what this framework is actually all about. Still not 100% convinced about this framework, I’m saving that until I see more.  Not being able to use it against a commercial application might limit my venture.

Technorati Tags: ,,,