GUI Automation – A waste of time? (Potentially first of many posts)

Alan Page, a Microsoft Tester, recently posted about GUI Automation and I just wanted to provide my view. Alan’s main comment was:

“For 95% of all software applications, automating the GUI is a waste of time.”

Bold statement, but one I have to agree with – to a point.

Personally, I think we should automate UI’s, however, we need to be very clear and careful about what we automate and the tools we use. The most important part of UI testing is how you structure your tests and know your aim of the tests.

Generally, there are two types of automated testing tools. The first is the ‘record and playback’ approach, using tools such as QTP (HP QuickTest Professional). These tools have been around for a long time, and from my point of view has generally given automated GUI testing a bad name.

QTP records all the users interaction with the application (Web or Desktop) and scripts it out to scripting language built on top of VBScript file for playback.

Sadly the application doesn’t support sharing of steps which means you have to repeat the same actions within each test. If this same repeated action changes, you need to change delete and recreate all of your existing tests – something which is really frustrating when you are just about to release but need to change the name of a button.  This causes a lot of waste.

The other approach is more programmatic, where you program against an object model which interacts with the SUT (System Under Test). For example, with Watin, the html ‘’ turns into Ie.Link object. I’ve spoken before about WatiN and Project White, two of the talks I suggested for DeveloperDay are based on these topics.

As mentioned in the blog posts, these frameworks allow you to interact in a more domain driven way. They allow you to write the test in a reusable way using your application terminology making them much more readable and maintainable. However, from my experience its still relatively slow to start creating the tests.  Once they are created, it fine, but identitying the GUI controls and how to interact with them can take a while.

Are these tests worth writing? Or are they just a waste?

That really depends on your aim. Both approaches will get your good test coverage. The programmatic approach will produce much more maintainable tests which results in less waste.

One approach which I have recently been thinking about is having a set of core smoke tests, created using WatinWhite, focusing on core functionality of the application. These tests are just to ensure that the happy scenario of the application works, if someone picks up the latest build will they be able to use it? Or when it launches, is the first thing a user sees an unhandled exception?

For example, with Windows Live Writer, the test would create a new post, write some text, click publish and then verify the blog was posted. If that breaks, I really want to know, I want to know as soon as possible. I guess this is the 5% of UI tests.

The rest of the GUI (95% general waste) is automated using a similar approach to an APIbusiness logic. You are still automating the UI, but you are not interacting directly with the UI, this is covered by manual test cases, exploratory testing and generally making sure the application is useable – something which you can only do manually. I want to cover this in more details in a later post, but the CompositeWPF (Prism) project from Patterns and Practices is a great start.

Together, this should provide you with confidence that the application works as you expect and want in the most maintainable way.  If you can’t test the 95% using the correct patterns, then you might need to look at using WatinWhite for more than just core functionality.

As I said, this is just one approach, which has advantages and disadvantages.

UPDATE: I think I might have been wrong with my percentages, in fact ignore the percentages. There should be a bigger ratio of tests using WhiteWatin, but they should only focus on core functionality.

My main point was that these tests shouldn’t attempt to cover all possible inputs and outputs, their aim should be to give you confidence in your application and that it will work on a given platform. In order to have this confidence, you might want more tests, which is fine as long as you are structuring your tests correctly and using frameworks such as Watin.

Technorati Tags: , ,

Project White: Automated UI Testing

After using WaTiN, I have been thinking about UI Testing for WinForms, if it’s possible and if it’s even worth it. On the MbUnit mailing list I posted some syntax for an approach to WinForms and I had some good ideas, I brought up the subject again at Alt.Net.UK and while people have had success using WaTiN, they didn’t seem that interested in WinForm testing. I know others had been talking about WPF Testing during the day and problems with it.

As it happens, I read on Jeremy Miller’s blog that Thoughtworks have released ‘Project White’ which is a UI Testing framework for WPF, WinForms, Win32 and SWT (Java) and works based on Microsoft’s UIAutomation library and windows messages. Sounds promising so I decided to take a closer look, this post just discusses me playing around with the framework and a simple form to get an understanding of how it works.

Firstly, I created a standard Windows Forms application with just a single form. First test – does it display?

The form looks like this:

image

Using White and MbUnit, the test looks like this:

private const string path = @”……White_HelloWorldbinDebugWhite_HelloWorld.exe”;
#1 [Test]
public void ApplicationLaunch_NoArgs_Form1Displayed()
{
Application application = Application.Launch(path); #2
Window window = application.GetWindow(“Form1”, InitializeOption.NoCache); #3
Assert.IsNotNull(window);
Assert.IsTrue(window.DisplayState == DisplayState.Restored); #4

application.Kill(); #5
}

#1 We need to define the path to our executable. This is fine if you know your always going to be building into the same folder (both test and live assemblies), bit difficult when you have separate output directories.
#2 I then use White to execute the exe
#3 Once the application has launched, we get the form displayed as an object. This works based on the form’s title – in this case, Form1
#4 I then check the Window state to see if it has been displayed
#5 Finally, I close the application.

That’s a very basic test. Let’s add some functionality and explore the framework in more depth. What happens if the framework cannot find the form?

[Test]
[ExpectedException(typeof(Core.UIItems.UIActionException))]
public void ApplicationLaunch_NoArgs_Form2NotDisplayed()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“Form2”, InitializeOption.NoCache);
application.Kill();
}

White will attempt to find a window called Form2, if the timeout expires it throws the UIActionException. This is the same if it cannot find a control on the form.

To make this more interesting, I created an additional form with some buttons and labels.

image

The first button has a simple action, when you click it the text of the button changes to be Hello World!!. We can then create a test for this as follows:

[Test]
public void ButtonClickable_btnClick1_ChangesText()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“White Hello World”, InitializeOption.NoCache);



Button button = window.Get