Alan Page, a Microsoft Tester, recently posted about GUI Automation and I just wanted to provide my view. Alan’s main comment was:
“For 95% of all software applications, automating the GUI is a waste of time.”
Bold statement, but one I have to agree with – to a point.
Personally, I think we should automate UI’s, however, we need to be very clear and careful about what we automate and the tools we use. The most important part of UI testing is how you structure your tests and know your aim of the tests.
Generally, there are two types of automated testing tools. The first is the ‘record and playback’ approach, using tools such as QTP (HP QuickTest Professional). These tools have been around for a long time, and from my point of view has generally given automated GUI testing a bad name.
QTP records all the users interaction with the application (Web or Desktop) and scripts it out to scripting language built on top of VBScript file for playback.
Sadly the application doesn’t support sharing of steps which means you have to repeat the same actions within each test. If this same repeated action changes, you need to change delete and recreate all of your existing tests – something which is really frustrating when you are just about to release but need to change the name of a button. This causes a lot of waste.
The other approach is more programmatic, where you program against an object model which interacts with the SUT (System Under Test). For example, with Watin, the html ‘’ turns into Ie.Link object. I’ve spoken before about WatiN and Project White, two of the talks I suggested for DeveloperDay are based on these topics.
As mentioned in the blog posts, these frameworks allow you to interact in a more domain driven way. They allow you to write the test in a reusable way using your application terminology making them much more readable and maintainable. However, from my experience its still relatively slow to start creating the tests. Once they are created, it fine, but identitying the GUI controls and how to interact with them can take a while.
Are these tests worth writing? Or are they just a waste?
That really depends on your aim. Both approaches will get your good test coverage. The programmatic approach will produce much more maintainable tests which results in less waste.
One approach which I have recently been thinking about is having a set of core smoke tests, created using WatinWhite, focusing on core functionality of the application. These tests are just to ensure that the happy scenario of the application works, if someone picks up the latest build will they be able to use it? Or when it launches, is the first thing a user sees an unhandled exception?
For example, with Windows Live Writer, the test would create a new post, write some text, click publish and then verify the blog was posted. If that breaks, I really want to know, I want to know as soon as possible. I guess this is the 5% of UI tests.
The rest of the GUI (95% general waste) is automated using a similar approach to an APIbusiness logic. You are still automating the UI, but you are not interacting directly with the UI, this is covered by manual test cases, exploratory testing and generally making sure the application is useable – something which you can only do manually. I want to cover this in more details in a later post, but the CompositeWPF (Prism) project from Patterns and Practices is a great start.
Together, this should provide you with confidence that the application works as you expect and want in the most maintainable way. If you can’t test the 95% using the correct patterns, then you might need to look at using WatinWhite for more than just core functionality.
As I said, this is just one approach, which has advantages and disadvantages.
UPDATE: I think I might have been wrong with my percentages, in fact ignore the percentages. There should be a bigger ratio of tests using WhiteWatin, but they should only focus on core functionality.
My main point was that these tests shouldn’t attempt to cover all possible inputs and outputs, their aim should be to give you confidence in your application and that it will work on a given platform. In order to have this confidence, you might want more tests, which is fine as long as you are structuring your tests correctly and using frameworks such as Watin.