After using WaTiN, I have been thinking about UI Testing for WinForms, if it’s possible and if it’s even worth it. On the MbUnit mailing list I posted some syntax for an approach to WinForms and I had some good ideas, I brought up the subject again at Alt.Net.UK and while people have had success using WaTiN, they didn’t seem that interested in WinForm testing. I know others had been talking about WPF Testing during the day and problems with it.
As it happens, I read on Jeremy Miller’s blog that Thoughtworks have released ‘Project White’ which is a UI Testing framework for WPF, WinForms, Win32 and SWT (Java) and works based on Microsoft’s UIAutomation library and windows messages. Sounds promising so I decided to take a closer look, this post just discusses me playing around with the framework and a simple form to get an understanding of how it works.
Firstly, I created a standard Windows Forms application with just a single form. First test – does it display?
The form looks like this:
Using White and MbUnit, the test looks like this:
private const string path = @”……White_HelloWorldbinDebugWhite_HelloWorld.exe”;
#1 [Test]
public void ApplicationLaunch_NoArgs_Form1Displayed()
{
Application application = Application.Launch(path); #2
Window window = application.GetWindow(“Form1”, InitializeOption.NoCache); #3
Assert.IsNotNull(window);
Assert.IsTrue(window.DisplayState == DisplayState.Restored); #4
application.Kill(); #5
}
#1 We need to define the path to our executable. This is fine if you know your always going to be building into the same folder (both test and live assemblies), bit difficult when you have separate output directories.
#2 I then use White to execute the exe
#3 Once the application has launched, we get the form displayed as an object. This works based on the form’s title – in this case, Form1
#4 I then check the Window state to see if it has been displayed
#5 Finally, I close the application.
That’s a very basic test. Let’s add some functionality and explore the framework in more depth. What happens if the framework cannot find the form?
[Test]
[ExpectedException(typeof(Core.UIItems.UIActionException))]
public void ApplicationLaunch_NoArgs_Form2NotDisplayed()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“Form2”, InitializeOption.NoCache);
application.Kill();
}
White will attempt to find a window called Form2, if the timeout expires it throws the UIActionException. This is the same if it cannot find a control on the form.
To make this more interesting, I created an additional form with some buttons and labels.
The first button has a simple action, when you click it the text of the button changes to be Hello World!!. We can then create a test for this as follows:
[Test]
public void ButtonClickable_btnClick1_ChangesText()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“White Hello World”, InitializeOption.NoCache);
Button button = window.Get
button.Click(); #2
Assert.AreEqual(“Hello World!!”, button.Name); #3
}
#1 Using the Get method, we can give it the type and name of the control we want to access.
#2 We can then call the Click method which will move the mouse cursor over to the button and click it.
#3 We can then verify that the action was correctly performed, in this case the Name has changed.
Sometimes, we want to be more flexible than referring to the object by it’s name. As such, we can use the SearchCriteria object which allows us to access the control in different ways, for example by it’s text:
SearchCriteria searchCriteria = SearchCriteria.ByText(“Click Me!”);
Button button = window.Get
One problem I did encounter was with unhandled exceptions. With one of my buttons, when clicked it will cause an exception to be thrown.
The test looks like this:
[Test]
//Doesn’t work? No way to get the exception…I guess the test would fail so you would reproduce manually.
public void ClickButton_ThrowsException()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“White Hello World”, InitializeOption.NoCache);
Button button = window.Get
The test fails because of the assertion, sadly it doesn’t report back saying that an exception was thrown which I would have liked.
failed: Equal assertion failed: [[Hello World!!]]!=[[Throws Exception]]
Another problem which I wanted to see if the framework would handle was with message boxes. When clicking btnMsg, a message box is displayed on screen. Using the framework, we can use the MessageBox() method, giving it the title of the message box in order to get a Window object (would have preferred this to be a MessageBox object). We can then treat it just like a normal window which is nice.
Button button = window.Get
One last simple test I wanted to perform was how to verify that when a button is pressed, that a label is updated correctly.
[Test]
public void ClickButton_btnLabelText_ChangesLabel()
{
Application application = Application.Launch(path);
Window window = application.GetWindow(“White Hello World”, InitializeOption.NoCache);
Button button = window.Get
#1 We can access a label in the same way we access another other control, such as a button. No problems at all. There are other objects for difference controls available.
One thing about all of these tests is that they aren’t very readable, we have a lot of noise about getting access to the objects which is making it harder to read what the test is actually testing. One quick way to improve readability is to move the launch and kill calls into Setup and Teardown methods.
private Application _app = null;
[SetUp]
public void Setup()
{
_app = Application.Launch(path);
}
[TearDown]
public void Teardown()
{
_app.Kill();
}
But it can be cleaned up more, one technique I’ve discussed before with WaTiN is to create a wrapper around the UI and test against that to make our tests more readable and less fragile.
I’ve taken the code from the last test and refactored it into a wrapper. The wrapper looks like this:
public class Test2 #1
{
private Application _host = null;
private Window _main = null;
public Test2(Application host) #2
{
_host = host;
_main = _host.GetWindow(“White Hello World”, InitializeOption.NoCache);
}
public Window Main
{
get { return _main; }
}
public Button btnLabelText #3
{
get { return Main.Get
public Label lblText
{
get { return Main.Get
#1 This is the same name as the Form in the code under test for readability
#2 Pass in the application so we can gain access to the running exe
#3 This could be called anything for readability
#4 Use the same code to access the button and return it to the calling test.
Using this wrapper and the Setup/Teardown, the same test as before would be this:
[Test]
public void ClickButton_btnLabelText_ChangesLabel_ImprovedSyntax()
{
Test2 form = new Test2(_app);
form.btnLabelText.Click();
Assert.AreEqual(“Updated”, form.lblText);
}
This could be reduced more by moving the form creation into Setup as well. By using this simply step, we have made the tests more much readable. The other advantage is that if the name of a button changes, we only have to change the wrapper and not all of the dependent tests.
One final problem you might encounter is finding out what each control on the form is actually called. The .Net 3.0Windows SDK includes a tool called UISpy. This allows you to see all the properties on a running application, as such all the information required for use with White.
Another good tool is HawkEye (Updated Link).
In summary, I’m really impressed with the framework. There are a few missing features, but hopefully they can be added over time, the fact that the framework does multiple different UI technologies is great, means you don’t have to worry about what to use to write your UI tests or if your UI deals with the different technologies. Can’t wait to use it on a real project…
Keep an eye out for a post how to use this with WPF and more advanced scenarios.
Project White Homepage – http://www.codeplex.com/white
Code Sample – http://blog.benhall.me.uk/Code/ProjectWhite/White_HelloWorld.zip
I liked you idea of looking at the WinForm exceptions as exception on operation performed on a control. I have added it to my list of things.
There is something coming up in white very soon which would support screen object pattern. It is similar to the wrapper that you have mentioned here. For every
logical window in your application you can have a screen class. All the operations that you would perform on that screen would go in that class.
e.g. LoginScreen class containing userId, password, submitButton, cancelButton fields. Having a Login as a method on it.
any particular reason you haven’t tried NUnitForms for win gui testing?
i have played with it a lot:
http://weblogs.sqlteam.com/mladenp/archive/2008/01/03/Automated-GUI-testing—Is-it-worth-it.aspx
At the first glance: I think White is for desktop apps what Selenium is for web apps. It uses a higher level of abstraction when testing than NUnitForms does, that is why the same code could be used to test different things (WinForms, SWT…). I think they both have their place in testing, depending on what exactly you want to achieve.
I have looked at it before however I didn’t take to it. I like the API of White (What is the reason for xTester in NUnitforms?) and the level it aims at.
For the work I am doing, I feel White could be a better fit, however if I get chance I will revisit NUnitForms after using White on a production project and see how the two compare
Where did you get UISpy?
I can’t find it anywhere.
It’s supposed to have been issued with the .Net 3.0 SDK a long time ago, but even though I think I’ve got everything installed up to date, I can’t find UISpy.exe anywhere, not even as a downloadable…
I found UISpy difficult to find, if I remember correctly it is included in the Microsoft® Windows® Software Development Kit for Windows Vista™ and .NET Framework 3.0 Runtime Components (Windows SDK for .Net 3.0)
http://www.microsoft.com/downloads/info.aspx?na=40&p=3&SrcDisplayLang=en&SrcCategoryId=&SrcFamilyId=10cc340b-f857-4a14-83f5-25634c3bf043&u=http%3a%2f%2fgo.microsoft.com%2ffwlink%2f%3fLinkId%3d74726
There is also one for 3.5 now but I haven’t tried it.
http://www.microsoft.com/downloads/details.aspx?FamilyId=F26B1AA4-741A-433A-9BE5-FA919850BDBF&displaylang=en
Hope this helps.
Ben
You raise the question “is it worth it?” at the beginning of the post, and I’d like to elaborate on that.
Since GUI changes so much, more than API (or controller interface), the cost of maintaining the tests rises. Apart from the technical ability to build the tests and its obvious merit, well, is it worth it?
UISpy — I struggled with this the other day. The only place I could find it was by installing Microsoft Windows SDK 6.0 — it gave me an entry in the Start menu. If it doesn’t then you should be able to find it in the bin directory. I was not able to find UISpy with other versions of the SDK – 5.0, 6.0A or 6.1.
http://www.microsoft.com/downloads/details.aspx?FamilyID=4377f86d-c913-4b5c-b87e-ef72e5b4e065&displaylang=en
I couldn’t access the Hawkeye link. Any ideas?
HTTP Error 403 – Forbidden
Internet Explorer
Sorry, link was old. I’ve updated it now. Thanks
http://www.acorns.com.au/projects/hawkeye/
Re the comments about using NunitForms… I have used it for several years. The main reason I will not continue to use it seems to be a “dead” project.
The most recent release is May 2006. And, that release did not include the source. I encountered a problem or two that I wanted to debug and fix. But, without the source, that was impossible.
I also have not had any email from the NunitForms mailing list in over a year.
As far as I am concerned that’s three strikes with NUnitForms at bat…
an ou help me in getting the URL from Mozilla firefox. I’ve written something like this:
Process[] myProcess = Process.GetProcesses();
Process firefox = new Process();
AutomationElement aeBrowser;
System.Windows.Automation.Condition conLocation = new PropertyCondition(AutomationElement.NameProperty, “Location”);
foreach (Process p in myProcess)
{
if (p.ProcessName == “firefox”)
{
firefox = p;
Core.Application a = Core.Application.Attach(firefox);
SearchCriteria sc = SearchCriteria.All.AndByText(firefox.MainWindowTitle);
SearchCriteria sc2 = SearchCriteria.ByControlType(ControlType.ComboBox).AndByText(“Location”);
Window w = a.GetWindow(sc, InitializeOption.NoCache);
ComboBox cb = (ComboBox)w.Get(sc2);
break;
}
}
Hi
thanks for sharing this awesome piece of work. just tried it to automate testing of a small .Net app and it was quite refreshing 🙂
BTW, could I use this for testing Java UI? could you point me to some sample please? TIA
As I have mentioned in the documentation, you can use white to automate Java SWT/(possibly AWT, never tested) applications. It would not work for swing applications though.
Vivek
>>Another good tool is HawkEye (Updated Link).
It seems doesn’t work with WPF. I tried to inspect the menu of WPF but didn’t work.
HawkEye could well be a WinForms only tool, which is a shame.
I haven’t used it with WPF so I am unable to comment
Anyone ever tried to use White with a Silverlight 2 app? The software that I´m testing will change to this plataform soon and I need a solution for gui testing it =/
About Exceptions.
I’m struggling a little bit with that also. I’m in a project that uses white although this problem isn’t specific of white.
It seems that A Windows Forms application treats all unhandled exceptions inside the application by default if a debugger isn’t attached. So even if you surrounded all the code with a try/catch block for unhandled exceptions you wouldn’t be able to catch it. Again this is if no debugger is attached, i.e., if you run the application in Visual Studio exceptions will be caught, but if you run them normally they won’t and that dialog will appear.
The only way I found to surpass this problem is by adding this code to the main method of the application you are going to test Application.SetUnhandledExceptionMode
(UnhandledExceptionMode.ThrowException);
before Application.Run.
This really sucks because if you are testing a 3rd party application with no source code you won’t be able to add this.
Any ideas?
Have you tried Test Automation FX? (http://www.testautomationfx.com)
I would be interesting to know what you think of that!
When I tried the white it was good and nice actually, but after that i face a lot of problems to recognize the DevExpress controls, so i was wondering is it my wrong that i can’t use it in the right way or it is the abilities of the tool.
Please help…
mOHAMM, Thanks for posting this comment, if you think it’s a problem with DX controls, then drop me a mail at garysATDevexpressDOTcom
Hello,
Could you, please, tell me, what GUI(or nonGUI) Tests-Runner i must use to run White tests.
I use NUnit to run nonGUI unit test. But,unfortunately, i can’t use it to run White tests.
Thank.
Hello,
I would like to know how can we write the same code for NUnit
I was not able to get the application class in C#.
Can any one help me in this???
Maybe you could take a look at Test Automation FX as well. (www.testautomationfx.com)