Automation of desktop applications

This article presents principles and best practices related to automation of a desktop application. I will try to explain differences between desktop, web and client/server application to help you to understand influence of unique factors on automation of a desktop application. I am also going to write a few words about tools that are designed to make automation process feasible and easy.

For the purpose of this article I will focus on standalone desktop applications, that have all necessary modules available locally (on the same machine) or remote modules (e.g. database) replaced by mocks. Low-level issues such as connecting tests to CI or running automated tests on remote machine are not the subject of this article. Those topics are very broad so they will be discussed in the next article.

1. Differences between desktop and web app automation

There are three the most popular testing environments where tests are performed namely, Desktop application, Web application and Client/Server application.

Desktop application (or Standalone app) is designed to run on single work stations or PC, so when you are about to test the desktop application you need to put your attention to a specific
environment prepared for testing. It is easy to broadly automate a standalone application because of the fact that test tool has direct access to tested application.

Automation of Web application might be (and in most cases is) a bit more difficult to test, because tester must be aware of the fact that he will not have that much control over an application as in the desktop one. The application is running on the server-side which location might be physically unavailable or even unknown. Furthermore, the most important factors that could affect test results are web browser and OS.

1.1. Unique factors for each type of apps

Each kind of application has its own characteristic factors. Features that are characteristic for all of them were intentionally skipped [1].
Desktop:

• Application runs in single memory (Front end and Back end in one place),
• Single user only (operating system user account),
• In general, no resource sharing (files and databases),
• State of an application can be almost always determined,
• Access to the local file system.

Web:

• Application designed to run on two or more machines (distributed infrastructure),
• URL-driven [2],
• Disconnected mode (state less),
• Theoretically unlimited number of simultaneous users,
• Many hard-to-track issues, such as hardware compatibility, browser compatibility, version compatibility, security issues, performance issues, etc.,
• Public API allows tester to perform load testing via HTTP requests.

2. Tools

Several examples of automation tools for desktop application.

2.1. Types of available tools

The best tool does not exist. The purpose of each tool is different and test engineer should be aware of that. There are tools that work better with applications containing custom controls, other tools are more efficient with small applications and there are even tools that are quite good for beginners in automation.

2.1.1. Window handle based

Interaction between test tool and desktop application is based on handles to windows, messages and other objects. Those handles are treated more or less like a “bridge” between testing environment and test application. Therefore, it is crucial to establish proper handle to object, which belongs to test application, before conducting any action.

Pic 1. Handle based approach.

This approach is quite uncomfortable because handles have tendency to change during each session with application. Especially, when the application has many forms, modal windows or even custom (third-party) objects. As a result, it is necessary to double check if handle is still valid for each object on which any action can be performed. Otherwise, whole test case or even test suit will be vulnerable to changes, which is unacceptable.

2.1.2. Based on properties of controls

All controls that belong to tested application can be mapped in hierarchical order within the test tool. The map allows simplifying communication process with application under testing.

Pic 2. Properties based approach.

Test tool communicates with single control (or group of controls) instead of a whole window that belongs to tested application. This approach is appreciated by engineers who are working on automation, mostly because of the fact that any interaction between parts of tested application is much faster than operating on the whole window. Even complicated test cases which consist of many steps, can be executed within a couple of seconds, if they are designed properly.

Basing on controls gives us additional benefits:
1. It is possible to verify state of almost all controls (except forms which content is being changed by the user) that belong to tested application. It means that it is possible to verify application state at any time. It does not matter in which stage the test is at the moment.
2. Conducting actions on controls by test tool is based on the same approach that can be observed during normal use of tested application. For example, in order to conduct click action on a button, testing tool will send proper event to the button. The event is exactly the same as the one which occurs during manual click on the button. It gives certainty that application under testing behaves in exactly the same way as during normal usage.
3. Invisible controls are also handled without any problems because testing tool has access to properties of the whole application.
4. Changing state of a single property that belongs to a control (or a group of controls) is possible. It should be carried out very carefully because it might be dangerous – all supported values can be set, however not all of them can be set during normal usage of the application.

2.1.3. Record and playback

This kind of a tool is designed to use the simplest technique of interacting with tested application. Almost all actions are based on image, properties or component recognition to identify GUI components. This approach is very useful in cases where there is no easy access to a GUI or source code of a tested application.

Pic 3. Record & playback.

This approach has several unique advantages:
1. Test cases can be prepared by the person who has very little knowledge of software automation.
2. Every kind of scenario can be automated – not only UI of tested application, but also all actions that are related to OS, web page, network etc.
3. Implementing new test cases is quite fast process.
4. Sometimes this is the only solution that can be used when tested application consists of many non-standard controls.

Unfortunately, there are also some disadvantages that significantly influence automation process:
1. Test cases are strongly related to screenshots that have been prepared, so even the smallest change in UI may require corrections in the previously prepared test cases. It depends on percentage tolerance between images that were prepared previously and screenshots that were captured in test case execution.
2. Quality of tests is related to quality of screenshots. Low resolution screenshots, the ones that contain a lot of details or big screenshots have negative impact on quality of test.
3. Preparing tests for a different system theme requires additional screenshots that are valid for the theme.
4. Time needed for executing a test is greater than in other approaches because test tool must prepare a screenshot of tested application every time the state of the application is changed.

2.2. Most popular tools

Nowadays, there is a lot of different tools offering various features that are worth to be explored. Here are some examples that are worth presenting.

2.2.1. Sikuli

Sikuli is designed to automate almost any kind of computer operations using simplified computer vision engine. The engine recognizes areas of Graphical User Interface based on patterns from screenshots that were prepared during test creation. Sikuli uses Jython/Java/Python script languages to take actions.

[table id=3 /]

Tutorials with detailed explanations can be found under this link.

Example code:

Pic 4. Example test case.

The screenshot-based approach proves that reading this kind of „code” is much more user friendly than all other approaches, especially for the beginners.

2.2.2. AutoIT

AutoIT is a very small, self-contained automation tool that is designed mostly for automating application for Windows platform. However, nothing stands in the way to prepare scripts for routine actions, like file operations, application handling, resources monitoring, and so on.

It is based on communication with Windows (or tested application) by generating common Windows events: sending keys combination, mouse movement and click events, or even windows form/control manipulation. At this time, the tool is considered as the fastest available automation tool. It is perfect for carrying out all kinds of tests that are related to performance: performance testing, stability testing, etc.

[table id=4 /]

Example script:

#include "common_tools.au3";

If FileExists("StartupContextMenu.xml") == 1 Then
   FileDelete("StartupContextMenu.xml")
EndIf

;Actual position
$ccPoz = WinGetPos("Notepad");
If $ccPoz == 0 Then
   _Error("Notepad window not found")
   _GenerateXML("StartupWindows.xml", "StartupContextMenu", true)
EndIf

WinActivate("Notepad")

;Check if section Tools->StartUp is opened
If ControlCommand("Notepad", "", 1133 ,"isVisible") == 0 Then
   _Error("Wrong section opened")
   _GenerateXML("StartupContexMenu.xml", "StartupContextMenu", true)
EndIf

WinActivate("Notepad")

;Check if window panel is opened
Local $hWnd = WinWait("[CLASS: Notepad]", "", 10)
ControlClick($hWnd, "", "Help")
If ControlCommand("Notepad", "", "[CLASS:SysHeader32; INSTANCE:4]", "isVisible") == 0 Then
   _Error("Wrong panel opened")
   _GenerateXML("StartupContexMenu.xml", "StartupContexMenu", true)
EndIf

Pure scripting language is considered as an advantage by experienced engineers. That is why this tool is widely used also for other purposes than test automation for instance, automation of routine actions that are related to operating system (empty temp locations, set-up environment, etc.).

2.2.3. TestComplete

This tool is aimed at both professionals and beginners in software automation. It offers logical and intuitive interface, which is easily adjustable to meet individual requirements. TestComplete is all-in-one package that lets anyone start automating tests in minutes with no special skills (as compared to other tools). It does not depend on any development tool or a kind of application.

[table id=5 /]

Example code:

Pic 5. Example test case.

In the picture you can see an exemplary test case showing how easy it is to navigate through the application using TestComplete. It is possible to perform almost all actions over tested application thanks to aliases (special handles to components).

2.2.4. Microsoft UI Automation library

Microsoft UI automation is a library that provides programmatic access to most of UI elements of desktop application. Unfortunately, the library is very complex so it requires extensive knowledge about architecture of desktop application and Windows operating system.

White framework was invited in order to provide a consistent object oriented API and to hide all the complexity of Microsoft’s UI Automation library. It is written in C#, distributed as open-source. White supports all rich client applications, which are SWT (JAVA), WinForm and Win32. It is based on .NET. Any kind of proprietary scripting languages are not required so testing environment is easy to set up.

Pic 6. White Interaction with Application model.

[table id=6 /]

How does white work?

Pic 7. High level data flow diagram.

Before you decide to use White Framework in your project you should perform basic integration tests. If you intend to or have third-party components (like controls) in tested application, you should verify whether these components can be supported by the framework. For this purpose you can use UIAutomationVerify, which is available on the manufacturer’s website [4]. Whenever application contains any controls that are supported by the tool it would show you the inner details of the control in the form of a tree.

3. Summary

Automation of a desktop application is different than any other kind of automation. It is based on GUI elements (or on custom API if provided by manufacturer). In general, it is worth investing money in buying TestComplete because this tool provides everything that is necessary for desktop automation. This tool makes automation smooth and easy. If spending 4000 USD is problematic then White Framework is worth taking into account. It provides consistent structure of libraries, which allows you to carry on automation process smoothly.

4. Bibliography

[1] Source: http://www.geekinterview.com/talk/2699-difference-between-web-based-client-server.html
[2] More information available at: http://www.thenerdary.net/post/20965429580/url-driven