Editor’s Note: In an attempt to broaden our interaction with our readers we have created this Reader Forum for those with something meaningful to say to the wireless industry. We want to keep this as open as possible, but we maintain some editorial control to keep it free of commercials or attacks. Please send along submissions for this section to our editors at: dmeyer@rcrwireless.com.
It’s no secret that automating the mobile application testing process is the best way to achieve the quick, precise results needed to accommodate fast-development cycles. However, this is more challenging than it may initially seem, as mobile devices are less accessible and less open than standard desktop environments or Web-based applications.
That is why vendors of automated testing tools typically offer specialized modules or additional tools as part of their products designed for mobile applications. There also are a couple of free, open-source tools that can be used to automate mobile application tests.
We all know that automation decreases the duration of the testing/development/time-to-market cycle. On the other hand, a positive return on investment is also expected. In the case of mobile automation, the duration benefit is much more achievable and tangible than the economic effect. The cost of solution development as well as total cost of ownership is much higher when compared to the Web and even desktop.
The reason is the vast number of technical issues, which can require added effort and prove to be quite time-consuming. Below, I will describe the common issues, which should be taken into account when evaluating automation tools.
The choice of automation testing tools and approaches for testing mobile applications depends on a number of factors, such as:
Web/native application
When testing mobile versions of Web apps, QA automation engineers usually apply the same approach used for conventional Web applications. However, I would recommend WebDriver with driver versions developed for Android and iOS browsers. Moreover, bear in mind that if a Web application does not include frequent and complex Javascript calculations, and complicated and dynamic page layout and AJAX requests, the testing of the website mobile version will provide results similar to those of tests performed in the non-mobile browser.
Compliance with different platforms
In the event you need to run tests on several platforms, you will likely face some challenges from the very beginning. In this situation, you should consider a paid-for solution like T-Plan Robot, SeeTest or MonkeyTalk, to name a few. Going this route will give you the opportunity to re-use the source code and launch tests on several platforms. Even so, business logic and applications realization peculiarities for various platforms are very different. Therefore, it is likely you will need to run Android and iOS application tests through different solutions; requiring extra test development time even if code compatibility is expected.
OS versions and API
When developing automated tests, you must remember to define from the very beginning which OS is being used. For example, iOS 7.0+ versions have strict security setup conditions that won’t allow a background process – usually masked as a “GPS navigating software,” but an automation testing agent in reality) – to simulate user input on the devices. It is worth mentioning that this update has significantly influenced UI automation techniques for iOS.
Integration with third-party API is also an important aspect. For example, if an application uses Google Maps, a new version of Google API may cause changes in the map control elements layout. Some APIs can become charged; some will disappear and become incompatible with new versions. Taking these peculiarities into account when launching automated tests will allow you to avoid unnecessary struggles in the testing process.
Jail-broken or non-jail-broken for iOS
A so-called “jail-break” of iOS devices directly influences the opportunity of remote control via virtual network computing protocol. This protocol provides a picture from the device screen through an immediate client-server connection and allows the user to simultaneously send user actions to the device. The only way for VNC server realization for iOS devices today is to use the Veency app.
Control outside the tested application
The area covered by automated tests can be limited by the tested application (native app) or a browser (Web app). Sometimes there is a need to create a shortcut on the desktop or make a change in the OS settings.
This need to control things outside of the application undergoing testing limits the choice of automation tools, as some of them allow you to control only the application itself.
Objects identification
There are two applicable ways to identify objects: by object properties or by imaging. Image recognition can be accessed from a local source (if the test is performed on the emulator), or remotely (via the above-mentioned VNC protocol, AirPlay application or other proprietary technologies).
The properties of UI objects – their completeness and accuracy of the values determined – will depend upon the specific instrument being used. In other words, when testing the same application with different tools, engineers will obtain access to different sets of object properties. Object recognition may also be different in cases when you instrument your application or use native mechanisms to interact with UI.
My personal belief is that tools incorporating all the methods, image recognition and object recognitions – both instrumented and native approaches – will become the leaders in the market.
Emulators/physical devices
Few automation tools support physical devices as well as emulators. If you launch emulators while testing an iOS system, consider the requirements defined for the test environment.
Need for parallel launch
Launching tests simultaneously on several devices will require some additional environment set up – USB or Wi-Fi router setup, port forwarding, continuous integration server settings, etc. Keep in mind that performing parallel tests may not be an option if you are relying solely on a test launcher built-in automation tool. The environment is the factor that will significantly increase the total cost of ownership of an automation solution in the usage stage when compared with Web and desktop test automation practices.
Development platform for native applications
To develop a native application, it is possible to use third-party platforms that are not officially supported by the OS. For example, an application for iOS can be developed in the Adobe Air environment; then, Adobe can be used to wrap it into an IPA file and deploy it on the device. The application would look like a usual iOS app, but testing will be a bit difficult, as the object won’t be identified by standard means.
As long as you are aware of the factors influencing the choice of the test automation tools, getting the one you need is not a big deal. This is still a developing area, so there are no clear market leaders yet. Recently, I had a conversation with one of the tools’ producers who mentioned that close interaction with the client during the automation process helps adapt the tool to the particular needs of the customer. This goes to show that this sphere of mobile test automation is one where both test developers and test tool producers can really come together to create a bright future for automated testing of mobile applications.
Dmitry Tishchenko is a software testing specialist with over a decade of experience in business leadership. At A1QA, he is responsible for KPI optimization and business process automation. With his profound expertise, Tishchenko guides the clients through the implementation and working process optimization. Tishchenko is the author of several articles on test automation process, where he covers the pros and cons of testing workflow. He also holds a B.Sc. degree in software engineering.
