September 10, 2012 9:30 AM
Extending test automation to mobile apps
Mobile devices are continuing to gain momentum at increasing speed and if enterprises don’t keep up they’re at risk of losing the race. IT managers are faced with the choice to either jump on board or delay mobile adoption, but with the latter comes the danger that their organisation gets left behind.
One aspect of mobility that isn’t always discussed however is the number of unique challenges that can arise when developing and implementing mobile applications. Compared to the desktop environment, mobile applications have their own user interface (UI) requirements, business process flows, and infrastructure dependencies. Coupled with the growing variety of mobile devices and operating systems, increased strain can be put on IT to build, port and maintain mobile applications and the risk of quality and performance problems is also heightened.
Potential glitches can be solved however by putting certain mobile applications through what’s sometimes referred to as ‘functional validation’. In simple terms, functional validation is a way of testing mobile applications to ensure that they do what they are intended to do and is based on testing concepts such as test asset management, collaboration, reusability, accurate reporting and analysis.
The majority of testing on mobile devices
is currently done manually though there are some who are skeptical about the
accuracy of manual testing. The problem
with manually testing mobile applications is that it can often lack coverage
and scalability and also have considerable associated costs, especially if an
application needs to be retested multiple times to ensure development and
While a test engineer can manually key in a handful of transactions, he/she cannot continuously test and monitor the connection for application availability or test for all possible scenario permutations. Additionally, a tester is not able to quickly identify problems, fix them, and rerun all the tests required to validate the fix.
What about using mobile handsets for testing?
Many industry experts argue that using handsets for testing requires an overhead of logistics versus testing the application on a normal browser or emulator. However, in order to ensure applications function and perform, they must be tested on mobile handsets and there are several reasons ranging from the theoretical to the practical:
- Usability: ability testing on emulators and browsers with any extension do not represent what will be shown on the actual device. With Android specifically, emulated “vanilla” environments almost never represent the actual behavior of the phone as vendors tend to modify the “source code” of the handset and add their own customisation layer
- Environment related testing: this is critical as mobile applications rely on the location, accelerometer and network. Simply put, the test cannot be done in a simulated environment
- Interoperability: a mobile handset “lives” in a communicated environment and behaves accordingly. For example, an incoming phone call always receives priority over any application that is in use. Testing scenarios to see what happens to an application when a phone call is received is critical, as this often an everyday occurrence
- Network-related testing: as many mobile users travel internationally, the proper network related testing must be conducted. For instance, an application developed for Vodafone in the UK, is not guaranteed to work on the Verizon network in the US. Network neutrality has not been integrated in the cellular world, which means traffic manipulation and modification is still very common in the cellular environment
- Security: this is a common concern amongst mobile; people are sensitive to data, such as bank account numbers that remain on handsets, or if passwords are displayed on the screen. Testing for these types of security concerns in a simulated environment is not a good use of time because what is required to be tested is the behavior of the actual handset. Adding to the challenge, not all handsets have the same security designs, so each device must be individually tested
A second issue facing IT when it comes to mobile applications is the need for performance validation - applications must be validated to ensure performance expectations are met. It’s worth remembering that the user expectations from these applications may even be higher than those of their desktop counter-parts. Performance validation for mobile applications is therefore crucial and at times, more complex than for desktop applications.
In addition to this, mobile applications are affected by mobile network conditions such as bandwidth limitations that are more pronounced than land-line networks. These restrictions may adversely affect the communication between a handset and the back-end servers.
In order to reach the large number of simulated users that will be accessing the tested application at any given moment, performance validation needs to be automated, which can be achieved through simulated users running on emulators and browsers. Without this kind of automation it is virtually impossible to create a realistic load on the application servers and infrastructure.
When replicating the appropriate network conditions it needs to be ensured that a simulated mobile user is experiencing the same bandwidth throttling and network delays that a real mobile user would. In order to complete the test, real handsets executing the relevant business processes should be running parallel to the automated performance test. This will allow the end user experience to be measured while also testing the behavior of the application’s servers and infrastructure under load.
Automated functional and performance testing has a proven track-record in desktop-based applications. It helps companies deploy higher-quality software applications, reduce business risk, and accelerate problem to resolution. This will ultimately help organisations avoid costly interruptions and errors. The growing demand for instant information, especially through mobile applications, should encourage organisations to adopt automated testing strategies and solutions that are designed specifically for their mobile application needs.
Posted by Ed Hill, Applications Business Leader, HP Software UK & Ireland