Interfaces

If a testing framework provides the features above, it's possible to create robust tests. This way a click on a tree item may be emulated using a function call such as clickItem("files_Tree", "readme.txt") instead of a click to a coordinate position or unknown generic DOM element. If the Web page or the tree control changes, the tool will still locate the correct tree control and item in it to click it. Even if the internal implementation of the tree control changes, the tool can be adjusted to emulate the click correctly while the test script won't have to be changed.

Synchronization
One special challenge when creating tests for AJAX applications is dealing with the asynchronous nature of Web applications. Unlike desktop applications, where most operations are instantaneous, most operations in AJAX applications require communication with one or multiple servers. The connections to these servers may have varying bandwidths or even be unreliable.

This means a test needs to be able to synchronize with the state of the Web application. Good testing frameworks provide such synchronization mechanisms. There are several kinds of synchronizations that can be useful:

• Page loading state: A function to wait until the currently loaded page is completed
• Object availability: A function to wait until a given object is available
• Arbitrary expression: A function to evaluate an arbitrary script expression and wait until it evaluates to true. For example, to wait until an element's property has a certain value.

Using such functions it's possible to create test scripts independent of timing and network speed. Of course, the synchronization functions should also provide the possibility of specifying a timeout to make sure a test doesn't hang forever due to, say, an unexpected error.

This uncovers another feature that's very useful: Event handlers. To deal with unexpected errors such as message box popping up, many tools provide a way to handle such events and, for example, close the unexpected message box after logging its content in the test result log.

Verifications
Besides emulating user events, the main task covered by automated UI tests is the automatic verification that will lead to test passes and fails. One common mistake is the excessive use of screenshot comparisons in automated UI tests.

While screenshot comparisons look promising at first glance because they make it easy to verify that the screen looks as expected, this approach is doomed to fail once the application undergoes the slightest change, not to mention the graphical differences of the rendering engines when running tests in different Web browsers.

The approach of comparing the structure of the complete DOM tree against expected data isn't much better. Both approaches compare more than 90% irrelevant data (noise) and less than 10% of the really interesting and relevant data. This means there's a very high probability that a test fails because something irrelevant changed.

The best and most robust approach is to verify only the mentioned 10% in a page. This means a test should retrieve references to the object and controls you want to verify and query, and compare the states and properties on this objects against expected values. This will ensure that only relevant checks are made and if a test fails, it means that a real regression has been found. Unnecessary checks make tests unmaintainable and unusable in the long run. (Table 1)

As mentioned several times already, this requires a deep knowledge of the Web client framework used and the availability of a scripting language in the testing tool.

Of course, there are some scenarios where screenshot verification can be used. This is true when checking graphical views such as charts, curves, etc.

Creating Tests
To aid in creating tests, a testing framework should offer several individual tools. The most prominent is the event recorder. Using that tool a test engineer can run the Web application and execute the test scenario manually while the testing tool records all the actions and generates a test script.

To create a useful test script, the testing tool has to recognize the real widgets in the Web page and record high-level actions as discussed.

Another useful tool is a program to insert verification points. Such a tool should let the test engineer visually pick the objects and properties to verify and automatically create verification statements.

Additional tools such as a verification point editor and a test data editor come in handy too. Often all the tools and some test management facilities are combined in an IDE.

Post-Editing, Refactoring, and Data-Driven Testing
Using tools such as an event recorder have been deliberately downplayed in this article. A common beginner's mistake is to think that automated tests should be created only using an event recorder and verification point editor. While such a test script will run and deliver results quickly, it's always advisable to post-edit test scripts to make them more robust and easier to maintain.

Such post-editing tasks include factoring common actions into functions, inserting synchronization points, documenting code parts, and removing unnecessary steps.

Another methodology that can be introduced is data-driven testing. This means that the test data is moved into a separate file and the test script only contains the test logic. The test script reads the data file and executes the test logic for each test data record. This way new tests can be added without having to modify the test scripts.

As an example of data-driven testing imagine a script that reads article numbers from a text file and enters those numbers into a form of your Web application. This script is "driven" by the data that's fed to it.

Test-Automation, Distribution, and Different Environments
Once tests are created following the guidelines outlined here, the next step is to automate the test execution and reporting.

There are several things one should consider to make effective use of test automation. The following key points make up successful test automation:

• Multiple Platform and Browser Support: A great advantage to Web applications is the possibility of running them on different platforms and in different browsers. As different types of browsers exhibit different runtime behavior and bugs the tester has no choice but to verify correct behavior on each platform and browser. Needless to say this would be a drag if done manually. But as soon as a test is automated it just needs a click of the button to replay it in a different environment.

  Cross-platform and cross-browser support is an important criterion if quality assurance is taken serious. This doesn't apply only to the test execution but also to the cross-platform availability of each component that makes up the testing process.

• Command-Line Tools and Remote-Control: To automate tests successfully, the testing tool should offer command-line tools to execute them. This way it's simple to integrate the UI tests into existing frameworks and test management systems and integrate the generated results.

  It should also be possible to control all the test runs from a central machine and run several tests simultaneously on different machines, platforms, and browsers to make effective use of automated testing.

Once the test execution is automated, a procedure for following up on the test results has to be put in place. This can be a Web site displaying the results or e-mails with errors being sent to the engineers or managers responsible. The best tests and results are of no use if they aren't inspected regularly to fix regressions and any issues uncovered.

Support
One very important feature of a test automation framework is vendor support. When starting with automated testing many questions will come up and usually assistance is required. Also some adoptions in the testing tool might be necessary to make the most effective use of it.

So a crucial factor in the success or failure of any test automation effort is the support you get from the testing tool vendor. During an evaluation you should make use of the vendor's support service to find out if you get issues resolved quickly and if the vendor is open to necessary adoptions.

Choosing the Right Tools
Since the author of this article isn't completely unbiased and works for froglogic, the vendor of the popular Web testing tool Squish, this article won't attempt to recommend a specific tool. There are several Web testing tools out there. A few meet most of the requirements mentioned above, many don't. So careful evaluation is necessary. Here's a short summary of what you should look for in a Web testing tool when evaluating it:

• Cross-platform and cross-browser support without having to adjust test scripts
• Complicated setups such as special proxy servers shouldn't be necessary for easy deployment
• Support for a scripting language for test scripts (preferable an open language such as Python or JavaScript instead of a proprietary language)
• Built-in knowledge of the Web client framework used in the AUT (or a way to adopt and get the tool adopted by the Web client toolkit used)
• The possibility of freely post-editing and re-factoring tests
• Tools aiding in the creating and management of tests preferably combined in an IDE
• Command-line tools to automate tests
• Qualified and responsive vendor support service

When evaluating a tool, don't pay too much attention to components such as the event recorder. While it should work reliably, it doesn't have to be perfect. Features that help create robust and maintainable tests are more important. Don't let yourself be fooled by false promises and quick short-term results that don't result in maintainable tests.

If a vendor or tool makes one the following promises you should be very skeptical and stay away:

• The tool automatically creates tests for you or requires little effort to create tests
• Absolutely no programming skills needed
• You don't need test engineers anymore
• Delivers 100% test automation

Add Your Feedback

In order to post a comment you need to be registered and logged in.

Register | Log in

Please wait while we process your request...