SMARTest

SMARTest

SMARTest is our internally developed keyword-driven test automation framework.

Keyword Driven Approach

The recommended approach to building an automated testing framework is to create a Keyword Driven Framework.

Tests are recorded at the scenario level as high-level test steps. It is common to record these in spreadsheets as follows:

The spreadsheet provides a useful way of documenting the test design at the same time as defining the automated test steps. The values in column A denote keywords that are interpreted at runtime to execute tests.Rows without a keyword are interpreted as comments, and are frequently used to document keyword parameters as above.

Note also the use of variable references. keyword adds a customer, during which the customer is allocated a unique identifier at creation time. tag assigns the value to a variable which can then be retrieved at a later time using the &ref tag.

Note also that additional keywords are provided for alternative behaviour, e.g. the add_customer_fail keyword which is used to test invalid inputs and error handling.

For these tests to execute a keyword analyser is required to interpret each keyword. Effectively the above spreadsheets are exported to csv files and the keyword analyzer executes each keyword using its supplied arguments. keyword takes different arguments to define the initial values of the customer, such as prefix, name, and other details. The code to execute this keyword is built as application specific test code and added into the framework.

The elements of the code supplied are:

• the actions to be performed based on the arguments provided.
• the way that the correct outcome is verified (perhaps in terms of response back from the application).
• the logging of the keyword outcomes, as well as low level audit trail detail which is used in debugging.
  

Keyword code needs to be provided in a consistent structure, particular in the way arguments are passed and interpreted, and the ways in which results are logged.

Using the action words quite detailed test scenarios can be defined to entensively test functions or to follow through tests of use cases.

Designing automated tests in this way separates the test design from the implementation of the tests.Effectively the spreadsheets also serve as test design documents, hiding away navigation detail.In some instances our customers have also used this spreadsheet format fod documenting the manual test procedures as well, as testers have become familiar with the structure and find the action words as a useful way of hiding internal detail.

Automation Options

Approach	Pros	Cons
Capture/Playback tools Commercial tools are available that record windows events and responses at the client interface level.They record keystrokes, mouse events, etc, and can do bitmap comparisons of screens, compare text box contents etc.	A great deal of infrastructure is provided by the vendor, such as recorders, logging, etc. Tests what the user actually interacts with	License costs Client specific Doesn’t scale well from maintenance perspective Still need to build frameworks and libraries on top of the tool to make the testing effective.
Unit test tools A number of free tools and utilities are provided at the code level to enable developers to write unit tests.The tools are generically call XUnit tools, and the are a wide range of tools for Java (JUnit) as well as other more limited tools for other languages such as Delphi (DUnit).	Developers are comfortable using these tools as it is fundamentally writing code to test code	Too low level Difficult for non-developers to use test framework Doesn't scale well for managing large suites of tests Not focused at the system test level, focuses on testing individual classes and modules.
Data Driven Testing Using the either of the above approaches datapools are used to provide inputs and expected outcomes of tests to run	Enables large numbers of tests to be defined Test data can be defined by non-developers	Focuses on one function at a time, and doesn't cover scenario and integration tests. Not well-suited to system level tests
Keyword / Action-word Frameworks Built on top of the frameworks provided above, this focuses on building a library of test routines that high level steps of a business scenario. These can then be linked together to form test scenarios, where arguments are changed to test various behaviour.	Non-developers can build new test scenarios Scenarios are like a test design, serving a dual role of test documentation Framework reduces maintenance overheads as application changes	Time to setup initial framework Requires technical support to build and maintain library routines for each keyword

Architecture

The architecture of a keyword-driven automation framework is shown in the diagram below.

The main component of the architecture is the Keyword Interpreter, which reads information from the test scripts and configuration files. It is usually supported by other modules such as variable storage support, so that intermediate stored values can be managed, e.g. customer unique id's generated by the application, as these values may change from test run to test run.

For each keyword, transactions are constructed using components populated in the test library.Each transaction must interact with the application under test using a variety of interfaces, such as the GUI, back-end database, or application programming interfaces (APIs). The transaction may make use of underlying test utilities, built through a variety of interfaces such as Commercial Test Tools, like WinRunner or custom built test routines.

Each keyword transaction is implemented as a plug-in code, which is added to the test framework as required.This requires a common structure for calling keywords, passing parameters, writing output to the test log, and for returning results of tests. Using a test library concept for keyword transactions enables more effective test maintenance as the application changes.

When testing through the GUI interface we provide additional navigation maps that provide a mapping between logical identifiers for GUI elements to internal identifiers.This enables more effective maintenance of automated tests as the application GUI is changed.

Test log files are to be created at different levels:

• High level - a log file that matches the scenario script (at the keyword level)
• Detail level - a log file with detailed information.A single text log file with a header than contains information such as Log type, timestamp, keyword, test number, etc, can be filtered to extract specific details on a particular test step.