Developing Test Scripts Using Popular Automation Frameworks

In this article, we will learn how to develop test scripts using the top 5 most popular test automation frameworks. Let’s get started.

When you learn about test automation, you come across the term “test automation framework”. Maybe some of you get uncomfortable with this term and feel that it is difficult to understand and tough to implement.

This tutorial is written to help you understand test automation frameworks as simply as possible. Read all the tutorials in this ‘Automation Testing Tutorials’ series here.

Test automation framework (in a very simple language) is a “set of rules.” Rules help us write scripts in such a manner that results in “lower maintenance”.

Developing Test Scripts

To completely understand the concept of the framework, we first have to learn how we write simple scripts and how to implement a framework on them.

In test automation, we write scripts. The scripting is basically about three ‘A’s:

1. ARRANGEMENT
2. ACTION
3. ASSERTION

Below are the details of each A, with examples

#1) ARRANGEMENT or Object Identification

We identify objects (buttons, dropdowns, etc.) either by their IDs, names or by their Window Titles, etc.

In the case of the web application, we identify by user ID, XPath, CSS, Class Name, etc. If nothing works, we then identify objects by using mouse coordinates (But it is not a reliable method of object identification)

Take this example of a Selenium WebDriver (with C#) in which we identify objects using the id. (Web application)

IWebElement txtServer = _webDriver.FindElement(By.Id("tfsServerURL"));

Another example from MS Coded UI (desktop application)

WinButton btnAdd = new WinButton(calWindow);
btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add";

After identification, we arrange or store these objects in UIMaps or Object Repository to reuse them in our scripts. That is why this step is called ARRANGEMENT.

#2) ACTION on Identified Object

When the objects are identified, we perform actions on them either by mouse or by keyboard. For example, either we click, or we double-click, or we mouse hover over it or sometimes we drag-drop. Sometimes we write on text boxes. So this second step covers any kind of action we perform on these objects.

Example #1: (Selenium WebDriver with C#)

txtServer.Clear();
txtServer.SendKeys(“Some sample text”);

Example #2: (MS Coded UI with C#)

Mouse.Click(buttonAdd);

#3) ASSERTION

The assertion is checking the object with some expected result. For example, if we press 2+3 on the calculator, the screen should show 5. In this case, our expected result is 5. We already explained this concept in our first tutorial.

Here we give an example of assertion:

Assert.AreEqual("5", txtResult.DisplayText);

Nearly every script written in test automation contains these three things: Arrangement, Action and Assertion.

Now take a look at the complete script which contains all these steps. The script will open a calculator, press 1 + 6 and then check whether the screen shows 7 or not.

Example A:

[TestMethod]
[TestMethod]
public void TestCalculator()
{
 
var app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");
//Object identification part (ARRANGEMENT)
 
//----*Calculator Window----*//
WinWindow calWindow = new WinWindow(app);
calWindow.SearchProperties[WinWindow.PropertyNames.Name] = "Calculator";
calWindow.SearchProperties[WinWindow.PropertyNames.ClassName] = "CalcFrame";
 
//----*Button1 ----*//
WinButton btn1 = new WinButton(calWindow);
btn1.SearchProperties[WinButton.PropertyNames.Name] = "1";
 
//----*Button Add ----*//
WinButton btnAdd = new WinButton(calWindow);
btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add";
 
//----*Button 6 ----*//
WinButton btn6 = new WinButton(calWindow);
btn6.SearchProperties[WinButton.PropertyNames.Name] = "6";
 
//----*Button Equals ----*//
WinButton btnEquals = new WinButton(calWindow);
btnEquals.SearchProperties[WinButton.PropertyNames.Name] = "Equals";
 
//----*Text Box Results----*//
WinText txtResult = new WinText(calWindow);
txtResult.SearchProperties[WinText.PropertyNames.Name] = "Result";
 
//(ACTIONS Part)
// Click '1' button
Mouse.Click(btn1);
 
// Click 'Add' button
Mouse.Click(btnAdd);
 
// Click '6' button
Mouse.Click(btn6);
 
// Click 'Equals' button
Mouse.Click(btnEquals);
 
//evaluate the results (ASSERTIONS)
Assert.AreEqual("7", txtResult.DisplayText, “Screen is not displaying 7);
 
//close the application
app.Close();
 
}

What’s wrong with that Script?

The script is easy to understand and I hope you get the concept of three ‘A’s in the above example. But all is not well with that script.

This script does not allow easy maintenance. Take the example of the calculator again. If we have to write test cases of each function of the calculator, there will be many test cases. If there are 10 test cases and in each test, we have to define the same object, then if any change occurs in the name or ID of the object, we have to change the object identification part in 10 test cases.

For example, take the example of button ADD in the script.

WinButton btnAdd = new WinButton(calWindow);
btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add";

Let’s say this line is used in 10 test cases. Now, in the next version of the calculator, the developer has changed the name of the button from “Add” to “Plus”. Now when we run our test cases, they will fail and we have to change the above line to this in 10 test cases.

btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Plus";

So we have to improve this test case. We should follow the famous DRY principle in our coding. DRY stands for “Do not Repeat Yourself”. We should write the object identification part in such a manner that the object should be identified only in one place and should be called everywhere.

Look at the improved script.

Example B:

//defining the objects outside the script and only once.
ApplicationUnderTest app = null;
public WinWindow calWindow
{
 
get {
WinWindow _calWindow = new WinWindow(app);
_calWindow.SearchProperties[WinWindow.PropertyNames.Name] = "Calculator";
_calWindow.SearchProperties[WinWindow.PropertyNames.ClassName] = "CalcFrame";
return _calWindow;
}
}
 
public WinText txtResult
{
get
{
WinText _txtResult = new WinText(calWindow);
_txtResult.SearchProperties[WinText.PropertyNames.Name] = "Result";
return _txtResult;
}
}
 
//making functions for similar kind of tasks
public void ClickButton(string BtnName)
{
WinButton button = new WinButton(calWindow);
button.SearchProperties[WinButton.PropertyNames.Name] = BtnName ;
Mouse.Click(button);
}
 
public void AddTwoNumbers(string number1, string number2)
{
ClickButton(number1);
ClickButton("Add");
ClickButton(number2);
ClickButton("Equals");
}
 
//Test case becomes simple and easy to maintain.
[TestMethod]
 
public void TestCalculatorModular()
{
app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");
 
//do all the operations
AddTwoNumbers("6", "1");
 
//evaluate the results

Assert.AreEqual("7", txtResult.DisplayText, “screen is not displaying 7”);
 
//close the application
app.Close();
 
}

In the above example, we have separated the calWindow and txtResult objects and moved them to the top so that they can be used across different test methods. We have defined them only once and we can use them in as many test cases as we want.

We have also created two functions. ClickButton() which accepts a button name and clicks on it and AddTwoNumbers() which takes any two numbers and adds them using the click button function inside it.

The moment we start “improving” our code and making it reusable and maintainable, it means we are making use of any automation framework. Now it’s getting interesting.

See Also => Why do we need the framework for test automation?

List of Popular Frameworks For Test Automation

Here is the popular list:

1. Linear
2. Modularity
3. Data Driven
4. Keyword Driven
5. Hybrid

We will now explain each framework with the help of its characteristics.

#1) Linear Framework

Characteristics

• Everything related to a script is defined inside the script.
• Does not care about abstraction and code duplication.
• Records and playback normally generate linear code.
• Easy to get started.
• Maintenance Nightmare.

By reading the above 5 characteristics of Linear Framework, we can easily relate our Example A to them. This example is a Linear framework. Everything related to a script is defined inside the script. The call window and TxtResult are defined inside the script. The script does not care about abstraction and code duplication. As I have explained earlier, it is also a maintenance nightmare.

So why should we use this framework?

This framework can be used in small-scale projects with few UI screens. Also, when we use any automation tool for the first time, it normally generates code in Linear Form. You can learn what code the Automation tool generates for specific actions. Apart from these reasons, avoid using this framework in your scripting.

=> See here the example of Linear and Keyword Framework with QTP example.

#2) Modularity Framework

Characteristics

• Once defined, the objects can be reused in all test methods.
• Small and to-the-point methods are created for individual functionalities.
• The test case is the collection of these small methods and reusable objects
• This allows us to write maintainable code.

By reading the above characteristics, we can relate our Example B to these characteristics. In that example, we have created an abstraction by moving the calWindow to the top and defining it inside a property that can be used everywhere.

We have created two small and independent functions called ClickButton() and AddTwoNumbers(). We combine these two small functions to create our final script, which tests the “Add” functionality of the calculator.

This results in easier maintenance. If any change occurs in the calculator UI, we have to change only in functions. Our scripts will remain intact. This framework is highly used in automation. The famous Page Object Framework (which is used with Selenium) is also a kind of modularity framework. We distribute the whole web application into separate pages.

The buttons, dropdowns, and checkboxes of each page are defined inside the class of that page. If any changes occur on the website, we will only have to change that page class and other pages will remain intact. This leads to improved maintenance and enhanced readability of scripts.

The only downside of this framework is that it requires good Object-Oriented concepts and strong development skills. If you have those, I highly recommend this framework.

#3) Data Driven Framework

Characteristics

• Test Data (input and output values) are separated from the script and stored in External Files. This could be a.CSV file, an Excel spreadsheet, or a Database.
• When the script is executed, these values are picked from external files, stored in variables, and replaced with hard-coded values, if present.
• Useful in places where the same test case has to be run with different inputs.

Example C

We want to run the add test case with three different inputs.

The data is
7 + 2 = 9
5 + 2 = 7
3 + 2 = 5

We stored this data (both input and output) in an external CSV file.

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\data.csv", "data#csv", DataAccessMethod.<strong>Sequential</strong>), DeploymentItem("TestCalc\\data.csv"), TestMethod]
 
public void TestCalculatorDataDrivsen()
{
app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");
 
//do all the operations
AddTwoNumbers(FromCSV.ADD1, FromCSV.ADD2);
 
//evaluate the results
Assert.AreEqual(FromCSV.Sum, txtResult.DisplayText);
 
//close the application
app.Close();
 
}

In the above script, we define our data source at the top of the script, which is a .csv file.

We have given the path of that.CSV file and told the script to parse it “Sequentially”. This means that the script will run as many times as there are rows present in the CSV file. In our case, the script will run 3 times. In each run, it will add the two numbers defined in the first two columns and verify that the sum of these two numbers matches the number present in the third column.

There are several advantages of this framework. All the values are stored outside the script, so if any changes occur in the next build, we just have to change the data in the external file and the script will remain intact.

The second advantage is that the same script can be run for different inputs. Take the example of an ERP in which you have to test the registration of 100 employees. You can write one script and store the names and other data related to employees in an external file. You will execute one script and it will run 100 times.

Each time has different employee data. You can easily detect what data the script cannot register the employee. It will be an added advantage when you are doing negative testing.

=> See here the example of Data driven and Hybrid framework with QTP.

#4) Keyword-Driven Framework

Characteristics

• Both data and actions are defined outside the script.
• It required the development of keywords for different types of actions.
• The functionality that we have to test is written in a step-by-step manner in tabular form using the keywords we develop and the test data. We store this table in external files just like a data-driven framework.
• The script will parse this table and perform the corresponding actions.
• Allows manual testers who do not know about coding to be part of automation to some extent.

Example D

We defined the data (e.g. 1 + 3 = 4) as well as actions (e.g. Click, Clear, etc.) in an Excel file in tabular form.

The script will be something like this (the code below is just written for understanding purposes)

[TestMethod]
public void TestCalculator()
{
app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");
 
Table tb = ReadFromExcel();
Foreach(WinRow row in tb)
{
 
WinCell Window = row.Cells[“Window”];
WinCell Control = row.Cells[“Control”];
WinCell Action = row.Cells[“Action”];
WinCell Arguments = row.Cells[“Arguments”];
UITestControl c = GetControl(Control.Text,Window.Text);
 
If(Action.Text == “Click”)
Mouse.Click (c);
 
If (Action.Text == “Clear”)
c.Clear();
 
if(Action.Text == “Verify Result”)
Assert.AreEqual(c.Text, Arguments.Text)
 
//….and so on
}
 
}

The above script is just a parser of the excel file. It parses the excel file line by line and looks for keywords to perform respective actions. If it finds the keyword “Click”, it will click on the defined object. If it finds “Verify Result”, it will perform the assertion.

There are several advantages to using the keyword-driven framework.

The first advantage is that this framework is very helpful in those scenarios where there are great chances of changes in test cases. If any step changes in a test case, we don’t need to touch the code. We just have to update the Excel file and the script will be updated.

You can define all your scripts in an excel file and hand over this excel file to manual testers to add new scripts or update the existing ones. In this way, manual testers can also become part of test automation because they do not need to code anything. They will just update this excel file when there is a need and the scripts will automatically be updated.

Another benefit is that your script is not tied to any specific tool. Scripts can be stored in an excel file, making it easy to switch automation tools by creating an excel parser.

The downside of this framework is that you need to invent keywords for various types of actions. In large-scale projects, there will be so many keywords that you need to remember and organize your scripts and keywords. This itself becomes a cumbersome task at some point.

In some complex scenarios, where objects cannot easily be identified and we need to use mouse coordinates and other techniques, this framework is not very helpful.

Keyword driven is still a favorite framework for many automation testers. Robot framework by Google is a popular keyword driven framework that is supported by an active community.

#5) Hybrid Test Automation Framework

Characteristics

• The combination of two or more of the above techniques, taking from their strengths and minimizing their weaknesses.
• The framework can use a modular approach along with either a data-driven or keyword-driven framework.
• The framework can use scripts to perform tasks that might be too difficult to implement in a purely keyword-driven approach.

Hybrid framework, in simple words, uses a combination of the above-mentioned techniques. We can use a data-driven framework which is also modular in nature. For some test cases, we can use a keyword-driven approach and for remaining we can use modular. So whenever we mix two or more techniques mentioned in this article, we are using a hybrid approach.

Conclusion

I hope that the test automation framework is no longer a scary term for you now. I tried to explain the most popular frameworks as simply as possible.

Frameworks are here to make your life easier. They help you write maintainable and reliable scripts. Without using frameworks, the test automation field is a nightmare. You need to change your code in hundreds of places for every small change in the application.

So an understanding of these frameworks is a must for every tester who wants a taste of test automation.

In our next tutorial in this series, we will learn about the “Execution and reporting of Test Automation’.

If there’s anything I missed in this article or if you have any questions, feel free to ask in the comments section below. Your thoughts are highly valued and we would love to hear them. 

PREV Tutorial #4 | NEXT Tutorial #6