Introduction
Overview
Robot Framework is a keyword-driven test automation framework. Test cases live in HTML or TSV (tab-separated values) test files and make use of keywords implemented in test libraries to drive the software under test. Because Robot Framework is flexible and extensible, it is ideally suited to testing complex software with a variety of interfaces: user interfaces, command line, web services, proprietary programming interfaces, etc.
Robot Framework is open source software and installation packages, source code and further documentation is available through http://robotframework.org. This guide is designed to introduce the basic concepts of Robot Framework. For a detailed technical description, please refer to Robot Framework User Guide.
Installation
A precondition for installing Robot Framework is having Python installed. In addition, to use test libraries written in Java, Jython must be installed. To use this Quick Start Guide, Python is enough.
There are three different ways to install Robot Framework, of which the most suitable can be chosen. Detailed installation instructions can be found from the project web pages.
- There is a binary installer for Windows platform. It is enough to double-click the installer and follow instructions.
- On every platform, Robot Framework can be installed from source. To install from source, extract the source distribution and run command python setup.py install.
- If Python package management system Easy Install is available, Robot Framework can be installed by issuing command easy_install robotframework. On Windows you also need to run robot_postinstall.py script manually.
After the framework is installed, it is useful to include the directory containing start-up scripts in PATH environment variable. On UNIX-like systems this should actually happen automatically, but on Windows this must be done from Control Panel > System > Advanced > Environment Variables by adding e.g. C:\Python25\Scripts to PATH.
Successful installation can be verified with command pybot --version which should output something like:
$ pybot --version Robot Framework 2.0.3 (Python 2.5.2 on linux2)
Running this demo
This Quick Start Guide also acts as an executable demo. To run it, open a command shell, cd to the directory where this file is, and type the following command at the command line:
pybot quickstart.html
The tests in this file will execute and generate the following reports:
- report.html
- the test results summary
- log.html
- the test results details
- output.xml
- the test results in a portable XML format for integration with other tools
Open report.html (the link works only after this guide has been executed) in your browser, then click on the links to explore the results. The report.html file links to the log.html file.
There are also a number of command line options that can be used to control the test execution and generated outputs. Complete list can be viewed by issuing pybot --help. For example the following command changes the name of the log file and the name of the top level test suite:
pybot --log mylog.html --name My_Fine_Tests quickstart.html
Note
Executing this quickstart guide with jybot startup script does not work with Jython 2.2. Executing with Jython 2.5 requires setting correct executable by using -Dsys.executable option.
Introducing the sample application
The sample application for this guide is a variation on a classic login example: it is a command-line based authentication server written in Python. At the moment, the sample application allows a user to do three things:
- Create an account with a valid password.
- Log in with a valid user name and password.
- Change the password of an existing account.
The application itself is in the sut directory and can be executed with a command python sut/login.py. Attempting to log in with a non-existent user account or with an invalid password results in the same error message:
> python sut/login.py login nobody P4ssw0rd Access Denied
After creating a user account with valid password login succeeds:
> python sut/login.py create fred P4ssw0rd SUCCESS > python sut/login.py login fred P4ssw0rd Logged In
There are two requirements that a password must fulfill to be valid: it must be between 7-12 characters long, and it must contain lower and upper case letters and numbers, but it must not contain special characters. Trying to create a user with invalid password fails:
> python sut/login.py create fred short Creating user failed: Password must be 7-12 characters long > python sut/login.py create fred invalid Creating user failed: Password must be a combination of lowercase and uppercase letters and numbers
Changing password with invalid credentials results in the same error message as logging in with invalid credentials. The validity of new password is verified and if not valid, an error message is given:
> python sut/login.py change-password fred wrong NewP4ss Changing password failed: Access Denied > python sut/login.py change-password fred P4ssw0rd short Changing password failed: Password must be 7-12 characters long > python sut/login.py change-password fred P4ssw0rd NewP4ss SUCCESS
The application uses a simple database file to keep track on user statuses. The file is located in operating system dependent temporary directory.
Test cases
First test cases
Robot Framework test cases are created using a simple tabular syntax. For example the following table has two tests:
- User can create an account and log in
- User cannot log in with bad password
| Test Case | Action | Argument | Argument |
|---|---|---|---|
| User can create an account and log in | Create Valid User | fred | P4ssw0rd |
| Attempt to Login with Credentials | fred | P4ssw0rd | |
| Status Should Be | Logged In | ||
| User cannot log in with bad password | Create Valid User | betty | P4ssw0rd |
| Attempt to Login with Credentials | betty | wrong | |
| Status Should Be | Access Denied |
Notice that these tests read almost like manual tests written in English rather than like automated test cases. Robot Framework uses the keyword-driven approach that supports writing tests that capture the essence of the actions and expectations in natural language. Test cases are constructed from keywords (normally in the second column) and their possible arguments.
Higher-level test cases
Test cases can also be created using only high-level keywords that take no arguments. This style allows using totally free text which is suitable for communication even with non-technical customers or other stakeholders. Robot Framework does not enforce any particular style for writing test cases, and it is possible to use for example given-when-then format popularized by behavior-driven development (BDD) like in the example below.
| Test Case | Steps |
|---|---|
| User can change password | Given a user has a valid account |
| when she changes her password | |
| then she can log in with the new password | |
| and she cannot use the old password anymore |
This kind of use-case or user-story-like test cases are ideally suited for acceptance test-driven development (ATDD). In ATDD acceptance tests are written before implementing actual product features and they act also as requirements.
Data-driven test cases
Quite often several test cases are otherwise similar but they have slightly different input or output data. In these situations data-driven test cases, like six tests below, allow varying the test data without duplicating the workflow.
| Test Case | Action | Password | Expected error message |
|---|---|---|---|
| Too short password | Creating user with invalid password should fail | abCD5 | ${PWD INVALID LENGTH} |
| Too long password | Creating user with invalid password should fail | abCD567890123 | ${PWD INVALID LENGTH} |
| Password without lowercase letters | Creating user with invalid password should fail | 123DEFG | ${PWD INVALID CONTENT} |
| Password without capital letters | Creating user with invalid password should fail | abcd56789 | ${PWD INVALID CONTENT} |
| Password without numbers | Creating user with invalid password should fail | AbCdEfGh | ${PWD INVALID CONTENT} |
| Password with special characters | Creating user with invalid password should fail | abCD56+ | ${PWD INVALID CONTENT} |
In these tests there is only one keyword per test case, and it is responsible for trying to create a user with the provided password and checking that creation fails with an expected error message. Notice that the error messages are specified using variables.
Keywords
Test cases are created from keywords that can come from three sources: built-in keywords are always available, library keywords come from imported test libraries, and so called user keywords can be created using the same tabular syntax that is used for creating test cases.
Built-in keywords
Some generally useful keywords such as Get Time and Should Be Equal are always available. Technically these keywords come from a test library called BuiltIn and you can see its documentation for a complete list of available keywords.
Library keywords
All lowest level keywords are defined in test libraries which are implemented using standard programming languages. Robot Framework comes with a handful of libraries including an OperatingSystem library to support common operating system functions, and a Screenshot library for taking screenshots. In addition to these standard libraries, there are other libraries distributed in separate open source projects, such as SeleniumLibrary for Web testing. It is also easy to implement your own libraries when there is no suitable library available.
To be able to use keywords provided by a test library, it must be taken into use. Tests in this file need keywords from the standard OperatingSystem library (e.g. Remove File) as well as from a custom made LoginLibrary (e.g. Attempt to login with credentials). Both of these libraries are imported in so called setting table below.
| Setting | Value |
|---|---|
| Library | OperatingSystem |
| Library | testlibs/LoginLibrary.py |
User-defined keywords
One of the most powerful features of Robot Framework is the ability to easily create new higher-level keywords from other keywords. The syntax for creating these so called user-defined keywords, or user keywords for short, is similar to the syntax that is used for creating test cases. All the higher-level keywords needed in previous test cases are created in the keyword table below.
| Keyword | Action | Argument | Argument |
|---|---|---|---|
| Clear login database | Remove file | ${DATABASE FILE} | |
| Create valid user | [Arguments] | ${username} | ${password} |
| Create user | ${username} | ${password} | |
| Status should be | SUCCESS | ||
| Creating user with invalid password should fail | [Arguments] | ${password} | ${error} |
| Create user | example | ${password} | |
| Status should be | Creating user failed: ${error} | ||
| Login | [Arguments] | ${username} | ${password} |
| Attempt to login with credentials | ${username} | ${password} | |
| Status should be | Logged In | ||
| # Used by BDD test cases (this is a comment) | |||
| Given a user has a valid account | Create valid user | ${USERNAME} | ${PASSWORD} |
| When she changes her password | Change password | ${USERNAME} | ${PASSWORD} |
| ... | ${NEW PASSWORD} | ||
| Status should be | SUCCESS | ||
| Then she can log in with the new password | Login | ${USERNAME} | ${NEW PASSWORD} |
| And she cannot use the old password anymore | Attempt to login with credentials | ${USERNAME} | ${PASSWORD} |
| Status should be | Access Denied |
User-defined keywords can include actions defined by other user-defined keywords, built-in keywords, or library keywords. As you can see from this example, user-defined keywords can take parameters. They can also return values and even contain FOR loops. For now, the important thing to know is that user-defined keywords enable test creators to create reusable steps for common action sequences. User-defined keywords can also help the test author keep the tests as readable as possible and use appropriate abstraction levels in different situations.
Variables
Defining Variables
Variables are an integral part of Robot Framework. Usually any data used in tests that is subject to change is best defined as variables. Syntax for variable definition is quite simple, as seen in this table:
| Variable | Value |
|---|---|
| ${USERNAME} | janedoe |
| ${PASSWORD} | J4n3D0e |
| ${NEW PASSWORD} | e0D3n4J |
| ${DATABASE FILE} | ${TEMPDIR}${/}robotframework-quickstart-db.txt |
| ${PWD INVALID LENGTH} | Password must be 7-12 characters long |
| ${PWD INVALID CONTENT} | Password must be a combination of lowercase and uppercase letters and numbers |
Variables can also be given from the command line which is useful if the tests need to be executed in different environments. For example this demo can be executed like:
pybot --variable USERNAME:johndoe --variable PASSWORD:J0hnD0e quickstart.html
In addition to user defined variables, there are some built-in variables that are always available. These variables include ${TEMPDIR} and ${/} which are used in the above table.
Using variables
Variables can be used in most places in the test data. They are most commonly used as arguments to keywords like the following test case demonstrates. Return values from keywords can also be assigned to variables and used later. For example following Database Should Contain user keyword sets database content to ${database} variable and then verifies the content using built-in keyword Should Contain. Both library and user defined keywords can return values.
| Test Case | Action | Argument | Argument | Argument |
|---|---|---|---|---|
| User status is stored in database | [Tags] | variables | database | |
| Create Valid User | ${USERNAME} | ${PASSWORD} | ||
| Database Should Contain | ${USERNAME} | ${PASSWORD} | Inactive | |
| Login | ${USERNAME} | ${PASSWORD} | ||
| Database Should Contain | ${USERNAME} | ${PASSWORD} | Active |
| Keyword | Action | Argument | Argument | Argument |
|---|---|---|---|---|
| Database Should Contain | [Arguments] | ${username} | ${password} | ${status} |
| ${database} = | Get File | ${DATABASE FILE} | ||
| Should Contain | ${database} | ${username}\t${password}\t${status} |
Organizing test cases
Test suites
Collections of test cases are called test suites in Robot Framework. Every input file which contains test cases forms a test suite. When running this demo, you see test suite Quickstart in the console output. This name is got from the file name and it is also visible in the report and log.
It is possible to organize test cases hierarchically by placing test case files into directories and these directories into other directories. All these directories automatically create higher level test suites that get their names from directory names. Since test suites are just files and directories, they are trivially placed into any version control system.
You can test running a directory as a test suite by running following command in the directory where this guide is located:
pybot .
Setup and teardown
If you want a set of actions to occur before and after each test executes, use the Test Setup and Test Teardown settings like so:
| Setting | Value |
|---|---|
| Test Setup | Clear Login Database |
| Test Teardown |
Similarly you can use the Suite Setup and Suite Teardown settings to specify actions to be executed before and after an entire test suite executes.
Using tags
Robot Framework allows setting tags for test cases to give them free metadata. Tags can be set for all test cases in a file with Default Tags or Force Tags settings like in the table below. It is also possible to define tags for single test case like in earlier User status is stored in database test.
| Setting | Value | Value |
|---|---|---|
| Force Tags | quickstart | |
| Default Tags | example | smoke |
When you look at a report after test execution, you can see that tests have specified tags associated with them and there are also statistics generated based on tags. Tags can also be used for many other purposes, one of the most important being the possibility to select what tests to execute. You can try for example following commands:
pybot --include smoke quickstart.html pybot --exclude database quickstart.html
Creating test libraries
Robot Framework offers a simple API for creating test libraries, both with Python and Java. The user guide contains detailed description with examples.
Below is the source code of LoginLibrary test library used in this guide. You can see, for example, how the keyword Create User is mapped to actual implementation of method create_user.
import os import sys class LoginLibrary: def __init__(self): self._sut_path = os.path.join(os.path.dirname(__file__), '..', 'sut', 'login.py') self._status = '' def create_user(self, username, password): self._run_command('create', username, password) def change_password(self, username, old_pwd, new_pwd): self._run_command('change-password', username, old_pwd, new_pwd) def attempt_to_login_with_credentials(self, username, password): self._run_command('login', username, password) def status_should_be(self, expected_status): if expected_status != self._status: raise AssertionError("Expected status to be '%s' but was '%s'" % (expected_status, self._status)) def _run_command(self, command, *args): command = '"%s" %s%s' % (self._sut_path, command, ' '.join(args)) process = os.popen(command) self._status = process.read().strip() process.close()