quilla/docs/validation_files.md

Validation Files

Quilla operates through validation files, which are JSON files that define the environment and process for performing a validation in a declarative way. Each validation file requires the following:

• One or more target browsers (Firefox, Chrome, Edge)
• A starting URL path
• A list of steps

Each step can be either a setup step, or a validation step. Setup steps do not ordinarily provide any reports if they are successful, and Quilla will abort the test if any steps cause an error, as it will assume that the error is not recoverable. In this case, a StepFailureReport will be produced, and the ReportSummary will indicate that there is a critical failure.

Validation steps will produce a ValidationReport, which can result in either a success or a failure. Although the hope is that every test you write will result in a success, Quilla will not abort if a validation fails, and will continue the test until it is finished or otherwise aborted. However, if in attempting to perform the validation some unrecoverable error occurs, Quilla will instead produce a StepFailureReport with the exception that occurred and abort the execution.

All Quilla integration tests are written as quilla tests and therefore can be referenced as examples when writing new Quilla tests.

The Quilla docs also include a cookbook to give examples for each of the available actions.

Supported actions

The table below summarizes all the supported actions, what they do, and what they require. The Validate and OutputValue actions are omitted from this table and are shown in later sections.

Action	Description	Target	Parameters
Refresh	Refreshes the current page	None	None
NavigateBack	Navigates back to the last page	None	None
NavigateForward	Navigates forward to the next page	None	None
NavigateTo	Navigates to the target URL	URL	None
Click	Clicks an element on the page	XPath	None
Clear	Clears the text from an element on the page	XPath	None
Hover	Hovers the cursor over the target element on the page	XPath	None
SendKeys	Types the specified keys onto the target element on the page	XPath	data
WaitForExistence	Waits until a target element exists on the page	XPath	timeoutInSeconds
WaitForVisibility	Waits until a target element is visible on the page	XPath	timeoutInSeconds
SetBrowserSize	Sets the browser size to a specific width & height	None	width, height

Output Values

Quilla is able to create 'outputs', which allows users to use values from the page within their own validations. This can come in handy when you need to react to the data on the page. A classic example is an application that displays some text that it requires the user to then type into a text field: instead of hardcoding the value for the UI tests (which may be impossible if the values are dynamically generated), a Quilla test could instead read the value from the XPath and use it in the same validation.

Below is a table displaying the supported output sources, the targets they support, and the parameters they require. Every single output requires an outputName parameter.

Source	Description	Target type	Parameters
Literal	Creates an output from the literal value of the target	string	None
XPathText	Creates an output from the inner text value of the target	XPath	None
XPathProperty	Creates an output from the value of the specified property name of the target	XPath	parameterName

Validations

Each validation is performed when the "action" is set to Validate. The kind of validation is provided by the "type" key, and by default Quilla currently supports the URL and XPath types.

Each validation type supports a series of states provided by the "state" key, and requires some form of target to specify what is being validated.

XPath Validation

Each XPath validation requires a "target" key with an XPath that describes an element on the page. The state of this XPath will then be validated, according to the valid states.

Some XPath validations also require parameters, such as the Attribute and Parameter-based validations of web elements.

The table below describes what the supported states are, and what they are validating

State	Description	Parameters
Exists	The specified target exists on the page	None
NotExists	The specified target does not exist on the page	None
Visible	The specified target is visible on the page	None
NotVisible	The specified target is not visible on the page	None
TextMatches	Validates that the element text matches a regular expression pattern	pattern
NotTextMatches	Validates that the element text does not match a regular expression pattern	pattern
HasProperty	Ensures that the element has a property with a matching name	name
NotHasProperty	Ensures that the element does not have a property with a matching name	name
HasAttribute	Ensures that the element has an attribute with a matching name	name
NotHasAttribute	Ensures that the element does not have an attribute with a matching name	name
PropertyHasValue	Ensures that the property has a value matching the one specified	name, value
NotPropertyHasValue	Ensures that the property does not have a value matching the one specified	name, value
AttributeHasValue	Ensures that the attribute has a value matching the one specified	name, value
NotAttributeHasValue	Ensures that the attribute does not have a value matching the one specified	name, value
VisualParity	Checks previous baseline images pixel-by-pixel to ensure that sections have not changed	baselineID

Note: The VisualParity state is discussed more at length in the visual parity section. For information on how to write storage plugins for VisualParity to use, check out the "Storage Plugins" section of the plugins docs.

URL Validation

Each URL validation requires a "target" key with a string. The precise target will be defined by the "state" property.

State	Description
Equals	The specified target is exactly equal to the URL of the page at the time the validation runs
NotEquals	The specified target is not equal to the URL of the page at the time the validation runs
Contains	The specified target is a substring of the URL of the page at the time the validation runs
NotContains	The specified target is not a substring of the URL of the page at the time the validation runs

Examples

Searching Bing for puppies

Example written in 2021-06-16

{
    "definitions": {
        "HomePage": {
            "SearchTextBox": "//input[@id='sb_form_q']",
            "SearchButton": "//label[@for='sb_form_go']",
        },
        "ResultsPage": {
            "MainInfoCard": "//div[@class='lite-entcard-main']"
        }
    },
    "targetBrowsers": ["Firefox"],
    "path": "https://www.bing.com",
    "steps": [
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "bing"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "state": "Exists",
            "target": "${{ Definitions.HomePage.SearchTextBox }}"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "state": "Exists",
            "target": "${{ Definitions.HomePage.SearchButton }}"
        },
        {
            "action": "SendKeys",
            "target": "${{ Definitions.HomePage.SearchTextBox }}",
            "parameters": {
                "data": "Puppies"
            }
        },
        {
            "action": "Click",
            "target": "${{ Definitions.HomePage.SearchButton }}"
        },
        {
            "action": "WaitForExistence",
            "target": "${{ Definitions.ResultsPage.MainInfoCard }}",
            "parameters": {
                "timeoutInSeconds": 10
            }
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "search?q=Puppies"
        }
    ]
}

Signing In to Github

Example written on 2021-06-24

{
    "definitions": {
        "Username": "${{ Environment.GITHUB_EXAMPLE_USER_USERNAME }}",
        "Password": "${{ Environment.GITHUB_EXAMPLE_USER_PASSWORD }}",
        "WelcomePage": {
            "SignInButton": "//div[@class='position-relative mr-3']/a"
        },
        "SignInPage": {
            "UsernameInputField": "//input[@id='login_field']",
            "PasswordInputField": "//input[@id='password']",
            "SubmitButton": "//input[@class='btn btn-primary btn-block']"
        },
        "HomePage": {
            "UserMenuIcon": "//div[@class='Header-item position-relative mr-0 d-none d-md-flex']",
            "YourProfileDropdown": "//details-menu/a[@href='/${{ Definitions.Username }}']"
        }
    },
    "targetBrowsers": ["Firefox"],
    "path": "https://github.com",
    "steps": [
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "github"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "state": "Exists",
            "target": "${{ Definitions.WelcomePage.SignInButton }}"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "target": "${{ Definitions.WelcomePage.SignInButton }}",
            "state": "TextMatches",
            "parameters": {
                "pattern": "[Ss]ign[ -]?[Ii]n"
            }
        },
        {
            "action": "Click",
            "target": "${{ Definitions.WelcomePage.SignInButton }}"
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "/login"
        },
        {
            "action": "Clear",
            "target": "${{ Definitions.SignInPage.UsernameInputField }}"
        },
        {
            "action": "Clear",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}"
        },
        {
            "action": "SendKeys",
            "target":"${{ Definitions.SignInPage.UsernameInputField }}",
            "parameters": {
                "data": "${{ Definitions.Username }}"
            }
        },
        {
            "action": "SendKeys",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}",
            "parameters": {
                "data": "${{ Definitions.Password }}"
            }
        },
        {
            "action": "Validate",
            "type": "XPath",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}",
            "state": "HasAttribute",
            "parameters": {
                "name": "id"
            }
        },
        {
            "action": "Validate",
            "type": "XPath",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}",
            "state": "AttributeHasValue",
            "parameters": {
                "name": "id",
                "value": "password"
            }
        },
        {
            "action": "Click",
            "target": "${{ Definitions.SignInPage.SubmitButton }}"
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Equals",
            "target": "https://github.com/"
        },
        {
            "action": "Click",
            "target": "${{ Definitions.HomePage.UserMenuIcon }}"
        },
        {
            "action": "Click",
            "target": "${{ Definitions.HomePage.YourProfileDropdown }}"
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "${{ Definitions.Username }}"
        }
    ]
}