new: implemented exclusion XPath feature and added cookbook (#56)
This commit is contained in:
Natalia Maximo
GitHub
Родитель
b14beec304
Коммит
a2f248d190
|
|
@ -40,3 +40,10 @@ jobs:
|
|||
|
||||
- name: Run tests
|
||||
run: pytest --quilla-opts="--image-directory ./images"
|
||||
|
||||
- name: Export docs as an artifact
|
||||
uses: actions/upload-artifact@v2
|
||||
if: ${{ failure() }}
|
||||
with:
|
||||
name: image_artifacts
|
||||
path: ./images/runs
|
||||
|
|
|
|||
2
VERSION
2
VERSION
|
|
@ -1 +1 @@
|
|||
0.4
|
||||
0.5
|
||||
|
|
|
|||
|
|
@ -0,0 +1,517 @@
|
|||
# Quilla Step Cookbook
|
||||
|
||||
To make it easier to use Quilla, included below is a "cookbook" that showcases examples for every step definition that Quilla contains.
|
||||
|
||||
## Refreshing the page
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Refresh"
|
||||
}
|
||||
```
|
||||
|
||||
## Navigate Back to the Last Page
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "NavigateBack"
|
||||
}
|
||||
```
|
||||
|
||||
## Navigate Forward to the Next Page
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "NavigateForward"
|
||||
}
|
||||
```
|
||||
|
||||
## Navigate to a Page Given Its URL
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "NavigateTo",
|
||||
"target": "https://bing.com"
|
||||
}
|
||||
```
|
||||
|
||||
## Clicking on a Page Element
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Click",
|
||||
"target": "${{ Definitions.SubmitButton }}"
|
||||
}
|
||||
```
|
||||
|
||||
## Clearing an Input Box
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Clear",
|
||||
"target": "${{ Definitions.UsernameInputBox }}"
|
||||
}
|
||||
```
|
||||
|
||||
## Hovering Over an Element
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Hover",
|
||||
"target": "${{ Definitions.InfoIcon }}"
|
||||
}
|
||||
```
|
||||
|
||||
## Writing Text Into an Input
|
||||
|
||||
### Using Static Data
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "SendKeys",
|
||||
"target": "${{ Definitions.UsernameField }}",
|
||||
"parameters": {
|
||||
"data": "test_user1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Using Environment Data
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "SendKeys",
|
||||
"target": "${{ Definitions.PasswordField }}",
|
||||
"parameters": {
|
||||
"data": "${{ Environment.TEST_USER_PASSWORD }}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Waiting
|
||||
|
||||
### Wait for an Element to Exist
|
||||
|
||||
Waiting for at most 5 seconds
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "WaitForExistence",
|
||||
"target": "${{ Definitions.SubmitButton }}",
|
||||
"parameters": {
|
||||
"timeoutInSeconds": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Wait for an Element to be Visible
|
||||
|
||||
Waiting for at most 5 seconds
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "WaitForVisibility",
|
||||
"target": "${{ Definitions.HeaderBannerImage }}",
|
||||
"parameter": {
|
||||
"timeoutInSeconds": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Setting Browser Size
|
||||
|
||||
Setting the window to be 800px by 600px
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "SetBrowserSize",
|
||||
"parameters": {
|
||||
"width": 800,
|
||||
"height": 600
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Creating Outputs
|
||||
|
||||
### Creating a Literal Output
|
||||
|
||||
`Literal` outputs take the *exact* value provided by the `target` field, after any context expressions are reslved, and provide them as an output.
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "OutputValue",
|
||||
"target": "This is some text that will be output exactly",
|
||||
"parameters": {
|
||||
"source": "Literal",
|
||||
"outputName": "ExampleOutput"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
It can then be consumed through the `Validation` context object like so:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "SendKeys",
|
||||
"target": "${{ Definitions.TextBox }}",
|
||||
"parameters": {
|
||||
"data": "${{ Validation.ExampleOutput }}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `"SendKeys"` example above would evaluate to be equivalent to:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "SendKeys",
|
||||
"target": "${{ Definitions.TextBox }}",
|
||||
"parameters": {
|
||||
"data": "This is some text that will be output exactly"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Creating Output from XPath Text
|
||||
|
||||
`XPathText` outputs retrieve the inner text value of the element described by the `target` XPath.
|
||||
|
||||
Assuming that `${{ Definitions.SomeLabel }}` points to a label that has the text "This is some example text", then the following step would create the output `${{ Validation.SomeLabelText }}` with value "This is some example text".
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "OutputValue",
|
||||
"target": "${{ Definitions.SomeLabel }}",
|
||||
"parameters": {
|
||||
"source": "XPathText",
|
||||
"outputName": "SomeLabelText"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Creating output from XPath Property
|
||||
|
||||
Similarly, the `XPathProperty` source can be used to retrieve a property of an XPath
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "OutputValue",
|
||||
"target": "${{ Definitions.SomeLabel }}",
|
||||
"parameters": {
|
||||
"parameterName": "className",
|
||||
"source": "XPathProperty",
|
||||
"outputName": "SomeLabelClassName"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## XPath Validations
|
||||
|
||||
### Validating an Element Exists on the Page
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.UsernameField }}",
|
||||
"state": "Exists"
|
||||
}
|
||||
```
|
||||
|
||||
### Validating an Element Does Not Exist on the Page
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "NotExists"
|
||||
}
|
||||
```
|
||||
|
||||
### Validating an Element is Visible on the Page
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "Visible"
|
||||
}
|
||||
```
|
||||
|
||||
### Validating an Element is Not Visisble on the Page
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.HiddenInputField }}",
|
||||
"state": "NotVisible"
|
||||
}
|
||||
```
|
||||
|
||||
### Validating the XPath Text Matches a Pattern
|
||||
|
||||
Checking that it contains some text
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "TextMatches",
|
||||
"parameters": {
|
||||
"pattern": "Sign In"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Checking that it matches the text exactly
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "TextMatches",
|
||||
"parameters": {
|
||||
"pattern": "^Sign In$"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Matching a more advanced regular expression:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "TextMatches",
|
||||
"parameters": {
|
||||
"pattern": "^[Ss]ign[ -]?[Ii]n!?$"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The above example will match "Sign in", "sign in", "signin", "sign-In", "Sign In!", etc.
|
||||
|
||||
Quilla uses Python syntax regular expressions for text matching. For more information on how to write regular expressions, refer to the [regular expression HOWTO](https://docs.python.org/3/howto/regex.html)
|
||||
|
||||
### Validating that the XPath Text Does Not Match a Pattern
|
||||
|
||||
The inverse of the `TextMatches` operation. The following example will be a success if and only if the text does not include the text "Sign In" anywhere.
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "NotTextMatches",
|
||||
"parameters": {
|
||||
"pattern": "Sign In"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Has Some Property
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "HasProperty",
|
||||
"parameters": {
|
||||
"name": "className"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Does Not Have Some Property
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "NotHasProperty",
|
||||
"parameters": {
|
||||
"name": "className"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Has Some Attribute
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "HasAttribute",
|
||||
"parameters": {
|
||||
"name": "class"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Does Not Have Some Attribute
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "NotHasAttribute",
|
||||
"parameters": {
|
||||
"name": "class"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Has Some Property With a Specific Value
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "PropertyHasValue",
|
||||
"parameters": {
|
||||
"name": "className",
|
||||
"value": "mr-3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Does Not Have Some Property With a Specific Value
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "NotPropertyHasValue",
|
||||
"parameters": {
|
||||
"name": "className",
|
||||
"value": "mr-3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Has Some Attribute With a Specific Value
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "AttributeHasValue",
|
||||
"parameters": {
|
||||
"name": "class",
|
||||
"value": "mr-3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validating that the XPath Does Not Have Some Attribute With a Specific Value
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.SignInButton }}",
|
||||
"state": "NotAttributeHasValue",
|
||||
"parameters": {
|
||||
"name": "class",
|
||||
"value": "mr-3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Performing VisualParity Validation
|
||||
|
||||
Although it is a form of XPath validation, VisualParity require additional setup.
|
||||
|
||||
It is *not* required to give a specific target the same name in a Definition file/section and in the Baseline ID used for the target. However, doing so makes it easier to navigate the resulting images.
|
||||
|
||||
Performing VisualParity validation on a page element
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.HomePageContentContainer }}",
|
||||
"state": "VisualParity",
|
||||
"parameters": {
|
||||
"baselineID": "HomePageContentContainer"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Performing VisualParity validation on a page element while ignoring some sub-elements
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.HomePageContentContainer }}",
|
||||
"state": "VisualParity",
|
||||
"parameters": {
|
||||
"baselineID": "HomePageContentContainer",
|
||||
"excludeXPaths": [
|
||||
"${{ Definitions.TodaysDateLabel }}",
|
||||
"${{ Definitions.TodaysWeatherIcon }}",
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Performing URL Validations
|
||||
|
||||
### Validating URL Constains Some Text
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "URL",
|
||||
"state": "Contains",
|
||||
"target": "?q=${{ Validation.SearchQuery }}"
|
||||
}
|
||||
```
|
||||
|
||||
### Validating URL Does Not Contain Some Text
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "URL",
|
||||
"state": "NotContains",
|
||||
"target": "?q=${{ Validation.SearchQuery }}"
|
||||
}
|
||||
```
|
||||
|
||||
### Validating URL Matches Some Text Exactly
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "URL",
|
||||
"state": "Equals",
|
||||
"target": "https://www.bing.com"
|
||||
}
|
||||
```
|
||||
|
||||
### Validating URL Does Not Match Some Text Exactly
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "URL",
|
||||
"state": "NotEquals",
|
||||
"target": "http://www.bing.com"
|
||||
}
|
||||
```
|
||||
|
|
@ -15,6 +15,8 @@ Welcome to Quilla's documentation!
|
|||
usage
|
||||
install
|
||||
validation_files
|
||||
cookbook
|
||||
visual_parity
|
||||
context_expressions
|
||||
plugins
|
||||
hooks
|
||||
|
|
|
|||
|
|
@ -12,6 +12,8 @@ Validation steps will produce a `ValidationReport`, which can result in either a
|
|||
|
||||
All Quilla integration tests are written as quilla tests and therefore can be referenced as examples when writing new Quilla tests.
|
||||
|
||||
The Quilla docs also include a [cookbook](cookbook.md) to give examples for each of the available actions.
|
||||
|
||||
## Supported actions
|
||||
|
||||
The table below summarizes all the supported actions, what they do, and what they require. The `Validate` and `OutputValue` actions are omitted from this table and are shown in later sections.
|
||||
|
|
@ -71,7 +73,7 @@ The table below describes what the supported states are, and what they are valid
|
|||
| `PropertyHasValue` | Ensures that the property has a value matching the one specified | `name`, `value` |
|
||||
| `NotPropertyHasValue` | Ensures that the property does not have a value matching the one specified | `name`, `value` |
|
||||
| `AttributeHasValue` | Ensures that the attribute has a value matching the one specified | `name`, `value` |
|
||||
| `AttributeHasValue` | Ensures that the attribute does not have a value matching the one specified | `name`, `value` |
|
||||
| `NotAttributeHasValue` | Ensures that the attribute does not have a value matching the one specified | `name`, `value` |
|
||||
| `VisualParity` | Checks previous baseline images pixel-by-pixel to ensure that sections have not changed | `baselineID` |
|
||||
|
||||
> Note: The `VisualParity` state is discussed more at length in the [visual parity](visual_parity.md) section. For information on how to write storage plugins for `VisualParity` to use, check out the "Storage Plugins" section of the [plugins](plugins.md) docs.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,48 @@ Quilla includes a special `XPath` validation state called `VisualParity`. This p
|
|||
|
||||
## What is VisualParity?
|
||||
|
||||
VisualParity is a validation state that uses previously accepted screenshots generated by Quilla ("baseline images") to compare against screenshots generated when Quilla is run during testing ("treatment images") to determine if the new screenshots match the old ones. By using VisualParity with small sections of a web page, Quilla can give assurances that each section remains the same as it has been from previous runs, allowing users to update only the baseline images related to the work they are doing and protecting the rest of the page from having unexpected visual changes introduced.
|
||||
VisualParity is a validation state that uses previously accepted screenshots generated by Quilla ("baseline images") to compare against screenshots generated when Quilla is run during testing ("treatment images") to determine if the new screenshots match the old ones. By using VisualParity with sections of a web page, or even the entire page itself, Quilla can give assurances that each section remains the same as it has been from previous runs, allowing users to update only the baseline images related to the work they are doing and protecting the rest of the page from having unexpected visual changes introduced.
|
||||
|
||||
## Writing VisualParity Validations
|
||||
|
||||
A VisualParity validation is just like any other XPath validation, except that it always requires a Baseline ID to be specified. This baseline ID is assumed to be globally unique- Multiple Quilla test files can refer to the same baseline image by specifying the same ID. An example definition of a VisualParity validation is given below:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.HomePageContainer }}",
|
||||
"state": "VisualParity",
|
||||
"parameters": {
|
||||
"baselineID": "HomePageContainer"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Using Exclusion XPaths
|
||||
|
||||
Generally speaking, the higher up in the DOM hierarchy that an element is, the more likely it is that a VisualParity test will be brittle. This can be due to a page that is being changed as part of the regular development cycle, or from having dynamic components (such as time of day/weather/etc.) that will never be consistent between runs. Yet, for most of these tests, what is being tested is not that each individual component (that might have their own VisualParity validation) still looks the same, but that the general layout of the individual components is the same.
|
||||
|
||||
To prevent this fragility, Quilla allows test writers to optionally provide a list of exclusion XPaths. These DOM elements that fall below the target XPath will then be censored (i.e. their contents will be covered). This allows VisualParity validations to become more meaningful tests, as individual changes to components that do not alter the visual layout of the page will still allow the validation to pass.
|
||||
|
||||
An example definition of a VisualParity validation that specifies exclusion XPaths is given below:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"target": "${{ Definitions.HomePageContainer }}",
|
||||
"state": "VisualParity",
|
||||
"parameters": {
|
||||
"baselineID": "HomePageContainer",
|
||||
"exclusionXPaths": [
|
||||
"${{ Definitions.HomePageHeader }}",
|
||||
"${{ Definitions.HomePageFooter }}",
|
||||
"${{ Definitions.HomePageDateLabel }}"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuring a Storage Plugin
|
||||
|
||||
|
|
|
|||
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 138 KiB |
Двоичный файл не отображается.
|
До Ширина: | Высота: | Размер: 1.9 KiB |
|
|
@ -97,8 +97,9 @@ def make_parser() -> argparse.ArgumentParser: # pragma: no cover
|
|||
'--create-baseline-if-absent',
|
||||
dest='create_baseline_if_none',
|
||||
action='store_true',
|
||||
help='A flag to set that will create a new baseline image if a storage '
|
||||
'mechanism is configured but no baseline image is found with the specified ID.'
|
||||
help='A flag to have Quilla create baseline images if '
|
||||
'no baseline exists when running a VisualParity test. '
|
||||
'This is not specific to a single test and will apply '
|
||||
)
|
||||
config_group.add_argument(
|
||||
'-d',
|
||||
|
|
|
|||
|
|
@ -38,10 +38,22 @@ class BaseStep(DriverHolder, EnumResolver):
|
|||
ctx: The runtime context for the application
|
||||
action_type: Enum defining which of the supported actions this class represents
|
||||
driver: An optional argument to allow the driver to be bound at object creation.
|
||||
target: Some form of locator for what this step will target. This is passed as a
|
||||
string, but what that string represents is specific to the step that is being
|
||||
performed. This can be a URL, XPath, or something else.
|
||||
parameters: A dictionary with any other auxiliary parameter that the step might
|
||||
require. Not all steps require extra data, but a specific step type could
|
||||
have some actions that require parameters and other actions that don't.
|
||||
|
||||
Attributes:
|
||||
ctx: The runtime context for the application
|
||||
action: Enum defining which of the supported actions this class represents
|
||||
target: Some form of locator for what this step will target. This is passed as a
|
||||
string, but what that string represents is specific to the step that is being
|
||||
performed. This can be a URL, XPath, or something else.
|
||||
parameters: A dictionary with any other auxiliary parameter that the step might
|
||||
require. Not all steps require extra data, but a specific step type could
|
||||
have some actions that require parameters and other actions that don't.
|
||||
'''
|
||||
def __init__(
|
||||
self,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,308 @@
|
|||
'''
|
||||
This module contains a class definition for the VisualParityValidation class, which
|
||||
contains all the logic for the VisualParity validation state. Although VisualParity
|
||||
is not a validation type on its own (it is a subset of XPath validations), it has particular
|
||||
idiosyncracies that make more sense to have a dedicated space to handle instead of inflating
|
||||
the XPathValidation class further.
|
||||
|
||||
This module is then used by the XPathValidation class to perform the VisualParity validation
|
||||
'''
|
||||
|
||||
from io import BytesIO
|
||||
from typing import (
|
||||
Optional,
|
||||
Tuple,
|
||||
)
|
||||
from math import (
|
||||
ceil,
|
||||
floor,
|
||||
)
|
||||
|
||||
from PIL import Image
|
||||
from selenium.webdriver.remote.webelement import WebElement
|
||||
|
||||
from quilla.ctx import Context
|
||||
from quilla.reports import (
|
||||
VisualParityReport,
|
||||
ValidationReport
|
||||
)
|
||||
from quilla.common.enums import (
|
||||
VisualParityImageType,
|
||||
UITestActions,
|
||||
)
|
||||
from quilla.common.exceptions import FailedStepException
|
||||
|
||||
from quilla.steps.base_steps import BaseStep
|
||||
|
||||
|
||||
class VisualParityState(BaseStep):
|
||||
'''
|
||||
Helper class to logically group methods and helper functions for VisualParity
|
||||
validation state. It inherits from BaseStep as BaseStep contains a lot of useful
|
||||
methods and properties.
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
ctx: Context,
|
||||
target: str,
|
||||
parameters: dict,
|
||||
):
|
||||
super().__init__(
|
||||
ctx=ctx,
|
||||
action_type=UITestActions.VALIDATE,
|
||||
target=target,
|
||||
parameters=parameters
|
||||
)
|
||||
|
||||
self._verify_parameters('baselineID')
|
||||
self.baseline_id = parameters['baselineID']
|
||||
self._driver = ctx.driver
|
||||
|
||||
def copy(self):
|
||||
return VisualParityState(self.ctx, self._target, self._parameters)
|
||||
|
||||
@property
|
||||
def hook(self):
|
||||
'''
|
||||
Pass-through property to make it easier to call hooks
|
||||
'''
|
||||
return self.ctx.pm.hook
|
||||
|
||||
def _create_report(
|
||||
self,
|
||||
success: bool,
|
||||
msg: str = '',
|
||||
baseline_image_uri: str = '',
|
||||
treatment_image_uri: str = '',
|
||||
) -> VisualParityReport:
|
||||
return VisualParityReport(
|
||||
success=success,
|
||||
target=self.target,
|
||||
browser_name=self.ctx.driver.name,
|
||||
baseline_id=self.baseline_id,
|
||||
msg=msg,
|
||||
baseline_image_uri=baseline_image_uri,
|
||||
treatment_image_uri=treatment_image_uri
|
||||
)
|
||||
|
||||
def _update_baseline(self):
|
||||
|
||||
baseline_bytes = self.element.screenshot_as_png
|
||||
baseline_image = Image.open(BytesIO(baseline_bytes))
|
||||
|
||||
self.perform_exclusions(baseline_image)
|
||||
|
||||
baseline_bytes = self._get_image_bytes(baseline_image)
|
||||
|
||||
image_uri = self.hook.quilla_store_image(
|
||||
ctx=self.ctx,
|
||||
baseline_id=self.baseline_id,
|
||||
image_bytes=baseline_bytes,
|
||||
image_type=VisualParityImageType.BASELINE
|
||||
)
|
||||
|
||||
if image_uri is None:
|
||||
return self._no_storage_mechanism_report
|
||||
|
||||
if image_uri == '':
|
||||
return self._create_report(
|
||||
success=False,
|
||||
msg='Unable to update the baseline image'
|
||||
)
|
||||
|
||||
return self._create_report(
|
||||
success=True,
|
||||
baseline_image_uri=image_uri,
|
||||
msg='Successfully updated baseline URI'
|
||||
)
|
||||
|
||||
@property
|
||||
def _no_storage_mechanism_report(self):
|
||||
return self._create_report(
|
||||
success=False,
|
||||
msg='No baseline storage mechanism configured',
|
||||
)
|
||||
|
||||
def _get_image_bytes(self, image: Image.Image):
|
||||
buffer = BytesIO()
|
||||
image.save(buffer, format='PNG')
|
||||
buffer.seek(0)
|
||||
|
||||
return buffer.getvalue()
|
||||
|
||||
def perform_exclusions(self, image: Image.Image):
|
||||
'''
|
||||
Using the 'excludeXPaths' parameter, grabs all the necessary elements
|
||||
that should be excluded, and removes them from the screenshot by covering
|
||||
the element position with a black box that is of the same size as the
|
||||
bounding box of that element.
|
||||
|
||||
This mutates the original image, so nothing is returned
|
||||
|
||||
Args:
|
||||
image: The image to perform the exclusions on
|
||||
'''
|
||||
|
||||
exclusion_xpaths = self._parameters.get('excludeXPaths', [])
|
||||
|
||||
for xpath in exclusion_xpaths:
|
||||
self._exclude_element_from_image(image, xpath)
|
||||
|
||||
def _exclude_element_from_image(self, image: Image.Image, exclude_target: str):
|
||||
resolved_exclusion_target = self.ctx.perform_replacements(exclude_target)
|
||||
exclude_element = self.driver.find_element_by_xpath(resolved_exclusion_target)
|
||||
|
||||
if not self._verify_exclude_target_within_bounding_box(exclude_element):
|
||||
raise FailedStepException(
|
||||
'Exclusion target element %s is not within the bounding box of %s' % (
|
||||
exclude_target, self._target
|
||||
)
|
||||
)
|
||||
|
||||
pos = self._get_element_location(exclude_element)
|
||||
size = self._get_element_size(exclude_element)
|
||||
|
||||
censor_image = Image.new(image.mode, size)
|
||||
|
||||
image.paste(censor_image, pos)
|
||||
|
||||
def _get_element_bbox(self, element: WebElement) -> Tuple[Tuple[int, int], Tuple[int, int]]:
|
||||
'''
|
||||
Given a web element, calculates its bounding box
|
||||
|
||||
Args:
|
||||
element: A web element retrieved by Selenium
|
||||
|
||||
Returns:
|
||||
A pair of tuples representing the bottom left and top right corners of the bounding box
|
||||
'''
|
||||
x, y = self._get_element_location(element)
|
||||
width, height = self._get_element_size(element)
|
||||
|
||||
top_left = (x, y)
|
||||
bottom_right = (x + width, y + height)
|
||||
|
||||
return top_left, bottom_right
|
||||
|
||||
def _get_element_size(self, element: WebElement) -> Tuple[int, int]:
|
||||
'''
|
||||
Gets the given element integer size
|
||||
|
||||
Args:
|
||||
element: A web element retrieved by Selenium
|
||||
|
||||
Returns:
|
||||
A (width, height) integer tuple describing the size of the element
|
||||
'''
|
||||
|
||||
size = element.size
|
||||
width = ceil(size['width'])
|
||||
height = ceil(size['height'])
|
||||
|
||||
return (width, height)
|
||||
|
||||
def _get_element_location(self, element: WebElement) -> Tuple[int, int]:
|
||||
'''
|
||||
Gets the integer location of the given element
|
||||
|
||||
Args:
|
||||
element: A web element retrieved by Selenium
|
||||
|
||||
Returns:
|
||||
A (x, y) integer tuple describing the location of the element
|
||||
'''
|
||||
loc = element.location
|
||||
|
||||
x = floor(loc['x'])
|
||||
y = floor(loc['y'])
|
||||
|
||||
return (x, y)
|
||||
|
||||
def _verify_exclude_target_within_bounding_box(self, exclude_element: WebElement) -> bool:
|
||||
'''
|
||||
Ensures that the XPath specified is within the actual bounding box of the
|
||||
current element target.
|
||||
|
||||
Note, this will return False if the XPath is only partially within the bounding
|
||||
box of the target.
|
||||
|
||||
Args:
|
||||
exclude_target: The web element that should be excluded
|
||||
|
||||
Returns:
|
||||
True if the element is within the target bounding box, False otherwise.
|
||||
'''
|
||||
exclude_left, exclude_right = self._get_element_bbox(exclude_element)
|
||||
target_left, target_right = self._get_element_bbox(self.element)
|
||||
|
||||
bottom_contained = target_left[0] <= exclude_left[0] and target_left[1] <= exclude_left[1]
|
||||
top_contained = exclude_right[0] <= target_right[0] and exclude_right[1] <= target_right[1]
|
||||
|
||||
return bottom_contained and top_contained
|
||||
|
||||
def perform(self) -> ValidationReport:
|
||||
self._verify_parameters('baselineID')
|
||||
|
||||
baseline_id = self.parameters['baselineID']
|
||||
update_baseline = (
|
||||
self.ctx.update_all_baselines or
|
||||
baseline_id in self.ctx.update_baseline
|
||||
)
|
||||
|
||||
if update_baseline:
|
||||
return self._update_baseline()
|
||||
|
||||
treatment_image_bytes = self.element.screenshot_as_png
|
||||
treatment_image = Image.open(BytesIO(treatment_image_bytes))
|
||||
treatment_image.load() # Make sure the image is actually loaded
|
||||
|
||||
self.perform_exclusions(treatment_image)
|
||||
|
||||
treatment_image_bytes = self._get_image_bytes(treatment_image)
|
||||
|
||||
baseline_image_bytes: Optional[bytes] = self.hook.quilla_get_visualparity_baseline(
|
||||
ctx=self.ctx,
|
||||
baseline_id=baseline_id
|
||||
)
|
||||
|
||||
if baseline_image_bytes is None:
|
||||
return self._no_storage_mechanism_report
|
||||
|
||||
if baseline_image_bytes == b'':
|
||||
if self.ctx.create_baseline_if_none:
|
||||
return self._update_baseline()
|
||||
|
||||
return self._create_report(
|
||||
success=False,
|
||||
msg='No baseline image found'
|
||||
)
|
||||
|
||||
baseline_image = Image.open(BytesIO(baseline_image_bytes))
|
||||
baseline_image.load() # Make sure the image is actualy loaded
|
||||
|
||||
success = baseline_image == treatment_image # Run the comparison with Pillow
|
||||
|
||||
if success:
|
||||
return self._create_report(
|
||||
success=success,
|
||||
)
|
||||
|
||||
treatment_uri = self.hook.quilla_store_image(
|
||||
ctx=self.ctx,
|
||||
baseline_id=baseline_id,
|
||||
image_bytes=treatment_image_bytes,
|
||||
image_type=VisualParityImageType.TREATMENT,
|
||||
)
|
||||
|
||||
baseline_uri = self.hook.quilla_get_baseline_uri(
|
||||
ctx=self.ctx,
|
||||
run_id=self.ctx.run_id,
|
||||
baseline_id=baseline_id
|
||||
)
|
||||
|
||||
return self._create_report(
|
||||
success=False,
|
||||
baseline_image_uri=baseline_uri,
|
||||
treatment_image_uri=treatment_uri,
|
||||
)
|
||||
|
|
@ -5,25 +5,22 @@ from typing import (
|
|||
Callable,
|
||||
List
|
||||
)
|
||||
from io import BytesIO
|
||||
|
||||
from selenium.webdriver.remote.webdriver import WebDriver
|
||||
from selenium.webdriver.remote.webelement import WebElement
|
||||
from selenium.webdriver.common.by import By
|
||||
from PIL import Image
|
||||
|
||||
from quilla.ctx import Context
|
||||
from quilla.common.enums import (
|
||||
XPathValidationStates,
|
||||
ValidationStates,
|
||||
ValidationTypes,
|
||||
VisualParityImageType,
|
||||
)
|
||||
from quilla.reports import (
|
||||
ValidationReport,
|
||||
VisualParityReport
|
||||
)
|
||||
from quilla.steps.base_steps import BaseValidation
|
||||
from quilla.steps.validations.visual_parity import VisualParityState
|
||||
|
||||
|
||||
class XPathValidation(BaseValidation):
|
||||
|
|
@ -208,124 +205,10 @@ class XPathValidation(BaseValidation):
|
|||
)
|
||||
|
||||
def _check_visual_parity(self) -> ValidationReport:
|
||||
self._verify_parameters('baselineID')
|
||||
|
||||
baseline_id = self.parameters['baselineID']
|
||||
update_baseline = (
|
||||
self.ctx.update_all_baselines or
|
||||
baseline_id in self.ctx.update_baseline
|
||||
visual_parity = VisualParityState(
|
||||
self.ctx,
|
||||
self._target,
|
||||
self._parameters
|
||||
)
|
||||
|
||||
if update_baseline:
|
||||
result = self.ctx.pm.hook.quilla_store_image(
|
||||
ctx=self.ctx,
|
||||
baseline_id=baseline_id,
|
||||
image_bytes=self.element.screenshot_as_png,
|
||||
image_type=VisualParityImageType.BASELINE
|
||||
)
|
||||
|
||||
if result is None:
|
||||
return VisualParityReport(
|
||||
success=False,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
msg='No baseline storage mechanism configured'
|
||||
)
|
||||
|
||||
baseline_uri = result
|
||||
|
||||
if baseline_uri == '':
|
||||
return VisualParityReport(
|
||||
success=False,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
msg='Unable to update the baseline image'
|
||||
)
|
||||
|
||||
return VisualParityReport(
|
||||
success=True,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
baseline_image_uri=baseline_uri,
|
||||
msg='Successfully updated baseline URI'
|
||||
)
|
||||
|
||||
treatment_image_bytes = self.element.screenshot_as_png
|
||||
treatment_image = Image.open(BytesIO(treatment_image_bytes))
|
||||
|
||||
plugin_result = self.ctx.pm.hook.quilla_get_visualparity_baseline(
|
||||
ctx=self.ctx,
|
||||
baseline_id=baseline_id
|
||||
)
|
||||
|
||||
if plugin_result is None:
|
||||
return VisualParityReport(
|
||||
success=False,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
msg='No baseline storage mechanism configured'
|
||||
)
|
||||
|
||||
if len(plugin_result) == 0:
|
||||
if self.ctx.create_baseline_if_none:
|
||||
baseline_uri = self.ctx.pm.hook.quilla_store_image(
|
||||
ctx=self.ctx,
|
||||
baseline_id=baseline_id,
|
||||
image_bytes=self.element.screenshot_as_png,
|
||||
image_type=VisualParityImageType.BASELINE
|
||||
)
|
||||
return VisualParityReport(
|
||||
success=True,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
baseline_image_uri=baseline_uri,
|
||||
msg='Successfully updated baseline URI'
|
||||
)
|
||||
|
||||
return VisualParityReport(
|
||||
success=False,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
msg='No baseline image found'
|
||||
)
|
||||
|
||||
baseline_image_bytes = plugin_result
|
||||
baseline_image = Image.open(BytesIO(baseline_image_bytes))
|
||||
|
||||
success = baseline_image == treatment_image # Run the comparison with Pillow
|
||||
|
||||
if success:
|
||||
return VisualParityReport(
|
||||
success=success,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
)
|
||||
|
||||
treatment_uri = self.ctx.pm.hook.quilla_store_image(
|
||||
ctx=self.ctx,
|
||||
baseline_id=baseline_id,
|
||||
image_bytes=treatment_image_bytes,
|
||||
image_type=VisualParityImageType.TREATMENT,
|
||||
)
|
||||
|
||||
baseline_uri = self.ctx.pm.hook.quilla_get_baseline_uri(
|
||||
ctx=self.ctx,
|
||||
run_id=self.ctx.run_id,
|
||||
baseline_id=baseline_id
|
||||
)
|
||||
|
||||
return VisualParityReport(
|
||||
success=False,
|
||||
target=self._target,
|
||||
browser_name=self.driver.name,
|
||||
baseline_id=baseline_id,
|
||||
baseline_image_uri=baseline_uri,
|
||||
treatment_image_uri=treatment_uri,
|
||||
)
|
||||
return visual_parity.perform()
|
||||
|
|
|
|||
|
|
@ -8,7 +8,9 @@
|
|||
"MerchantSubmitButton": "//button[@id='r3Butn']",
|
||||
"TrialSubmitButton": "//button[@id='checkButn']",
|
||||
"PasswordBanner": "//div[@id='passwordBanner']",
|
||||
"TrialCompleteBanner": "//div[@id='trialCompleteBanner']"
|
||||
"TrialCompleteBanner": "//div[@id='trialCompleteBanner']",
|
||||
"WholePage": "//html",
|
||||
"InfoFooter": "//div[@class='info-footer']"
|
||||
},
|
||||
"targetBrowsers": [
|
||||
"Firefox"
|
||||
|
|
@ -74,6 +76,18 @@
|
|||
"parameters": {
|
||||
"baselineID": "TrialCompleteBanner"
|
||||
}
|
||||
},
|
||||
{
|
||||
"action": "Validate",
|
||||
"type": "XPath",
|
||||
"state": "VisualParity",
|
||||
"target": "${{ Definitions.WholePage }}",
|
||||
"parameters": {
|
||||
"baselineID": "WholePage",
|
||||
"excludeXPaths": [
|
||||
"${{ Definitions.InfoFooter }}"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче