terraform-azurerm-caf-enterprise-scale/tests/README.md

Test Framework for the Azure landing zones Terraform module

This folder contains code relating to the test framework for this module. Testing is currently performed in the following stages:

1. Code Review (GitHub Actions)
2. Unit Tests (Azure Pipelines)
3. E2E Tests (Azure Pipelines)
4. Update Test Baseline (Azure Pipelines)

The decision to break testing up in this manner was to ensure developers get quick feedback when working on bug fixes and new features, whilst providing greater assurance that the latest updates work as expected and do not break existing functionality.

Code Review (GitHub Actions)

The first quality check ensures all code complies with recommended coding practices. We use GitHub Super-Linter (v4.1.0) to perform this initial check across the code base. By running this within a GitHub Action, anyone contributing to the code can get quick feedback on each commit pushed to GitHub.

GitHub Super-Linter is configured to run checks against the full codebase using the following Linters:

Language	Linter
JSON	jsonlint
Markdown	markdownlint
PowerShell	PSScriptAnalyzer
Shell	Shellcheck / [executable bit check] / shfmt
Terraform	tflint / terrascan
YAML	YamlLint

The Code Review GitHub Action runs automatically upon each commit to branches in the repository (including forks) other than main, patch-library and release/**. This is also a mandatory check on all PR's being raised against the main branch, enforced using branch protection rules.

Unit Tests (Azure Pipelines)

As linting only let's you know if the code is well written (according to a pre-determined set of standards), we also need to determine whether the code generates a valid Terraform plan.

To verify this, we have a set of unit tests which run additional checks against the module using a series of test deployments.

To give assurance that the module works with the specified range of supported versions of Terraform and the Azure provider, we use a matrix strategy) to automatically generate parallel running jobs for each version combination.

The Unit Tests consist of the following tasks:

Task Name	Description
Install Terraform Pre-requisites	Ensures the required version of Terraform is installed on the agent.
Prepare Terraform Environment	Retrieves credentials for the target test environment and sets a unique value for the root_id input variable.1
Terraform Linting (terraform fmt)	Runs terraform fmt against the entire repository in -check mode to ensure Terraform code is correctly formatted.
Install OPA/Conftest Pre-requisites	Ensure the required version of Conftest and jq are installed on the agent.
Test 001 (terraform init) Baseline	Initialize the root module for this test instance.
Test 001 (terraform plan) Baseline	Generate a Terraform plan for this test instance.
Test 001 (conftest) Baseline	Run Conftest to ensure the Terraform plan matches the expected configuration for this test instance.
Test 002 (terraform init) Add Custom Core	Initialize the root module for this test instance.
Test 002 (terraform plan) Add Custom Core	Generate a Terraform plan for this test instance.
Test 002 (conftest) Add Custom Core	Run Conftest to ensure the Terraform plan matches the expected configuration for this test instance.
Test 003 (terraform init) Add Management and Connectivity	Initialize the root module for this test instance.
Test 003 (terraform plan) Add Management and Connectivity	Generate a Terraform plan for this test instance.
Test 003 (conftest) Add Management and Connectivity	Run Conftest to ensure the Terraform plan matches the expected configuration for this test instance.

¹ Each job uses a dedicated SPN (with certificate based authentication) to connect to Azure. This is to minimize the risk of API rate limiting when running highly parallel resource deployments in the pipeline.

The Unit Tests Azure Pipeline is a mandatory check on all PR's being raised against the main branch, enforced using branch protection rules. To maintain security of the test environment, the Unit Tests Azure Pipeline must be manually initiated by a repository Admin or Maintainer once the submitted code changes have been reviewed. This is a security step as outlined below in the Why Azure Pipelines section of this page.

To run the Unit Tests Azure Pipeline, a repository Admin or Maintainer can add the comment /azp run unit to the PR.

E2E Tests (Azure Pipelines)

In addition to the Unit Tests, we have a set of full end-to-end tests. These are based on the same test modules as the Unit Tests, but run a full cycle of resource deployments using terraform apply, followed by a clean-up using terraform destroy.

The E2E Test workflow is designed to run the test modules in sequence, simulating an update scenario typical to how we might expect a customer to use this module. This approach gives assurance that the module works for both new deployments, and updates to existing deployments. It also allows us to verify that the module is able to successful destroy resources.

The E2E Tests consist of the following tasks:

Task Name	Description
Install Terraform Pre-requisites	Ensures the required version of Terraform is installed on the agent.
Prepare Terraform Environment	Retrieves credentials for the target test environment and sets a unique value for the root_id input variable.1
Terraform Linting (terraform fmt)	Runs terraform fmt against the entire repository in -check mode to ensure Terraform code is correctly formatted.
Test 001 (terraform init) Baseline	Initialize the root module for this test instance.
Test 001 (terraform plan) Baseline	Generate a Terraform plan for this test instance.
Test 001 (terraform apply) Baseline	Apply the Terraform plan for this test instance.
Test 002 (terraform init) Add Custom Core	Initialize the root module for this test instance.
Test 002 (terraform plan) Add Custom Core	Generate a Terraform plan for this test instance.
Test 002 (terraform apply) Add Custom Core	Apply the Terraform plan for this test instance.
Test 003 (terraform init) Add Management and Connectivity	Initialize the root module for this test instance.
Test 003 (terraform plan) Add Management and Connectivity	Generate a Terraform plan for this test instance.
Test 003 (terraform apply) Add Management and Connectivity	Apply the Terraform plan for this test instance.
Clean-up Test Environment (terraform destroy)	Run terraform destroy to clean-up the test environment.2

¹ Each job uses a dedicated SPN (with certificate based authentication) to connect to Azure. This is to minimize the risk of API rate limiting when running highly parallel resource deployments in the pipeline.

² The terraform destroy task uses the always() condition to ensure the environment is cleaned-up if any of the previous tasks fail after a partial deployment.

The E2E Tests Azure Pipeline is an optional check for PR's being raised against the main branch. Although not enforced through branch protection rules, this test should always be run before merging any code changes to the repository which could impact the functionality of the module. This test can be skipped for PRs containing documentation changes only. To maintain security of the test environment, the E2E Tests Azure Pipeline must be manually initiated by a repository Admin or Maintainer once the submitted code changes have been reviewed. This is a security step as outlined below in the Why Azure Pipelines section of this page.

To run the E2E Tests Azure Pipeline, a repository Admin or Maintainer can add the comment /azp run e2e to the PR.

Update Test Baseline (Azure Pipelines)

To provide a test baseline for the OPA checks within the Unit Tests Azure Pipeline, each test module contains a baseline_values.json file which is the JSON output of terraform plan containing a known good configuration.

These files must be updated when changes are made to the module to ensure it reflects the new plan.

The Update Tests Baseline Azure Pipeline ensures this is done in a consistent manner against the test environment.

The E2E Tests consist of the following tasks:

Task Name	Description
Checkout	Set with persistCredentials: true to ensure the OAuth credentials used for accessing the GitHub repository are available for other tasks.
Update OPA baseline values	Sequentially run through each test module, generating a Terraform plan to update the corresponding baseline_values.json files.
Merge changes to repository	Runs git commands to ensure the updated baseline_values.json files are added to the PR when changes are detected.

To run the Update Tests Baseline Azure Pipeline, a repository Admin or Maintainer can add the comment /azp run update to the PR. This should be run before the Unit Tests Azure Pipeline, and the generated git diff must be reviewed to understand the impact of the changes on the plan.

IMPORTANT: If unexpected changes are observed in the git diff for any baseline_values.json files, the code should be reviewed and updated to ensure the module is functioning as expected.

Why Azure Pipelines?

The Unit Tests, E2E Tests, and Update Test Baseline workflows need valid Azure credentials to authenticate with the Azure platform for Terraform to work. These tests are run on Azure Pipelines as a security measure, allowing contributed code from forked repositories to be reviewed before tests are manually triggered by a repository contributor using comment triggers. Although GitHub Actions could technically run these jobs, GitHub prevents access to secrets for jobs triggered from forks as s security measure. Using this approach ensures a repository Admin or Maintainer is always in control of code changes being run against our test environment.

Multi-job configuration (matrix strategy)

Azure Pipelines provides the option to define a multi-job configuration. This enables multi-configuration testing to be implemented from a common set of tasks, with the benefit of running multiple jobs on multiple agents in parallel.

Our implementation uses a programmatically generated matrix strategy to ensure we can meet our testing requirements. This is designed to ensure the module works with different combinations of Terraform and Azure provider versions. The strategy is generated by a PowerShell script, and is used by both the Unit and E2E tests.

The current strategy consists of running tests against the following version combinations:

• Terraform versions:
  • Minimum version supported by the module (1.7.0)
• Azure provider for Terraform versions:
  • Minimum version supported by the module (v3.107.0)
  • Latest version

The latest versions are determined programmatically by querying the publisher APIs. This negates the need to update the code or pipeline to ensure the latest version is being tested.

With the frequency at which we run tests these combinations give reasonable assurance that the module will work with all version combinations up to the latest versions, not withstanding any which temporarily introduce bugs.

The matrix strategy also uses the Microsoft.Subscription/aliases@2021-10-01 API to map Subscriptions to each job within the Matrix. This ensures that each job has dedicated Subscriptions to deploy resources into, and place within the Management Group hierarchy. In combination with the dedicated SPN per job, this also increases the API rate limits available to the pipeline.