sap-automation/boilerplate
Kimmo Forss 6164da83fc
Release 3.8 final fixes (#438)
* Add DNS registration for secondary DNS names

* Use count.index

* Incorrect variable used

* Provide the correct defaults

* SQL Disk fixes and DNS Resourcegroup name

* Use the correct zone name DOH :)

* add db-ha-done file

* make the domain_ou_path a variable

* task name quotes

* Add additional configuration options

* Persist the configuration settings

* Remove quotes

* remove encoding

* Adding the Configuration parameters and the instance numbers

* Add Linebreaks

* become orchestration_ansible_user for the bom downloader playbook

* Update the variable with the corrected value

* Add a default value

* Pass the DNS Resource group

* Mounting logic update

* Don't persist an empty collections

* Added local.

* Added the missing .local

* Experimental tweaks from ACSS debug session (#417)

* Missing variable for non AFS scenarios

* Add use secondary ip flag

* Add a default value

* Create the temp files

* Debug mounting

* Install

* Mounting logic

* Don't mount local sapmnt

* Alignment

* Missing DNS resource group

* Added the local.

* Reboot handler

* Correct the Jinja

* Typo in jinja

* Add the missing variable

* Typo in JINJA

* Remove the domain_name variable

* hdbuserstore as sidadm

* reboot if waconf is changed

* Adding rsdb/ssfs_connect to DEFAULT.PFL

* Change the failed when condition for SUSE

* Adding the python version to support older distros

* Add a rescue action for setting ENSA if MessageServer is down

* Added logic for restarting message server for HA scenarios

* Use scs_running_on is not defined instead

* Add try statements

* Add try statements

* Change the IP for ansible

* Add a rescue action for SUSEConnect

* Missing region name

* BoM update with same name

* Provide ssfs_connect environment variable

* Don't specify python for deployer

* Add a wait

* Update message

* db_server_temp fix

* Misc updates

* Create users for Oracle

* Don't run User creation

* sa flag

* Set SA Enabled

* Show Debug info

* DBLoad

* Check storage account

* Let SAP create the users

* Adding more verbosity

* Typo and missing variable

* Set sa flag

* Don't create users separately when deploying on Oracle

* set allow_world_readable_tmpfiles for hdbuserstore code

* and when to check for windows family

* fix indentation

* Addd Windows checks

* Adding ORACLE-ASM as platform

* Mount install on db node

* add acl package to all tiers

* add it for all vms

* Add TF_LOG to workload and system pipelines

---------

Co-authored-by: Kimmo Forss <kimforss@microsoft.com>
Co-authored-by: hdamecharla <hdamecharla@microsoft.com>

* Calculate domain if not specified (#419)

* Calculate domain if not specified

* Missing POOL variable

---------

Co-authored-by: Kimmo Forss <kimforss@microsoft.com>

* Release 3.7 pipeline updates (#420)

* Calculate domain if not specified

* Missing POOL variable

* Fix the assert statements

* Add the calculated domain

* Download files on Windows

* Set ARM_USE_MSI

* Add use_msi into the provider

* Use the correct backend

* Adding the suffix

* Change provider order

* Switch provider

* testing how to get MSI ID

* Write the MSI Object ID

* remove the sensitivity

* Remove the "raw"

* Show the Deployer MSI

* deliberate error

* Removed the error

* Adding MSI client ID

* Don't remove workload VNet changes

* Testing jq

* Removed jq

* source the configuration from deployer

* correct source

* USE UPN for login

* Fix the providers

* Add domain calculation

* remap providers

* Use the main provider

* Provide the correct provider for sapmnt

* Use source for removals

* Remove extra login

* Rename variable

* Correct the variable

* Correct the parameter

* Corrected the variable name

* Merge branch 'experimental' of https://github.com/KimForss/sap-automation into experimental

* fix typos for region code wus2

* Software download tests on Windows

* whitespace removal

---------

Co-authored-by: Kimmo Forss <kimforss@microsoft.com>
Co-authored-by: hdamecharla <hdamecharla@microsoft.com>

* Change Oracle -> OracleLinux

* Update github-actions-ansible-lint.yml

* Update github-actions-ansible-lint.yml

* ansible lint fixes - minor

* ansible lint fixes - minor 002

* ansible lint fixes - minor 003

* Set the Terraform environment variables for non MSFT created deployers

* Add Shared Disk for Windows clusters

* linting

* Added the missing counter

* Change download directory to be Agent.TempDirectory

* Add SCSHA and ERS inifile for HA deployments of S4HANA_2021_FP01 BOM (#430)

* Fix hardcoded HANA port in hdbuserstore list (#435)

HANA port in hdbuserstore is hardcoded to 30313 for App and PAS install tasks.
Fixed to correctly generate correct port for deployed HANA instances.

* Update github-actions-ansible-lint.yml

* Update github-actions-ansible-lint.yml

* Pre 3.8 release  (#434)

* update NSG name

* resource_name

* Resource name

* ConnectivityToSAPApplicationSubnetFromControlPlane-ssh-rdp-winrm

* admin subnet

* Fix Rescue task for ERS

* Use the first item

* Use the correct plan

* Add the plan details

* Add dictionary object type to transform enable Plan

* debug plan

* debug: plan

* Populate source_image_reference even if plan is defined

* Remove whitespace

* Add more ensa1/ensa2 logic

* Add the ability to override ensa1/ensa2

* Add ensa1 for JAVA

* Use scs_bom_instance_type

* debug - change the resource attribute

* Show Ensa1 & Ensa2

* Testing permission setting

* test permission DSC

* Test SqlPermission

* pass necessary variables to the sap specific os configuration

* try credssp

* further complicate life

* CNO needs to be disabled. Reverting credssp changes for now.

* checking if we really need the become

* enable credssp and try again

* remove the becomes

* give relative path

* changes required for enabling credssp and relative paths for tasks inside blocks.

* fixing typos-001

* debug for SQL Server service status

* change to using split instead of cidrnetmask

* become a domain user

* correct plan disctioanry

* add a random string to cluster name

* shared disk resources

* type was defined twice

* debug

* change names

* add clarity in task names

* correct the windows cluster svc account name

* ensure create cluster is adding the cluster disks

* typos in output name

* make some changes to the db ha playbook

* tier as a set fact persists and overrides the var passed to the role. So change tier to config_tier

* the sql server installation tasks are being called. Need to investigate.

* small adjustments

* small changes

* another set of small changes

* add a few debugs

* commit some small changes

* switch to include tasks and include role so that the role/tasks are included as they are encountered.

* rename tasks etc

* do the installation validation

* reorder tasks

* missing dsc modules

* correct casing for the agent account name

* typos in dsc resource names

* generic mistakes

* parameter names

* Test to run without PsDscRunAsCredential_username

* make serverpermission a dictionary

* pass on permissions

* should they be passed liek this

* pass the other two states

* become true for setting permissions

* fix typo again

* calculate ag name

* calculate ag listener name

* hardcoded backup path for now

* calculate subnet prefix

* debug statements

* sql server dsc modules call subnet prefix, unlike the cluster ones.

* removed plan

* subnet prefixes not available in same set fact

* Install CLI the long way

* Add support for providing the password for the SPNs

* save and push changes before re-run

* check if the backup path string is the problem

* change the backup path back

* Add debug info

* support re-run

* delegate the add db to ag task to the primary node

* The sqlagdatabase as the last task on each node

* Add RHEL support

* package manager init

* correct distro

* Add RHEL support

* add samples directory

* use interactive variables

* remove the double $

* Move lsb_release inside of case statement

* Azure CLI install

* Add the silent flag

* show distro

* remove the when condition

* add config_tier

* pass scs servername and add the domain service access to the SAP share

* create directories  that are needed for restore.

* jinja changes

* more jinja good ness

* change the location from where we call jinja and set fact for disks.

* Use the list

* More jinja

* More jinja

* remove the double backslashes

* rmove the SLES packages

* backslashes

* testing

* remove the second backslash

* remove the replace

* fixes

* testing

* indentation

* format

* virtualenv

* remove curlies

* remove local

* remove backslashes

* Debug

* debug

* split to list

* debugging

* Testing directory creation

* test the quotes

* reduced the output

* change quotes to see if regex escaping works differently

* Path formatting for windows-test001

* convert it to a list

* see if we can assign a var before

* let's give it one more shot

* more debug

* flatten before calling to mimic with_items

* add flattened list to debug

* set probe port as well

* print with loop_control

* check what index is being used

* modifications to use with_indexed_items

* get ip of host

* push with_indexed_items

* removed the hard coded

* probe port

* rename probe port script to use ps1 extension

* remove spaces between curlies

* start ag resource along with ip address

* sleep for a minute

* Fix the hdbuserstore command

* uncomment the sqlagdatabase

* give permissions to the domain account

* pass netbios domain name

* trying something by adding the machine accounts

* fixes to sqlpermissions

* become

* use only become and no dsc credentials

* switch to using sidadm account

* revert and give db_backupoperator

* service accounts

* add the user rights properly

* set them correctly

* fix params for SqlServiceAccount

* set dummy pass for gMSA accounts

* add a when clause

* assign spns

* change how dummy pass is encrypted

* create spns

* maybe we are passing the password wrong

* set the correct service account for the sql agent service.

* dsc seems to be throwing errors. switch to using win_service

* change the ordering.

* additional task to start agent service account and remove becomes for the adding database to AG

* use sid_for_disks to calculate the sid to use. use sap_sid if db_sid is not defined.

* final set of changes.

* some more changes

* pass in the variables needed

* add logic for reboot immediately after the updates.

* adding back the domain service account to sql logins

* Add a try statement

* add endpoint permissions on the second node

* Get passwords from key vault

* Get the credentials from key vault

* update probe port, and its configuration. Add accounts to sysadmin role.

* pass in the required variables.

* removed curlies

* pass in the domain name and the service accounts with the domain\accountname format

* Corrected the task name

* Set accounts for exports

* Do another restart

* change listener port

* try waiting for the secondary creation

* changes not pushed?

* check if DB is added to primary and secondary

* pass primary node to server name

* pass teh correct account name

* debug account names

* pass domain_sqlsvc_account

* remove the extra account calculation step

* convert domain name to upper

* caching facts leads to weird behaviour

* push changes

* profile changes etc.

* missing curly

* simplify the profile updates

* update task description

* Don't add the same sid multiple times

* Centralize the component installation

* Move packages call to run after domain join

* set tier

* Only check the DSC modules once

* move the call to playbooks

* change the rescue action for scs ha install to use builtin.shell instead of builtin.command

* Update Terraform version

* Test if the filter fixes disks_facts

* Testing if Volumes fixes it

* reverse the selection

* removed the typo

* update path

* update the community.windows from the palybook to fix the win_disk_facts module

* use builtin.raw with become

* pass bash as the executable to use

* Use correct venv

* Troubleshooting

* don't download module

* test without rejectattr

* Linting changes and removal of BOM-Catalog folder

* lint fixes - 0001

---------

Co-authored-by: Kimmo Forss <kimforss@microsoft.com>
Co-authored-by: hdamecharla <hdamecharla@microsoft.com>

* simplify the code

* add network rules

* Fix typos

* restore the code

* Changed the workload zone SPN naming logic

* use DNS if private endpoint

* lowercase DNS

* DNS and NSG

* Add DNS record

* Add DNS records

* deployer already has the packages

* Bump up the version

* rename sap_search.yamp file

* install Ansible

* removed the duplicate

* update SQL probe port to handle other cluster parameters as well.

* Update github-actions-ansible-lint.yml

* Update github-actions-ansible-lint.yml

* Update github-actions-ansible-lint.yml

* Update github-actions-ansible-lint.yml

* try with a linter file in another directory

* Update github-actions-ansible-lint.yml

* use the older lint-action plugin

* add ansible lint badge and fix markdown issues

* put all badges on the same line

* calculate the path fact for HDB using the db_instance_number

* Add SLES as supported deployer

* Add recursion

* Troubleshooting find

* Don't assign permissions for networks in different regions

* update the data source

* Fix the logic

* Adding sybase support as anydb offer with SAP Automation framework (#437)

* added ASE files

* added sybase specific configuration

* added parameter for sybase ASE

* removed BoM for ASE

* changed node.tier from ASE to SYBASE

* changed to SYBASE from ASE

* Added special OS package for Sybase DB

* added os config specific to ASE

* changed node tier from ase to sybase

* added sybase as node_tier

* added sybase instead of ase for node_tier

* changed the mount logic

* changed the syntax

* changed syntax

* syntax change for system mount

* changed disk names for ASE

* changed some syntax

* adding backup

* added log volume

* changed the logic for log volume

* changed the logic for data and log volumes

* changed the logic for data and log

* changed logic

* added sap keyword

* testing only for sapdata

* added saplog

* added sybase user with admin

* added admin user

* added sapsys

* added new users and groups

* added an extra line

* added an extra line

* removed exemple users

* added user and groups

* users and groups

* fixed some notation

* added new user

* changed the tier

* changed tier from sybase to sapos

* changes guid for sybase

* fixed user id for sap user

* re-shuffled users and groups

* correct syntax

* changed insallation jinga templete

* added ASE variables

* added a colon

* added blank default value

* removed values

* changed sap user

* added changes

* added sybase

* adeded new code for scs from kimforss repo

* change validate templete

* Temporarily changed deployer temp disk size

* added sybase to the pipeline for BoM

* added two additional disk

* changed SKU size

* changed disk name

* changed the logic of mount points

* aded new logic

* adjust a little bit

* corrected the mapping of the disk

* fixed the sybtemp issue

* added few new parameter

* changed the DB parameters

* changed users a little bit

* changed the ID of the users

* deleted old files

* added few extra logic

* fixed the line spacing

* reverted the change

* added new code for zip

* changed syntax

* syntax change

* added back the search logic

* added dirname

* reverted back the file chage

* added few extra codes

* changed the file permission

* fixed a little bit

* changed the value of recurse

* added new files for software installation

* added few lines of code to make ASE work

* merged few things

* deleted older installation files

* removed post installation

* finally prepared the main

* final validation of DB install

* changed the name of the file

* added kernel parameter for sybase

* did few cosmetic changes

* Done few cosmetic changes

* few cosmetic changes

* cleaned few tasks

* fixed few codes

* changes ini file generation

* renamed few files in the task

* added hyphen

* added right file name

* merged from kimmo's repo

* added new flag sa_enabled

* added few lines

* added few things

* removed the flag

* added new users

* added new users

* fixed sapinst group name

* changed the indent of the line

* changed little bit about the db account

* commented db users

* commented db users

* Changed the firewall sku from Standard to Basic

* Revert "Switch to use ansible.builtin.shell for web dispatcher install"

This reverts commit 0a8b724674.

* Revert "Merge remote-tracking branch 'upstream/main' into experimental"

This reverts commit 7ec0a2fe29, reversing
changes made to ef7d4a5e8b.

* Revert "Merge remote-tracking branch 'upstream/main'"

This reverts commit 305baa7deaff334c9c029130e8bab493c8f4a3b5, reversing
changes made to d4de3509d19b20e208d715e68d570cd1cc6253db.

* change the firewall SKU

* back to standard

* changed some users

* Lint batch 2

* lint part 3

* Lint errors v4

* linting

* Linting

* whitespace

* Linting fixes 6

* whitespace

* Unchanged file

* Batch 1 of Lint fixes

---------

Co-authored-by: sysadmin <admin@MngEnv994334.onmicrosoft.com>
Co-authored-by: prasroy <prasroy@microsoft.com>
Co-authored-by: Kimmo Forss <kimforss@microsoft.com>

* Cluster updates

* install pywinrm[credssp]

* Move the flag file creation earlier

* don't fail pywinrm

* Add refresh

* default the variables to "" and not null

* add missing parameters

* removing dnsmanagement

* remove dnsmanagement

* Revert "add missing parameters"

This reverts commit e4456cf45f.

* typo

* import issues

* remove the coalesce

* add a try statement

* Fix mgmt subscription

* simplify provider

* Don't register provider

* removing dnsmanagement

* remove dnsmanagement

* Revert "add missing parameters"

This reverts commit e4456cf45f.

* Revert "Revert "add missing parameters""

This reverts commit 13643fe734.

* reset dns management

* missing provider

* remove the refresh

* Add if statements

* Correct the path to the script

* remove the token caching

* removed the e5

* add pywinrm[credssp]

* Remove the credssp installation

* DB2 Install fix, added the storage account flag

* task naming

* Fixed the ARM deployment errors

* bump up the VM SKU

* define the ers hostname as well

* Fail if SCS installation fails

* Don't proceed with ERS if SCS fails

* fixed indentation

* Fixed resource typo

* enable storage account usage

* Add support for configuration_Settings

* Add additional import statements

* missing variables

* indentation

* incorrect variable name

* Add a better when condition for db2

* Read the bom even if db2 is installed

* removed extra variable for Oracle

* reboot scs/ers on Windows

* Better debug message

* Add installation telemetry

* set owner for bom subfolder

* Use ACSS error code handling

* Don't recycle the resources

* Don't fail for first validate

* Configureation settings serialization fix

* Upgrade to dotnet 7

* Standalone SCS

* make the scs_server = ansible_hostname

* removed duplicate when

* Add support for web_sid in web app

* Update to .Net 7

* SYBASE updates

* run generic user creation on Oracle as well

* added missing curly

* swap start order

* SCS Install: Create params directory

* debug service

* add backslashes

* restart sql

* Fix the file exists error

* restore the code

* set \ to the end of tmp

* set the primary and secondary nodes

* only set secondary_node if HA deployment

* corrected the if statement

* don't run windows HA tasks

* Update set path variable

* Debugging Oracle PAS install

* use tmpdir for odbc

* tmpdir.path

* workload zone output

* restore IgnoreNullValues

* ignore error message

* revert the secondary_node change

* misc fixes

* If plan fails stop

* Check plan error code

* validate kv name

* typo fix

* Support for deployments without web subnet

* Misc pre 3.8 fixes

* add kernel parameter call for PAS and APP deployments for HA configurations

* New DevOps creation assets

* Add zonal support for ANF disks

* remove null subnet IDs

* Add the web dispatcher instance ID

* Don't create endpoint for ANF

* Fail if plan contains errors

* fail if plan fails

* Misc fixes

* remove the plan output check

* report errors

* Fix apply after import

* ANF Updates

* Fix importing

* Use the correct provider

* use correct provider for ANF data source

* add   network_features    = "Standard"

* no AVG yet

* Switch to use new variables for ppg and avset

* Add the new ppg and avset variables to the web app

* keep the old variables in the model

* replaced the missing local variable

* update avset logic

* Updated scs to use the new use_ppg and use_avset variables

* Add new variables for the application servers

* update the Windows VMs as well

* variable updates

* fixed typo

* set obsolete variable to null

* Respect the _use_no flags

* Define the new variables for web dispatchers

* Linting fixes

* linting

* set maintenance mode on

* Remove pipeline updating

* try to deploy from deployer

* Update terraform version

* remove duplicate

* remove duplicate condition

* added task name

* set working directory

* test if dotnet is installed

* change order

* deploy web dispatchers with web_sid

* provide full path

* add web_sid

* show directory content

* Install dotnet

* Use installed dotnet

* export dotnet path

* Use the SDK

* missing semi colon

* Add , "sles_sap15.4"

* throttle mounting to reduce risk for AFS issues

* serialize SAP OS configuration

* corrected typo in task name

* changes based on testing

---------

Co-authored-by: Kimmo Forss <kimforss@microsoft.com>
Co-authored-by: hdamecharla <hdamecharla@microsoft.com>
Co-authored-by: Nadeen Noaman <95418928+nnoaman@users.noreply.github.com>
Co-authored-by: hdamecharla <71097261+hdamecharla@users.noreply.github.com>
Co-authored-by: Harm Jan Stam <harmjan+git@harmjan.com>
Co-authored-by: Shekhar Sorot ( MSFT ) <shekharssorot2002@gmail.com>
Co-authored-by: Prasenjit Roy <47930074+prasroy@users.noreply.github.com>
Co-authored-by: sysadmin <admin@MngEnv994334.onmicrosoft.com>
Co-authored-by: prasroy <prasroy@microsoft.com>
2023-04-24 14:46:38 +03:00
..
WORKSPACES v 3.6 release (#357) 2022-12-07 19:23:39 +05:30
custom v 3.6 release (#357) 2022-12-07 19:23:39 +05:30
01-deploy-control-plane.yml v 3.6 release (#357) 2022-12-07 19:23:39 +05:30
02-sap-workload-zone.yml Create PR for release v 3.7.0.0 (#421) 2023-02-21 18:51:15 +02:00
03-sap-system-deployment.yml v 3.6 release (#357) 2022-12-07 19:23:39 +05:30
04-sap-software-download.yml Release 3.8 final fixes (#438) 2023-04-24 14:46:38 +03:00
05-DB-and-SAP-installation.yml Release 3.8 final fixes (#438) 2023-04-24 14:46:38 +03:00
10-remover-terraform.yml v 3.6 release (#357) 2022-12-07 19:23:39 +05:30
11-remover-arm-fallback.yaml v 3.6 release (#357) 2022-12-07 19:23:39 +05:30
12-remove-control-plane.yaml v 3.6 release (#357) 2022-12-07 19:23:39 +05:30
README.md Create PR for release v 3.7.0.0 (#421) 2023-02-21 18:51:15 +02:00
resources.yml v 3.6 release (#357) 2022-12-07 19:23:39 +05:30

README.md

Boilerplate

The sap-automation repository is a framework that offers you automation to deploy a SAP certified installation. To use the sap-automation framework you need to create a repository of your own and reference the sap-automation repository. This way you keep your organization specific configuration and the sap-automation framework separate. Having the configuration separate prevents you from making unintended changes to the sap-automation framework and gives you the possibility to add your own processes and configuration on top of the sap-automation framework by including additional pipelines.

Getting started

To get started with your own configuration repository we added this boilerplate with the most basic configuration for the pipelines already configured. Just copy the content of the boilerplate folder into the root of your configuration repository and change the following configuration.

In the resources.yml configure the following configuration by replacing <MyGithubConnector> with the name of your Github connector.

    - repository: sap-automation
      type: github
      name: Azure/sap-automation
      endpoint: <MyGithubConnector>
      ref: refs/heads/main

The procedure to create a service connection is described here

How it works

During pipeline execution the two repositories will be present on the deployer-agent. Each of the repositories will be mapped using the following:

  • sap-automation will be mapped to /sap-automation
  • customer configuration repository will be mapped to /config

During execution the repositories will interact with each other by using the following:

flowchart LR
    subgraph deployer-agent
        pipeline--uses-->templated-pipelines
        templated-pipelines--uses-->workspace-configuration
        subgraph customer repository
            workspace-configuration
            pipeline
        end
        subgraph sap-automation
        templated-pipelines--executes-->sap-installation
        end
    end

Custom Nightly

To validate the remote repository every night on your own environment there is a nightly.yml inside custom. You can change this to your own needs. The configuration of variables is just an example. Look for the MGMTNIGHTLY and the NIGHT configuration inside the WORKSAPCES for the configuration you want to deploy there. The deployment will be removed after success.

Create the following libraries to support the nightly build:

  • MGMTNIGHTLY
  • NIGHT