From fc5a4ee268443846812fcf903653a55aee578d57 Mon Sep 17 00:00:00 2001 From: Sven Aelterman <17446043+SvenAelterman@users.noreply.github.com> Date: Fri, 12 Jan 2024 11:23:04 -0600 Subject: [PATCH] Automated deployment of database schema + several fixes (#57) * Do not install packages (lack of permission) Remove Apache commands * Instead of scraping SQL, execute it via REDCap * Comment updates * Deploy REDCap DB schema (`auto=1`) * Update comments with sample .bicepparam file * Minor debugging updates * Add new line to install.sh and print message after running install.php * Enhance clarity of sample param comments * Attempt to locate mysql executable * Bicep linting * Documentation updates * Fix case of storage account app setting names Minor fixes to variable names * Comment updates --- README.md | 36 +++++++++++++++---------------- main-sample.bicepparam | 20 +++++++++++------ main.bicep | 9 +++++--- manual.md | 8 +++++-- modules/sql/main.bicep | 2 ++ modules/sql/sql.bicep | 2 ++ modules/webapp/main.bicep | 4 ++-- modules/webapp/webapp.bicep | 14 +++++++----- scripts/bash/deploy.sh | 10 +++++---- scripts/bash/install.sh | 13 ++++++----- scripts/bash/postbuild.sh | 43 ++++++++----------------------------- scripts/bash/startup.sh | 21 ------------------ 12 files changed, 81 insertions(+), 101 deletions(-) diff --git a/README.md b/README.md index 7605226..c081d1c 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,26 @@ # REDCap Deployment on Azure -### Overview +## Overview + This repository provides you with the necessary resources and guidance to deploy the REDCap application on Microsoft’s Azure cloud platform. This allows you to leverage the power of cloud computing for your research data management needs. This template automates the deployment of the REDCap solution into Azure using managed PaaS resources. The template assumes you are deploying a version of REDCap that supports direct connection to Azure Blob Storage. If you deploy an older version, deployment will succeed but you will need to manually provision NFS storage in Azure, and delete the new storage account. For NFS, consider: +## Deployment Options +- ### Manual deployment -### Deployment Options - - ### Manual deployment + - For manual deployment process, please navigate [***here***](manual.md) - - For manual deployment process, please navigate [***here***](manual.md) +- ### CI/CD Deployment with GitHub - - ### CI/CD Deployment with GitHub + - Information pending - - Information pending +- ### CI/CD Deployment with Azure DevOps - - ### CI/CD Deployment with Azure DevOps + - Information pending - - Information pending - -### Details +## Details This template automates the deployment of the REDCap solution into Azure using managed PaaS resources. The template assumes you are deploying a version of REDCap that supports direct connection to Azure Blob Storage. If you deploy an older version, deployment will succeed but you will need to manually provision NFS storage in Azure, and delete the new storage account. For NFS, consider: @@ -54,11 +54,11 @@ If after deployment, you would instead like to use a different SMTP relay, edit If you use Exchange Online (part of the Microsoft 365 Suite), you can follow these steps to set it up and use it as an SMTP relay for this service: --> -### Setup +## Setup This template will automatically deploy the resources necessary to run REDCap in Azure using PaaS (Platform-as-a-Service) features. -**IMPORTANT**: _The "Workload Name" you choose will be re-used as part of the storage, website, and MySQL database name. Make sure you don't use characters that will be rejected by MySQL._ +**IMPORTANT**: *The "Workload Name" you choose will be re-used as part of the storage, website, and MySQL database name. Make sure you don't use characters that will be rejected.* After the template is deployed, deployment automation will download the REDCap ZIP file you specify, and install it in your web app. It will then automatically update the database connection information in the app. @@ -70,9 +70,9 @@ If you need to connect to the MySQL database using the MySQL client, you will ne The database user name defaults to `sqladmin` and the password is a random string of 25 characters. The password is stored in Key Vault. -### Post-Setup +## Post-Setup -After the deployment and installation of REDCap has completed, you will need to initialize the database manually. The application gets deployed via Kudu which calls the `deploy.sh` script. After deployment, the `postbuild.sh` script extracts the MySQL commands from REDCap's installation page (`install.php`) and drops the output into a file called `install.sql`. Both `install.sh` and `install.sql` files will be dropped into `/home` directory. +After the deployment and installation of REDCap has completed, you will need to configure some database settings manually. The application gets deployed via Kudu which calls the `deploy.sh` script. After deployment, the `postbuild.sh` script will call REDCap's built-in capability to deploy the database schema. However, the configuration of the attachment storage to Azure Storage requires executing SQL statements that cannot be automated at this time. There is an `install.sh` file that contains the statements to be executed. Once the source control deployment of REDCap has completed, you will need to SSH into the running container: @@ -81,13 +81,11 @@ Once the source control deployment of REDCap has completed, you will need to SSH Execute the following command from the `/home` directory: ```sh -bash install.sh +bash ./site/repository/scripts/bash/install.sh ``` ![ssh](images/install.png) -It will take a few minutes to execute the SQL. - Once you regain access to the console, you can navigate to the root of your app service and confirm everything shows green on the REDCap Configuration Check page - with the exception of CronJob status which you may have to manually invoke. If anything displays on that page in red or yellow, it is recommended that you perform a "Restart" of the Azure "App Service". This needs to be done due to the fact that some necessary server environment settings get changed after the initial deployment, but restarting the App Service will load the service with the intended settings. ## Note about REDCap "Easy Upgade" @@ -112,8 +110,8 @@ The "Easy Upgrade" feature in REDCap 8.11.0 and later is currently _not_ support - Updating PHP configurations -- Managed MySQL overview - +- MySQL Flexible Server overview + - SendGrid overview - Blob storage overview diff --git a/main-sample.bicepparam b/main-sample.bicepparam index d200821..b006e96 100644 --- a/main-sample.bicepparam +++ b/main-sample.bicepparam @@ -11,16 +11,24 @@ param sequence = 1 param identityObjectId = '' param vnetAddressSpace = '10.0.0.0/24' -// If providing redcapZipUrl, you do not need to provide REDCap community username and password. -// redcapZipUrl should not require authentication. +// There are two options for obtaining the REDCap zip file: +// 1. Specify a URL to a publicly accessible zip file. +// This URL should not require authentication of any kind. For example, an Azure blob storage URL with a SAS token is supported. +// 2. Specify a REDCap community username and password to download the zip file from the REDCap community. +// Do not specify a URL if you are using this option. The deployment script will download the zip file from the REDCap community. param redcapZipUrl = '' +// -- OR -- param redcapCommunityUsername = '' param redcapCommunityPassword = '' -param scmRepoUrl = '' -param scmRepoBranch = '' +// These values are used to configure the App Service Deployment Center. +// The defaults below are the Microsoft-maintained Azure REDCap PaaS repository. +// However, you should consider forking that repository and referencing your fork. +// If not specified, the deployment will use the Microsoft-maintained Azure REDCap PaaS repository. +param scmRepoUrl = 'https://github.com/Microsoft/azure-redcap-paas' +param scmRepoBranch = 'main' // ** Do not specify anything here! ** -// This parameter is required to be here but should be blank so the password doesn't leak. -// A password is generated for each deployment. +// This parameter is required to ensure the parameter file is valid, but should be blank so the password doesn't leak. +// A new password is generated for each deployment and stored in Key Vault. param sqlPassword = '' diff --git a/main.bicep b/main.bicep index 5ed03f8..11c1dc9 100644 --- a/main.bicep +++ b/main.bicep @@ -38,8 +38,8 @@ param redcapCommunityPassword string param scmRepoUrl string = 'https://github.com/microsoft/azure-redcap-paas' @description('Github Repo Branch where build scripts are downloaded from') param scmRepoBranch string = 'main' -@description('The prerequsites command before build to be run on the web app with an elevated privilege. This is used to install the required packages for REDCap.') -param preRequsitesCommand string = 'apt-get install unzip -y && apt-get install -y python3 python3-pip' +@description('The command before build to be run on the web app with an elevated privilege. This is used to install the required packages for REDCap deployment and operation.') +param prerequisiteCommand string = 'apt-get install unzip sendmail cron -y' param deploymentTime string = utcNow() @@ -342,6 +342,9 @@ module mySqlModule './modules/sql/main.bicep' = { resource webAppResourceGroup 'Microsoft.Resources/resourceGroups@2023-07-01' = { name: replace(rgNamingStructure, '{rgName}', 'web') location: location + tags: union(tags, { + workloadType: 'web' + }) } module webAppModule './modules/webapp/main.bicep' = { @@ -383,7 +386,7 @@ module webAppModule './modules/webapp/main.bicep' = { scmRepoUrl: scmRepoUrl scmRepoBranch: scmRepoBranch - preRequsitesCommand: preRequsitesCommand + prerequisiteCommand: prerequisiteCommand deploymentNameStructure: deploymentNameStructure diff --git a/manual.md b/manual.md index cd46714..07f4038 100644 --- a/manual.md +++ b/manual.md @@ -1,15 +1,17 @@ # Manually deploy Redcap using PowerShell -### Prerequisites: +## Prerequisites Install the following prerequisites on your local machine: + - **[PowerShell 7](https://learn.microsoft.com/powershell/scripting/install/installing-powershell?view=powershell-7.3)** - **[Az PowerShell module](https://learn.microsoft.com/powershell/azure/new-azureps-module-az?view=azps-10.3.0)** - **[Bicep tools](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/install)** - **[Git](https://git-scm.com/downloads)** - **[Visual Studio Code](https://code.visualstudio.com/download)** -### Deployment Steps: +## Deployment Steps + Perform the following steps to deploy the solution using PowerShell: - Fork this repository and clone it to your administrative workstation or alternatively you can just clone the repository and work with it directly: @@ -18,6 +20,7 @@ Perform the following steps to deploy the solution using PowerShell: ```powershell git clone https://github.com/Microsoft/azure-redcap-paas.git ``` + - Open the `azure-redcap-paas` folder in VSCode - Copy `main-sample.bicepparam` to a new file with a descriptive name, such as `main-*yourorg*.bicepparam` @@ -34,6 +37,7 @@ Perform the following steps to deploy the solution using PowerShell: - ***redcapCommunityPassword***: This is not required if redcapZipUrl is provided. Else The password for the Redcap community site. - ***scmRepoUrl***: If you have fork the repo, provide the URL to your forked repo. Else provide the URL to the original repo. - ***scmRepoBranch***: The branch of the repo to deploy from. The example of this parameter is `main` + - Execute `deploy.ps1` as shown below. ```PowerShell diff --git a/modules/sql/main.bicep b/modules/sql/main.bicep index 2e272bd..da3a0b6 100644 --- a/modules/sql/main.bicep +++ b/modules/sql/main.bicep @@ -5,6 +5,7 @@ param location string param tags object param customTags object param flexibleSqlServerName string +// TODO: Rename to integrationSubNetId param peSubnetId string param privateDnsZoneName string param sqlAdminUser string @@ -15,6 +16,7 @@ param deploymentScriptName string @description('MySQL version') @allowed([ + // TODO: Remove 5.7 '5.7' '8.0.21' //'8.0.32' diff --git a/modules/sql/sql.bicep b/modules/sql/sql.bicep index bbe09a4..82d58b6 100644 --- a/modules/sql/sql.bicep +++ b/modules/sql/sql.bicep @@ -114,6 +114,7 @@ resource database 'Microsoft.DBforMySQL/flexibleServers/databases@2021-12-01-pre } } +// Assign the Contributor role to the UAMI on the MySQL server to enable setting the "invisible primary key" parameter module uamiMySqlRoleAssignmentModule '../common/roleAssignment-mySql.bicep' = { name: 'mySqlRole' params: { @@ -123,6 +124,7 @@ module uamiMySqlRoleAssignmentModule '../common/roleAssignment-mySql.bicep' = { } } +// Turn off the "invisible primary key" parameter on the server resource dbConfigDeploymentScript 'Microsoft.Resources/deploymentScripts@2020-10-01' = { name: deploymentScriptName location: location diff --git a/modules/webapp/main.bicep b/modules/webapp/main.bicep index 207b285..b4f0c17 100644 --- a/modules/webapp/main.bicep +++ b/modules/webapp/main.bicep @@ -32,7 +32,7 @@ param redcapZipUrl string param redcapCommunityUsernameSecretRef string #disable-next-line secure-secrets-in-params param redcapCommunityPasswordSecretRef string -param preRequsitesCommand string +param prerequisiteCommand string param uamiId string @@ -71,7 +71,7 @@ module appService 'webapp.bicep' = { scmRepoUrl: scmRepoUrl scmRepoBranch: scmRepoBranch - preRequsitesCommand: preRequsitesCommand + prerequisiteCommand: prerequisiteCommand storageAccountContainerName: storageAccountContainerName storageAccountKeySecretRef: storageAccountKeySecretRef diff --git a/modules/webapp/webapp.bicep b/modules/webapp/webapp.bicep index 80b836c..f8fe2d0 100644 --- a/modules/webapp/webapp.bicep +++ b/modules/webapp/webapp.bicep @@ -24,7 +24,7 @@ param redcapCommunityUsernameSecretRef string param redcapCommunityPasswordSecretRef string param scmRepoUrl string param scmRepoBranch string = 'main' -param preRequsitesCommand string +param prerequisiteCommand string param appInsights_connectionString string param appInsights_instrumentationKey string @@ -68,7 +68,7 @@ resource webApp 'Microsoft.Web/sites@2022-03-01' = { linuxFxVersion: linuxFxVersion minTlsVersion: '1.2' ftpsState: 'FtpsOnly' - appCommandLine: preRequsitesCommand + appCommandLine: prerequisiteCommand appSettings: [ { name: 'redcapAppZip' @@ -127,17 +127,21 @@ resource webApp 'Microsoft.Web/sites@2022-03-01' = { value: '1' } { - name: 'storageKey' + name: 'StorageKey' value: storageAccountKeySecretRef } { - name: 'storageAccount' + name: 'StorageAccount' value: storageAccountName } { - name: 'storageContainerName' + name: 'StorageContainerName' value: storageAccountContainerName } + { + name: 'ENABLE_DYNAMIC_INSTALL' + value: 'true' + } ] } } diff --git a/scripts/bash/deploy.sh b/scripts/bash/deploy.sh index f28d945..0bbae9d 100644 --- a/scripts/bash/deploy.sh +++ b/scripts/bash/deploy.sh @@ -10,6 +10,7 @@ # Timestamp for log file # #################################################################################### + stamp=$(date +%Y-%m-%d-%H-%M) #################################################################################### @@ -20,8 +21,9 @@ stamp=$(date +%Y-%m-%d-%H-%M) echo "Configuring mysqli extension" >> /home/site/log-$stamp.txt cd /home/site -echo "extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqlnd_azure.so -extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqli.so" >> extensions.ini +# echo "extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqlnd_azure.so +# extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqli.so" >> extensions.ini +echo "extension=/usr/local/lib/php/extensions/no-debug-non-zts-20220829/mysqli.so" >> extensions.ini #################################################################################### # @@ -117,7 +119,7 @@ echo "session.cookie_secure = On" >> /home/site/redcap.ini #################################################################################### # -# Move postbuild.sh to PostDeploymentActions for execution after deployment +# Copy postbuild.sh to PostDeploymentActions for execution after deployment # #################################################################################### @@ -126,7 +128,7 @@ cp /home/site/repository/scripts/bash/postbuild.sh /home/site/deployments/tools/ #################################################################################### # -# Move startup.sh /home for a custom startup +# Copy startup.sh /home for a custom startup # #################################################################################### diff --git a/scripts/bash/install.sh b/scripts/bash/install.sh index 5bdedd5..e2c07fa 100644 --- a/scripts/bash/install.sh +++ b/scripts/bash/install.sh @@ -5,19 +5,22 @@ # # MIT License +echo -e "\nHello from install.sh" + +which mysql + #################################################################################### # -# Initialize the REDCap database using the SQL commands scraped from the -# install.php page, then update additional configuration settings including +# Update additional configuration settings including # user file uploading settings to Azure Blob Storage # #################################################################################### -mysql -u$APPSETTING_DBUserName -h$APPSETTING_DBHostName -p$APPSETTING_DBPassword --ssl=true --ssl-ca=/home/site/wwwroot/DigiCertGlobalRootCA.crt.pem < scraper.py -import requests -from bs4 import BeautifulSoup -page = requests.post("https://$WEBSITE_HOSTNAME/install.php") -soup = BeautifulSoup(page.content, "html.parser") -data = soup.find('textarea').text -with open("/home/install.sql", "w") as out: - for i in range(0, len(data)): - try: - out.write(data[i]) - except Exception: - 1+1 -EOF -python3 scraper.py -echo "completed running scraper.py with $?" -#################################################################################### -# -# Copy the install.sh file to the /home directory -# -#################################################################################### - -cp /home/site/repository/scripts/bash/install.sh /home/install.sh \ No newline at end of file +bash /home/site/repository/scripts/bash/install.sh diff --git a/scripts/bash/startup.sh b/scripts/bash/startup.sh index 2181a44..57f36e2 100644 --- a/scripts/bash/startup.sh +++ b/scripts/bash/startup.sh @@ -1,18 +1,5 @@ #!/bin/bash -a2enmod headers -echo "Header set MyHeader \"%D %t"\" >> /etc/apache2/apache2.conf -echo "Header always unset \"X-Powered-By\"" >> /etc/apache2/apache2.conf -echo "Header unset \"X-Powered-By\"" >> /etc/apache2/apache2.conf - -#################################################################################### -# -# Install some utilities for REDCap to work properly -# -#################################################################################### - -apt-get update -apt-get install -y sendmail cron #################################################################################### # @@ -22,11 +9,3 @@ apt-get install -y sendmail cron echo "* * * * * /usr/local/bin/php /home/site/wwwroot/cron.php > /dev/null" >> /etc/crontab service cron start - -#################################################################################### -# -# Start Apache -# -#################################################################################### - -/usr/sbin/apache2ctl -D FOREGROUND \ No newline at end of file