Automated deployment of database schema + several fixes (#57)

* Do not install packages (lack of permission) Remove Apache commands * Instead of scraping SQL, execute it via REDCap * Comment updates * Deploy REDCap DB schema (`auto=1`) * Update comments with sample .bicepparam file * Minor debugging updates * Add new line to install.sh and print message after running install.php * Enhance clarity of sample param comments * Attempt to locate mysql executable * Bicep linting * Documentation updates * Fix case of storage account app setting names Minor fixes to variable names * Comment updates
2024-01-12 11:23:04 -06:00 · 2024-01-12 11:23:04 -06:00 · fc5a4ee268
--- a/README.md
+++ b/README.md
@ -1,26 +1,26 @@
 # REDCap Deployment on Azure

-### Overview
+## Overview
+
 This repository provides you with the necessary resources and guidance to deploy the REDCap application on Microsoft’s Azure cloud platform. This allows you to leverage the power of cloud computing for your research data management needs.

 This template automates the deployment of the REDCap solution into Azure using managed PaaS resources. The template assumes you are deploying a version of REDCap that supports direct connection to Azure Blob Storage. If you deploy an older version, deployment will succeed but you will need to manually provision NFS storage in Azure, and delete the new storage account. For NFS, consider:

+## Deployment Options

+- ### Manual deployment

-### Deployment Options
-  - ### Manual deployment
+  - For manual deployment process, please navigate [***here***](manual.md)

-    - For manual deployment process, please navigate [***here***](manual.md)
+- ### CI/CD Deployment with GitHub

-  - ### CI/CD Deployment with GitHub
+  - Information pending

-    - Information pending
+- ### CI/CD Deployment with Azure DevOps

-  - ### CI/CD Deployment with Azure DevOps
+  - Information pending

-    - Information pending
-
-### Details
+## Details

 This template automates the deployment of the REDCap solution into Azure using managed PaaS resources. The template assumes you are deploying a version of REDCap that supports direct connection to Azure Blob Storage. If you deploy an older version, deployment will succeed but you will need to manually provision NFS storage in Azure, and delete the new storage account. For NFS, consider:

@ -54,11 +54,11 @@ If after deployment, you would instead like to use a different SMTP relay, edit

 If you use Exchange Online (part of the Microsoft 365 Suite), you can follow these steps to set it up and use it as an SMTP relay for this service: <https://learn.microsoft.com/Exchange/mail-flow-best-practices/how-to-set-up-a-multifunction-device-or-application-to-send-email-using-microsoft-365-or-office-365> -->

-### Setup
+## Setup

 This template will automatically deploy the resources necessary to run REDCap in Azure using PaaS (Platform-as-a-Service) features.

-**IMPORTANT**: _The "Workload Name" you choose will be re-used as part of the storage, website, and MySQL database name. Make sure you don't use characters that will be rejected by MySQL._
+**IMPORTANT**: *The "Workload Name" you choose will be re-used as part of the storage, website, and MySQL database name. Make sure you don't use characters that will be rejected.*

 After the template is deployed, deployment automation will download the REDCap ZIP file you specify, and install it in your web app. It will then automatically update the database connection information in the app.

@ -70,9 +70,9 @@ If you need to connect to the MySQL database using the MySQL client, you will ne

 The database user name defaults to `sqladmin` and the password is a random string of 25 characters. The password is stored in Key Vault.

-### Post-Setup
+## Post-Setup

-After the deployment and installation of REDCap has completed, you will need to initialize the database manually. The application gets deployed via Kudu which calls the `deploy.sh` script. After deployment, the `postbuild.sh` script extracts the MySQL commands from REDCap's installation page (`install.php`) and drops the output into a file called `install.sql`. Both `install.sh` and `install.sql` files will be dropped into `/home` directory.
+After the deployment and installation of REDCap has completed, you will need to configure some database settings manually. The application gets deployed via Kudu which calls the `deploy.sh` script. After deployment, the `postbuild.sh` script will call REDCap's built-in capability to deploy the database schema. However, the configuration of the attachment storage to Azure Storage requires executing SQL statements that cannot be automated at this time. There is an `install.sh` file that contains the statements to be executed.

 Once the source control deployment of REDCap has completed, you will need to SSH into the running container:

@ -81,13 +81,11 @@ Once the source control deployment of REDCap has completed, you will need to SSH
 Execute the following command from the `/home` directory:

 ```sh
-bash install.sh
+bash ./site/repository/scripts/bash/install.sh
 ```

 ![ssh](images/install.png)

-It will take a few minutes to execute the SQL.
-
 Once you regain access to the console, you can navigate to the root of your app service and confirm everything shows green on the REDCap Configuration Check page - with the exception of CronJob status which you may have to manually invoke. If anything displays on that page in red or yellow, it is recommended that you perform a "Restart" of the Azure "App Service". This needs to be done due to the fact that some necessary server environment settings get changed after the initial deployment, but restarting the App Service will load the service with the intended settings.

 ## Note about REDCap "Easy Upgade"
@ -112,8 +110,8 @@ The "Easy Upgrade" feature in REDCap 8.11.0 and later is currently _not_ support
  <https://learn.microsoft.com/azure/app-service/configure-ssl-certificate>
 - Updating PHP configurations
  <https://learn.microsoft.com/azure/app-service/configure-language-php?pivots=platform-linux#customize-phpini-settings>
- Managed MySQL overview
-  <https://learn.microsoft.com/azure/mysql/single-server/overview>
+- MySQL Flexible Server overview
+  <https://learn.microsoft.com/azure/mysql/flexible-server/overview>
 - SendGrid overview
  <https://docs.sendgrid.com/for-developers/partners/microsoft-azure-2021>
 - Blob storage overview
--- a/main-sample.bicepparam
+++ b/main-sample.bicepparam
@ -11,16 +11,24 @@ param sequence = 1
 param identityObjectId = '<Valid Entra ID object ID for permissions assignment>'
 param vnetAddressSpace = '10.0.0.0/24'

-// If providing redcapZipUrl, you do not need to provide REDCap community username and password.
-// redcapZipUrl should not require authentication.
+// There are two options for obtaining the REDCap zip file:
+// 1. Specify a URL to a publicly accessible zip file.
+//    This URL should not require authentication of any kind. For example, an Azure blob storage URL with a SAS token is supported.
+// 2. Specify a REDCap community username and password to download the zip file from the REDCap community.
+//    Do not specify a URL if you are using this option. The deployment script will download the zip file from the REDCap community.
 param redcapZipUrl = '<Valid Redcap Zip URL>'
+// -- OR --
 param redcapCommunityUsername = '<Valid Redcap Community Username>'
 param redcapCommunityPassword = '<Valid Redcap Community Password>'

-param scmRepoUrl = '<Valid Scm Repo URL where build scripts are downloaded from>'
-param scmRepoBranch = '<Valid Scm Repo Branch where build scripts are downloaded from>'
+// These values are used to configure the App Service Deployment Center.
+// The defaults below are the Microsoft-maintained Azure REDCap PaaS repository.
+// However, you should consider forking that repository and referencing your fork.
+// If not specified, the deployment will use the Microsoft-maintained Azure REDCap PaaS repository.
+param scmRepoUrl = 'https://github.com/Microsoft/azure-redcap-paas'
+param scmRepoBranch = 'main'

 // ** Do not specify anything here! **
-// This parameter is required to be here but should be blank so the password doesn't leak. 
-// A password is generated for each deployment.
+// This parameter is required to ensure the parameter file is valid, but should be blank so the password doesn't leak. 
+// A new password is generated for each deployment and stored in Key Vault.
 param sqlPassword = ''
--- a/main.bicep
+++ b/main.bicep
@ -38,8 +38,8 @@ param redcapCommunityPassword string
 param scmRepoUrl string = 'https://github.com/microsoft/azure-redcap-paas'
@description('Github Repo Branch where build scripts are downloaded from')
 param scmRepoBranch string = 'main'
-@description('The prerequsites command before build to be run on the web app with an elevated privilege. This is used to install the required packages for REDCap.')
-param preRequsitesCommand string = 'apt-get install unzip -y && apt-get install -y python3 python3-pip'
+@description('The command before build to be run on the web app with an elevated privilege. This is used to install the required packages for REDCap deployment and operation.')
+param prerequisiteCommand string = 'apt-get install unzip sendmail cron -y'

 param deploymentTime string = utcNow()

@ -342,6 +342,9 @@ module mySqlModule './modules/sql/main.bicep' = {
 resource webAppResourceGroup 'Microsoft.Resources/resourceGroups@2023-07-01' = {
  name: replace(rgNamingStructure, '{rgName}', 'web')
  location: location
+  tags: union(tags, {
+      workloadType: 'web'
+    })
 }

 module webAppModule './modules/webapp/main.bicep' = {
@ -383,7 +386,7 @@ module webAppModule './modules/webapp/main.bicep' = {

    scmRepoUrl: scmRepoUrl
    scmRepoBranch: scmRepoBranch
-    preRequsitesCommand: preRequsitesCommand
+    prerequisiteCommand: prerequisiteCommand

    deploymentNameStructure: deploymentNameStructure

--- a/manual.md
+++ b/manual.md
@ -1,15 +1,17 @@
 # Manually deploy Redcap using PowerShell

-### Prerequisites:
+## Prerequisites

 Install the following prerequisites on your local machine:
+
 - **[PowerShell 7](https://learn.microsoft.com/powershell/scripting/install/installing-powershell?view=powershell-7.3)**
 - **[Az PowerShell module](https://learn.microsoft.com/powershell/azure/new-azureps-module-az?view=azps-10.3.0)**
 - **[Bicep tools](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/install)**
 - **[Git](https://git-scm.com/downloads)**
 - **[Visual Studio Code](https://code.visualstudio.com/download)**

-### Deployment Steps:
+## Deployment Steps
+
 Perform the following steps to deploy the solution using PowerShell:

 - Fork this repository and clone it to your administrative workstation or alternatively you can just clone the repository and work with it directly:
@ -18,6 +20,7 @@ Perform the following steps to deploy the solution using PowerShell:
    ```powershell
    git clone https://github.com/Microsoft/azure-redcap-paas.git
    ```
+
 - Open the `azure-redcap-paas` folder in VSCode

 - Copy `main-sample.bicepparam` to a new file with a descriptive name, such as `main-*yourorg*.bicepparam`
@ -34,6 +37,7 @@ Perform the following steps to deploy the solution using PowerShell:
  - ***redcapCommunityPassword***: This is not required if redcapZipUrl is provided. Else The password for the Redcap community site.
  - ***scmRepoUrl***: If you have fork the repo, provide the URL to your forked repo. Else provide the URL to the original repo.
  - ***scmRepoBranch***: The branch of the repo to deploy from. The example of this parameter is `main`
+
 - Execute `deploy.ps1` as shown below.

    ```PowerShell
--- a/modules/sql/main.bicep
+++ b/modules/sql/main.bicep
@ -5,6 +5,7 @@ param location string
 param tags object
 param customTags object
 param flexibleSqlServerName string
+// TODO: Rename to integrationSubNetId
 param peSubnetId string
 param privateDnsZoneName string
 param sqlAdminUser string
@ -15,6 +16,7 @@ param deploymentScriptName string

@description('MySQL version')
@allowed([
+  // TODO: Remove 5.7
  '5.7'
  '8.0.21'
  //'8.0.32'
--- a/modules/sql/sql.bicep
+++ b/modules/sql/sql.bicep
@ -114,6 +114,7 @@ resource database 'Microsoft.DBforMySQL/flexibleServers/databases@2021-12-01-pre
  }
 }

+// Assign the Contributor role to the UAMI on the MySQL server to enable setting the "invisible primary key" parameter
 module uamiMySqlRoleAssignmentModule '../common/roleAssignment-mySql.bicep' = {
  name: 'mySqlRole'
  params: {
@ -123,6 +124,7 @@ module uamiMySqlRoleAssignmentModule '../common/roleAssignment-mySql.bicep' = {
  }
 }

+// Turn off the "invisible primary key" parameter on the server
 resource dbConfigDeploymentScript 'Microsoft.Resources/deploymentScripts@2020-10-01' = {
  name: deploymentScriptName
  location: location
--- a/modules/webapp/main.bicep
+++ b/modules/webapp/main.bicep
@ -32,7 +32,7 @@ param redcapZipUrl string
 param redcapCommunityUsernameSecretRef string
 #disable-next-line secure-secrets-in-params
 param redcapCommunityPasswordSecretRef string
-param preRequsitesCommand string
+param prerequisiteCommand string

 param uamiId string

@ -71,7 +71,7 @@ module appService 'webapp.bicep' = {

    scmRepoUrl: scmRepoUrl
    scmRepoBranch: scmRepoBranch
-    preRequsitesCommand: preRequsitesCommand
+    prerequisiteCommand: prerequisiteCommand

    storageAccountContainerName: storageAccountContainerName
    storageAccountKeySecretRef: storageAccountKeySecretRef
--- a/modules/webapp/webapp.bicep
+++ b/modules/webapp/webapp.bicep
@ -24,7 +24,7 @@ param redcapCommunityUsernameSecretRef string
 param redcapCommunityPasswordSecretRef string
 param scmRepoUrl string
 param scmRepoBranch string = 'main'
-param preRequsitesCommand string
+param prerequisiteCommand string

 param appInsights_connectionString string
 param appInsights_instrumentationKey string
@ -68,7 +68,7 @@ resource webApp 'Microsoft.Web/sites@2022-03-01' = {
      linuxFxVersion: linuxFxVersion
      minTlsVersion: '1.2'
      ftpsState: 'FtpsOnly'
-      appCommandLine: preRequsitesCommand
+      appCommandLine: prerequisiteCommand
      appSettings: [
        {
          name: 'redcapAppZip'
@ -127,17 +127,21 @@ resource webApp 'Microsoft.Web/sites@2022-03-01' = {
          value: '1'
        }
        {
-          name: 'storageKey'
+          name: 'StorageKey'
          value: storageAccountKeySecretRef
        }
        {
-          name: 'storageAccount'
+          name: 'StorageAccount'
          value: storageAccountName
        }
        {
-          name: 'storageContainerName'
+          name: 'StorageContainerName'
          value: storageAccountContainerName
        }
+        {
+          name: 'ENABLE_DYNAMIC_INSTALL'
+          value: 'true'
+        }
      ]
    }
  }
--- a/scripts/bash/deploy.sh
+++ b/scripts/bash/deploy.sh
@ -10,6 +10,7 @@
 # Timestamp for log file
 #
 ####################################################################################
+
 stamp=$(date +%Y-%m-%d-%H-%M)

 ####################################################################################
@ -20,8 +21,9 @@ stamp=$(date +%Y-%m-%d-%H-%M)

 echo "Configuring mysqli extension" >> /home/site/log-$stamp.txt
 cd /home/site
-echo "extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqlnd_azure.so
-extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqli.so" >> extensions.ini
+# echo "extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqlnd_azure.so
+# extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/mysqli.so" >> extensions.ini
+echo "extension=/usr/local/lib/php/extensions/no-debug-non-zts-20220829/mysqli.so" >> extensions.ini

 ####################################################################################
 #
@ -117,7 +119,7 @@ echo "session.cookie_secure = On" >> /home/site/redcap.ini

 ####################################################################################
 #
-# Move postbuild.sh to PostDeploymentActions for execution after deployment
+# Copy postbuild.sh to PostDeploymentActions for execution after deployment
 #
 ####################################################################################

@ -126,7 +128,7 @@ cp /home/site/repository/scripts/bash/postbuild.sh /home/site/deployments/tools/

 ####################################################################################
 #
-# Move startup.sh /home for a custom startup
+# Copy startup.sh /home for a custom startup
 #
 ####################################################################################

--- a/scripts/bash/install.sh
+++ b/scripts/bash/install.sh
@ -5,19 +5,22 @@
 #
 # MIT License

+echo -e "\nHello from install.sh"
+
+which mysql
+
 ####################################################################################
 #
-# Initialize the REDCap database using the SQL commands scraped from the 
-# install.php page, then update additional configuration settings including 
+# Update additional configuration settings including 
 # user file uploading settings to Azure Blob Storage
 #
 ####################################################################################
-mysql -u$APPSETTING_DBUserName -h$APPSETTING_DBHostName -p$APPSETTING_DBPassword --ssl=true --ssl-ca=/home/site/wwwroot/DigiCertGlobalRootCA.crt.pem <<EOF
-source /home/install.sql;
+
+/usr/bin/mysql -u$APPSETTING_DBUserName -h$APPSETTING_DBHostName -p$APPSETTING_DBPassword --ssl=true --ssl-ca=/home/site/wwwroot/DigiCertGlobalRootCA.crt.pem <<EOF
 UPDATE $APPSETTING_DBName.redcap_config SET value = 'https://$WEBSITE_HOSTNAME/' WHERE field_name = 'redcap_base_url';
 UPDATE $APPSETTING_DBName.redcap_config SET value = '$APPSETTING_StorageAccount' WHERE field_name = 'azure_app_name';
 UPDATE $APPSETTING_DBName.redcap_config SET value = '$APPSETTING_StorageKey' WHERE field_name = 'azure_app_secret';
 UPDATE $APPSETTING_DBName.redcap_config SET value = '$APPSETTING_StorageContainerName' WHERE field_name = 'azure_container';
 UPDATE $APPSETTING_DBName.redcap_config SET value = '4' WHERE field_name = 'edoc_storage_option';
 REPLACE INTO $APPSETTING_DBName.redcap_config (field_name, value) VALUES ('azure_quickstart', '1');
-EOF
+EOF
--- a/scripts/bash/postbuild.sh
+++ b/scripts/bash/postbuild.sh
@ -5,49 +5,24 @@
 #
 # MIT License

-echo "hello from postbuild.sh"
+echo "Hello from postbuild.sh"

 ####################################################################################
 #
-# Install some utilities for REDCap to work properly
+# Call the install.php file with the option to deploy the database schema.
+# This runs synchronously and will take a few seconds to complete.
 #
 ####################################################################################

+curl -sS https://$WEBSITE_HOSTNAME/install.php?auto=1

+echo -e "\nFinished running install.php"

 ####################################################################################
 #
-# Install Python3 modules used to scrape REDCap installation SQL script
-#
-####################################################################################
-curl -sS https://bootstrap.pypa.io/get-pip.py | python3
-python3 -m pip install beautifulsoup4
-python3 -m pip install requests
-####################################################################################
-#
-# Scrape the install.php page for SQL commands to execute
-#
+# Update additional configuration settings including
+# user file uploading settings to Azure Blob Storage
+# 
 ####################################################################################

-cat << EOF > scraper.py
-import requests
-from bs4 import BeautifulSoup
-page = requests.post("https://$WEBSITE_HOSTNAME/install.php")
-soup = BeautifulSoup(page.content, "html.parser")
-data = soup.find('textarea').text
-with open("/home/install.sql", "w") as out:
-  for i in range(0, len(data)):
-    try:
-      out.write(data[i])
-    except Exception:
-      1+1
-EOF
-python3 scraper.py
-echo "completed running scraper.py with $?"
-####################################################################################
-#
-# Copy the install.sh file to the /home directory
-#
-####################################################################################
-
-cp /home/site/repository/scripts/bash/install.sh /home/install.sh
+bash /home/site/repository/scripts/bash/install.sh
--- a/scripts/bash/startup.sh
+++ b/scripts/bash/startup.sh
@ -1,18 +1,5 @@
 #!/bin/bash
 	
-a2enmod headers
-echo "Header set MyHeader \"%D %t"\" >> /etc/apache2/apache2.conf
-echo "Header always unset \"X-Powered-By\"" >> /etc/apache2/apache2.conf
-echo "Header unset \"X-Powered-By\"" >> /etc/apache2/apache2.conf
-
-####################################################################################
-#
-# Install some utilities for REDCap to work properly
-#
-####################################################################################
-
-apt-get update
-apt-get install -y sendmail cron

 ####################################################################################
 #
@ -22,11 +9,3 @@ apt-get install -y sendmail cron

 echo "* * * * * /usr/local/bin/php /home/site/wwwroot/cron.php > /dev/null" >> /etc/crontab
 service cron start
-
-####################################################################################
-#
-# Start Apache
-#
-####################################################################################
-
-/usr/sbin/apache2ctl -D FOREGROUND