Setup
Issues Addressed
- I am trying to setup Cromwell on Azure for multiple users on the same subscription, what are the ways I can do this?
- I ran the Cromwell on Azure installer and it failed. How can I fix it?
- How can I upgrade my Cromwell on Azure instance?
Setup Cromwell on Azure for multiple users in the same Azure subscription
Cromwell on Azure is designed to be flexible for single and multiple user scenarios. Here we have envisioned four general scenarios and demonstrated how they relate to your Azure account, Azure Batch service, Subscription ID, and Resource Groups, each depicted below.
-
The Individual User: This is the current standard deployment configuration for Cromwell on Azure. No extra steps beyond the deployment guide are necessary.
-
The Lab: This scenario is envisioned for small lab groups and teams sharing a common Azure resource (i.e. a common bioinformatician(s), data scientist(s), or computational biologist(s) collaborating on projects from the same lab). Functionally, this setup does not differ from the "Individual User" configuration. We recommend a single "Cromwell Administrator" perform the initial Cromwell on Azure setup for the group. Ensure that this user has the appropriate role(s) on the Subscription ID as outlined here. Once deployed, this "Cromwell Administrator" can grant "Contributor" access to the created Cromwell storage account via the Azure Portal. This would allow granted users the ability to submit analysis jobs and retrieve results. It would also allow them the ability to view any analysis that has been run by the lab. As Cromwell submits all jobs to Azure Batch as one user, the billing for Cromwell on Azure usage would be collective for the entire lab, not broken down by individual users who submitted the jobs.
-
The Research Group: This scenario is envisioned for larger research groups where a common Azure subscription is shared, but users want/require their own instance of Cromwell on Azure. The initial Cromwell on Azure deployment is done as described in the deployment guide. After the first deployment of Cromwell on Azure is done on the Subscription, subsequent users will need to specify a separate Resource Group AND preexisting Azure Batch account name that is currently being utilized by the pre-existing deployment(s) of Cromwell on Azure. The Azure Batch account must exist in the same region as defined in the "--RegionName" configuration of the new Cromwell on Azure deployment. You can check all the configuration options here. See the invocation of the Linux deployment script for an example:
.\deploy-cromwell-on-azure-linux --SubscriptionId <Your subscription ID> --RegionName <Your region> --MainIdentifierPrefix <Your string> --ResourceGroupName <Your resource group> --BatchAccountName <Your Batch account name>
In this scenario, please note the lack of separation at the Azure Batch account level. While you will be able track resource usage independently due to the separate Cromwell users submitting analyses to Azure Batch (for your own tracking/internal billing purposes), anyone who has access to Azure Batch as a Contributor or Owner will be able to see everyone's Batch pools, and thus what they are running. For this scenario, we would recommend the Cromwell Administrator(s) be trusted personnel, such as your IT team.
-
The Institution: This is an enterprise level deployment scenario for a large organization with multiple Subscriptions and independent user groups within an internal hierarchy. In this scenario, due to the independent nature of the work being done and the desire/need to track specific resource usage (for your own internal billing purposes) you will have completely independent deployments of Cromwell on Azure.
To deploy, you'll need to verify whether an existing Azure Batch account already exists on your Subscription (to run Cromwell on Azure on the Subscription level), or within your Resource Group as described in the deployment guide, with appropriate roles set. If Azure Batch account is not deployed on your Subscription (or if you have available quota to create a new Batch account - the default for most accounts is 1 Batch account/region), then simply follow the deployment guide. If there is an existing Azure Batch account you're connecting to within your Subscription, simply follow the deployment recommendations outlined in [3], adding the appropriate flags for the deployment script. See the invocation of the Linux deployment script for an example:
.\deploy-cromwell-on-azure-linux --SubscriptionId <Your subscription ID> --RegionName <Your region> --MainIdentifierPrefix <Your string> --ResourceGroupName <Your resource group> --BatchAccountName <Your Batch account name>
Please note you can also mix scenarios 1, 2, and 3 within the Azure Enterprise Account in scenario 4.
Debug my Cromwell on Azure installation that ran into an error
When the Cromwell on Azure installer is run, if there are errors, the logs are printed in the terminal. Most errors are related to insufficient permissions to create resources in Azure on your behalf, or intermittent Azure failures. In case of an error, we terminate the installation process and begin deleting all the resources in the Resource Group if already created.
Deleting all the resources in the Resource Group may take a while but as soon as you see logs that the batch account was deleted, you may exit the current process using Ctrl+C or Command+C on terminal/command prompt/PowerShell. The deletion of other Azure resources can continue in the background on Azure. Re-run the installer after fixing any user errors like permissions from the previous try.
If you see an issue that is unrelated to your permissions, and re-trying the installer does not fix it, please file a bug on our GitHub issues.
Upgrade my Cromwell on Azure instance
Starting in version 1.x, for convenience, some configuration files are hosted in your Cromwell on Azure storage account, in the "configuration" container, for example containers-to-mount, cromwell-application.conf, and allowed-vm-sizes. You can modify and save these files using Azure Portal UI "Edit Blob" option or simply upload a new file to replace the existing one. Follow the instructions in the latest release to upgrade your Cromwell on Azure instance.