Sub-workflow spec proposal

Problem statement

It becomes difficult very quickly to organize a large workflow into stages with child templates. Specific problems:

The syntax for including an indented workflow fragment is complicated

    {{# indent depth=4 }}
    {{> child-template-name.yaml }}
    {{/ indent }}

The same working memory is used for the whole workflow, making it possible to consume values or produce output unexpectedly
Troubleshooting syntax errors becomes more difficult because the overall result becomes a single, large workflow file.
Individual stages can't be run directly
Workflow fragments often have re-usable logic. You end up with a situation where the same fragment file is copied into several locations, leaving you with the problem of keeping the

Solution proposal

Most package-based systems have a way of cross-referencing packages which contain re-usable units of work. This approach is similar, and in particular is inspired by how helm's sub-chart feature is implemented.

Declare references

Similar to how swagger reference details are declared in advance, workflow references will be declared in readme.md configuration blocks.

workflows:
  {workflow-set-name}:
    target: {workflow-target-prefix}
    source: {workflow-base-path}
    inputs:
    - {workflow-sub-path}
    - {workflow-sub-path}

The {workflow-set-name} is just a way to organize references. It doesn't cause any mechanical differences in how sub-workflows are referenced.

The {workflow-base-path} is an optional, absolute-or-relative, file-or-http(s) path to the location where this group of sub-workflows are loaded from. The default base path is the location where the current workflow has been loaded from, which means if the sub-workflows are located side-by-side with the parent workflow then the source: property does not need to be specified.

The {workflow-sub-path} is the folder path which contains the referenced workflow. In the following example folder structure:

./
  my-workflows/
    main-workflow/
      readme.md
      workflow.yaml
    step1/
      workflow.yaml
    step2/
      workflow.yaml

The ./my-workflows/main-workflow/readme.md could reference both sub-workflows with the following:

workflows:
  the-steps:
    inputs: [step1, step2]

The {workflow-target-prefix} is an optional relative path to include before the

Invoking workflows

A new operation type is introduced by this feature. An operation is a sub-workflow operation when the property workflow is present. This works the same as the request and template properties.

operations:
- message: Executing step 1
  workflow: step1
  values:
    foo: ( jmespath-for-foo )
    bar: ( jmespath-for-bar )
  output:
    quux: ( jmespath-for-quux )

The values json data essentially acts as the same as a values.yaml file for executing the sub-workflow. Note that only the values declared in the operation are carried forward into the child workflow. That is different from the other operations, where the declared values data is merged along with the rest of the working memory for the scope of the operation. That is done to ensure that a child workflow does not have unintended consequences if its working-memory assumptions conflict with the various workflows which invoke them.

The output json data works exactly as it does in a request or template operation. Any json data which the child workflow produced with its output statements become part of a result.xxx in working memory, and by putting jmespath expressions in the workflow operation then the information returned from the child workflow can be placed in working memory at a location under the main workflow's control.

The workflow property path is used to create a virtual sub-path. All of the files contained in a referenced sub-workflow appear to be files in a known sub-directory. So in this example the main workflow is executing the child workflow with the path workflows/step1/workflow.yaml.

Prefixing with the target property

If the reference above was also given a target prefix, then that value would appear in the virtual relative path and would also be used in the workflow.

readme.md

workflows:
  the-steps:
    target: shared-steps
    inputs: [step1, step2]

workflow.yaml

operations:
- workflow: shared-steps/step1

Child workflows in source control

GitHub urls can be used as source reference for child-workflows. This is intended to be used to create common libraries of reusable compound operations, e.g. deploying an arm template, upserting an aad application, or configuring a devops service connection.

In this case it would be a best practice to use a source reference with an exact sha value, to prevent the updates to shared workflows from breaking existing dependent workflows.

readme.md

workflows:
  common-library:
    source: https://github.com/Microsoft/Atlas/tree/c1f8234b6f1bc9eea70e0aad251ba873b1054f1d/library
    inputs: 
    - arm-deployment