Daily reference docs for JS (#15749)

* Add dev version to Save-Package-Properties.ps1 and includd in build process

* Now with fewer pipelines

* Add daily docs.ms work to docindex.yml

* Update-DocsMsMetadata and required functions

* Common

* Docs meatadata release and daily docs build

* eq

* Loop over artifacts in paramaters.Artifacts

* Add PackageInfo to path

* download: current

* Output artifacts

* Pool

* Use relative paths in package properties json file and use dev version as Version (if available) when setting metadata

* Skip checking out docs repo if it's already checked out

* Don't overcomplicate docs-metadata-release.yml

* With parameters properly defined and redundant logic removed

* Use TargetBranchName

* Retain original version to differentiate where metadata and readme docs end up

* Add docs repo support for GetDocsMetadata: first cut

* Sparse checkout needs to include metadata

* Add metadata from packages that aren't tracked in the CSV file yet

* Use "dev" if the package has a "dev" version in JS. This will not work in other languages and shouldn't be put into eng/common. That will need to be refactored into logic that could be in Language-Settings.ps1

* Refactor and use sparse checkout everywhere

* Refactor long path support

* Correct path

* Parameter names

* Remove unnecessary parameter

* Close quote

* Add pipeline starting

* Keep the single quotes

* Output link to CI build

* Refactor:
* Rename docs-metadata-release2.yml" to "update-docsms-metadata.yml"
* Documentation for Update-DocsMsMetadata.ps1
* Documentation for Update-DocsMsPackages.ps1
* Add function invocation to Update-DocsMsMetadata.ps1 to override the package metadata from PackageInfo before it's written into the metadata folder
* Refactor logic for generating daily branch name
* Add ability to override daily docs branch name at queue time

* Apply suggestions from code review

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

* Review feedback: Save-Package-Properties.ps1
* Update comments for path normalization
* Change path normalization logic to not switch current location
* Update docs

* Add ability to specify variable name for default branch

* Try convertToJson

* Tab scope

* Do doc artifact info loop in Update-DocsMsMetadata.ps1 instead of yaml template

* Single quotes

* Remove template loop

* Remove extra template references

* Move convertToJson

* Try different formatting given convertToJson

* Reverse Update-DocsMsMetadata.ps1 to operate on a single object. Looping can happen in yaml

* Loop in yaml, Rename Update-DocsMsMetadata parameter

* Try re-formatting yaml

* Spell parameters properly

* Missing backtick

* Remove extra logging

* First cut at default implementation

* Move business logic inside Update-DocsMsMetadata.ps1, PowerShell can handle an array or a single item in the cmdlet parameters

* Use proper function name

* PackageProps has a DevVersion

* No DefaultImplementation in this change. This should be part of a wider effort.

* Straggling master -> main rename

* Use existing devops starting logic

* Remove extra condition

* Fix backslashes

* Correct number of backslashes

* Use variable for branch name

* Update eng/scripts/Language-Settings.ps1

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

* Review feedback

* Do not replace the default branch with the tag in links when doing a Dev build (pointing to the default branch is preferable and the tag does not exist)

* Conditions on docs version for template project

* Do not run PublishDocsToNightlyBranch if TestPipeline is true

* Performance improvements, switch to Linux to take advantage of paths that don't look like escape sequences to ConvertFrom-Json

* Add markdown files to sparse checkout

* Reproduce failure for template publishing

* Remove eng, it's already there by default

* Fix instances where metadata might not be available because of filtering Hide == true (e.g. in template packages). The only information consumed from CSV metadata at this time is the service name which gets normalized and placed in the package overview metadata.

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

eng/pipelines/docindex.yml

@@ -18,6 +18,7 @@ jobs:
           Paths:
             - ci-configs/packages-latest.json
             - ci-configs/packages-preview.json
+            - metadata/
           Repositories:
             - Name: $(DocRepoOwner)/$(DocRepoName)
               WorkingDirectory: $(DocRepoLocation)
@@ -43,3 +44,45 @@ jobs:
           TargetRepoName: $(DocRepoName)
           TargetRepoOwner: $(DocRepoOwner)
           WorkingDirectory: $(DocRepoLocation)
+
+      # Prepare daily docs CI
+      - template: /eng/common/pipelines/templates/steps/set-daily-docs-branch-name.yml
+        parameters:
+          DailyBranchVariableName: DailyDocsBranchName
+      - pwsh: |
+          $ErrorActionPreference = "Continue"
+          git checkout "origin/$(DailyDocsBranchName)" 2>&1 | Out-Null
+          $LASTEXITCODE = 0 # This ignores any error from git checkout
+          git status
+        displayName: Checkout daily branch if it exists
+        workingDirectory: $(DocRepoLocation)
+      - task: Powershell@2
+        inputs:
+          pwsh: true
+          filePath: eng/common/scripts/Update-DocsMsPackages.ps1
+          arguments: -DocRepoLocation $(DocRepoLocation)
+        displayName: Update Docs Onboarding for Daily branch
+      - template: /eng/common/pipelines/templates/steps/git-push-changes.yml
+        parameters:
+          BaseRepoBranch: $(DailyDocsBranchName)
+          BaseRepoOwner: $(DocRepoOwner)
+          CommitMsg: "Update targeting packages based on release metadata. (Daily docs)"
+          TargetRepoName: $(DocRepoName)
+          TargetRepoOwner: $(DocRepoOwner)
+          WorkingDirectory: $(DocRepoLocation)
+          ScriptDirectory: $(Build.SourcesDirectory)/eng/common/scripts
+
+      - task: PowerShell@2
+        displayName: Queue Docs CI build
+        inputs:
+          pwsh: true
+          filePath: eng/common/scripts/Queue-Pipeline.ps1
+          # SourceBranch must be "master" as this repo is managed by the docs 
+          # team and will be updated according to their timeline.
+          arguments: >
+            -Organization "apidrop"
+            -Project "Content%20CI"
+            -SourceBranch "master"
+            -DefinitionId 3452
+            -Base64EncodedAuthToken "$(azuresdk-apidrop-devops-queue-build-pat)"
+            -BuildParametersJson '{"params":"{ \"target_repo\": { \"url\": \"https://github.com/MicrosoftDocs/azure-docs-sdk-node\", \"branch\": \"$(DailyDocsBranchName)\", \"folder\": \"./\" }, \"source_repos\": [] }"}'

eng/pipelines/templates/stages/archetype-js-release.yml

@@ -108,40 +108,31 @@ stages:
               dependsOn: PublishPackage
 
               pool:
-                name: azsdk-pool-mms-win-2019-general
-                vmImage: MMS2019
-              variables:
-                CIConfigs: "{'targets':[{'path_to_config':'ci-configs/packages-preview.json','Mode':'Preview','content_folder':'docs-ref-services/preview'},{'path_to_config':'ci-configs/packages-latest.json','mode':'Latest','content_folder':'docs-ref-services/latest'}]}"
+                name: azsdk-pool-mms-ubuntu-2004-general
+                vmImage: MMSUbuntu20.04
 
               strategy:
                 runOnce:
                   deploy:
                     steps:
-                      - checkout: self
-                      - pwsh: |
-                          Get-ChildItem -Recurse ${{parameters.ArtifactName}}/${{artifact.name}}
-                        workingDirectory: $(Pipeline.Workspace)
-                        displayName: Output Visible Artifacts
-                      - template: /eng/common/pipelines/templates/steps/get-pr-owners.yml
+                      - template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
                         parameters:
-                          TargetVariable: "OwningGHUser"
-                          ServiceDirectory: ${{parameters.ServiceDirectory}}
-                      - template: /eng/common/pipelines/templates/steps/docs-metadata-release.yml
+                          SkipDefaultCheckout: true 
+                          Paths:
+                            - sdk/**/*.md
+                      - download: current
+                      - template: /eng/common/pipelines/templates/steps/update-docsms-metadata.yml
                         parameters:
-                          ArtifactLocation: $(Pipeline.Workspace)/${{parameters.ArtifactName}}/${{artifact.name}}
-                          PackageRepository: NPM
-                          ReleaseSha: $(Build.SourceVersion)
+                          PackageInfoLocations: 
+                            - $(Pipeline.Workspace)/${{parameters.ArtifactName}}/PackageInfo/${{artifact.name}}.json
                           RepoId: Azure/azure-sdk-for-js
                           WorkingDirectory: $(System.DefaultWorkingDirectory)
                           TargetDocRepoOwner: ${{parameters.TargetDocRepoOwner}}
                           TargetDocRepoName: ${{parameters.TargetDocRepoName}}
-                          PRBranchName: onboarding-${{artifact.name}}-$(Build.BuildId)
-                          ArtifactName: ${{artifact.name}}
                           Language: 'javascript'
-                          DocRepoDestinationPath: 'docs-ref-services/'
-                          GHReviewersVariable: 'OwningGHUser'
-                          CIConfigs: $(CIConfigs)
-                          CloseAfterOpenForTesting: '${{ parameters.TestPipeline }}'
+                          SparseCheckoutPaths:
+                            - docs-ref-services/
+                            - metadata/
 
           - ${{if ne(artifact.skipPublishDocGithubIo, 'true')}}:
             - deployment: PublishDocsGitHubIO
@@ -261,3 +252,34 @@ stages:
                 targetType: filePath
                 filePath: "eng/tools/publish-to-npm.ps1"
                 arguments: '-pathToArtifacts $(Package.Archive) -accessLevel "public" -tag "dev" -registry "$(Registry)" -npmToken "$(NpmToken)"'
+
+    - job: PublishDocsToNightlyBranch
+      condition: or(eq(variables['SetDevVersion'], 'true'), and(eq(variables['SetDevVersion'], ''), eq(variables['Build.Reason'],'Schedule'), eq(variables['System.TeamProject'], 'internal')))
+      pool:
+        name: azsdk-pool-mms-ubuntu-2004-general
+        vmImage: MMSUbuntu20.04
+      steps:
+        - template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
+          parameters:
+            SkipDefaultCheckout: true 
+            Paths:
+              - sdk/**/*.md
+        - download: current
+        - pwsh: |
+            Get-ChildItem -Recurse $(Pipeline.Workspace)/${{parameters.ArtifactName}}/
+          displayName: Show visible artifacts
+
+        - template: /eng/common/pipelines/templates/steps/update-docsms-metadata.yml
+          parameters:
+            PackageInfoLocations:
+              - ${{ each artifact in parameters.Artifacts }}:
+                - $(Pipeline.Workspace)/${{parameters.ArtifactName}}/PackageInfo/${{artifact.name}}.json
+            RepoId: Azure/azure-sdk-for-js
+            WorkingDirectory: $(System.DefaultWorkingDirectory)
+            TargetDocRepoOwner: ${{parameters.TargetDocRepoOwner}}
+            TargetDocRepoName: ${{parameters.TargetDocRepoName}}
+            Language: 'javascript'
+            DailyDocsBuild: true
+            SparseCheckoutPaths:
+              - docs-ref-services/
+              - metadata/

eng/pipelines/templates/steps/build.yml

@@ -28,6 +28,18 @@ steps:
     condition: and(succeeded(),eq(variables['SetDevVersion'],'true'))
     displayName: "Update package versions for dev build"
 
+  - task: Powershell@2
+    inputs:
+      filePath: $(Build.SourcesDirectory)/eng/common/scripts/Save-Package-Properties.ps1
+      arguments: >
+        -ServiceDirectory ${{parameters.ServiceDirectory}}
+        -OutDirectory $(Build.ArtifactStagingDirectory)/PackageInfo
+        -AddDevVersion
+      pwsh: true
+      workingDirectory: $(Pipeline.Workspace)
+    displayName: Update package properties with dev version
+    condition: and(succeeded(),eq(variables['SetDevVersion'],'true'))
+
   - script: |
       node common/scripts/install-run-rush.js install
     condition: and(succeeded(),ne(variables['SetDevVersion'],'true'))

eng/scripts/Language-Settings.ps1

@@ -122,6 +122,24 @@ function Get-javascript-PackageInfoFromPackageFile ($pkg, $workingDirectory)
   return $resultObj
 }
 
+function Get-javascript-DocsMsMetadataForPackage($PackageInfo) { 
+  New-Object PSObject -Property @{ 
+    DocsMsReadMeName = $PackageInfo.Name -replace "^@azure/" , ""
+    LatestReadMeLocation = 'docs-ref-services/latest'
+    PreviewReadMeLocation = 'docs-ref-services/preview'
+    Suffix = ''
+  }
+}
+
+# In the case of NPM packages, the "dev version" produced for the given build
+# may not have been published if the code is identical to the code already 
+# published at the "dev" tag. To prevent using a version which does not exist in 
+# NPM, use the "dev" tag instead.
+function Get-javascript-DocsMsDevLanguageSpecificPackageInfo($packageInfo) {
+  $packageInfo.Version = 'dev'
+  return $packageInfo
+}
+
 # Stage and Upload Docs to blob Storage
 function Publish-javascript-GithubIODocs ($DocLocation, $PublicArtifactLocation)
 {