Update to deployment and docs generation: adjustments to deploy v2
artifact instead of main artifacts. Note: while the update aims to
deploy artifacts from the latest release branches instead of main, the
actual website build is still done on the main branch. This will prevent
the pipeline from deploying docs which were generated with an outdated
version of the website.
Notable changes:
deploy-docs:
- shifted most of the stages from build-docs to deploy-docs
- build-docs now runs check_branch_version -> upload-json -> trigger
deploy
build-docs:
- shifted check_branch_version to the top since its output is required
to properly in the upload json step
upload-json-step:
- added a new upload step which uploads artifacts with the name
"latest-v*.tar.gz"
- takes in new parameters from check_branch_version to ensure it only
uploads with this name when the pipeline is running on the latest
release branch
latestVersions:
- since upload-json-step requires majorVersion, I figured it was best to
include it as an output variable for the function as it seems more
robust compared to relying on a string parsing script in the pipeline
itself
download-apis:
- update such that it only downloads "latest-v*.tar.gz" instead of the
main branch artifacts
# Docusaurus
Updates the website (/docs) to leverage
[Docusaurus](https://docusaurus.io/) in place of Hugo as its static site
generator.
## Functional changes
### Versioning
The existing Hugo-based site only has a partial versioning story.
The API docs are versioned, but the rest of the content isn't.
This creates a messy story where our hand-written docs likely only
discuss topics related to the current version, and we have no place to
put docs discussing earlier versions.
Or, even worse, we have a mixed bag of documentation for different
versions, creating a very unclear user story.
This prototype includes an end-to-end versioning story, [automated by
Docusaurus](https://docusaurus.io/docs/versioning).
Current (v2) docs live under `docs`.
Old (v1) docs live under `versioned_docs/version-1`.
Most of the documentation has been duplicated between the two versions,
but some minor changes have been made to make the docs better line up
with the corresponding version of the API.
These changes should be reviewed before being merged into main /
deploying the new website.
### Search
This branch includes an offline implementation of search.
An offline solution comes with some downsides (slower build, larger
bundle), and probably isn't what we want long term.
That said, it is much better than what our current website has (no
search whatsoever).
We should come back to this after v1 of our new website.
---------
Co-authored-by: Wayne Ferrao <wayneferrao@microsoft.com>
## Description
We're moving our agent pools to the eastus2 region as part of a capacity
management initiative. This PR updates all pipelines to use the new
pools.
For various call signature API items, we frequently IDs like
`_call_-call`. There are a number of reasons for this, but mostly it
boils down to trying to avoid creating artificial name conflicts while
also ensuring generated files don't contain invalid characters. When we
generate heading IDs from these, we previously weren't escaping the
contents, which was invalid. Hugo previously handled this in most cases,
but Docusaurus (the tooling we're working to adopt in place of Hugo for
generating our website) does not.
This PR updates heading rendering to ensure ID contents are escaped for
Markdown.
[AB#23444](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/23444)
## Description
Fixes a "glob" pattern (ADO-specific syntax) so the Performance
Benchmarks pipeline can differentiate the `tgz` files for packages that
have partially overlapping names (e.g. `@fluid-experimental/tree` and
`@fluid-experimental/tree-react-api`).
Follow-up to https://github.com/microsoft/FluidFramework/pull/22995.
Current pattern ?.?.? doesn't support multi digit versions. Bump to
client 2.10.0 has caused the "Install package with perf tests" step to
be failing in the pipeline.
previous shouldDeploy condition have not been included in deployment
condition. This variable has been updated to "forceDeployment" and
included back in the deployment condition.
shouldDeploy has been updated to check if release builds are the latest
version in its major version series.
## Description
Adjusts the `condition` parameter of
`include-conditionally-run-stress-test` to apply to the entire stage
rather than the test-running job. This improves the clarity from the
pipeline overview page, see most recent run vs previous ones:
This also fixes the `stressMode` parameter not being given options at
queue time, which was introduced in #21702 and prevented the pipeline
from being queued.
---------
Co-authored-by: Abram Sanderson <absander@microsoft.com>
Updates the configuration for build-tools packages to use fluid-build to
build.
The changes were modest since it was already using fluid-build for most
of the build. Only the root-level tasks needed to be hooked up, and the
individual package scripts needed to be updated.
[AB#17062](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/17062)
## Description
Cleanup references to markdown-magic-template from lockfiles.
These projects have a `file:` reference to `@fluid-tools/markdown-magic`
in their package json. The only way I could find to clean up the
lockfile entries corresponding to the dependency tree for the dependency
that we removed in `@fluid-tools/markdown-magic` was to manually delete
that line from the lockfile in these packages and run `pnpm i
--no-frozen-lockfile` after.
## Description
`markdown-magic-template` is not used at all as far as I can tell, and
it brings in a transitive dependency on `lodash.template: 4.5.0` which
is flagged with a CVE that can't be addressed easily because `lodash`
seems to have stopped publishing each of their functions as a separate
package after 4.5.0.
## Description
Modify codecoverageAnalysis name to include job attempt number. This is
because the reupload attempts fails, if the folder already exists.
Co-authored-by: Jatin Garg <jatingarg@Jatins-MacBook-Pro-2.local>
## Description
Updates the `get-func-name` transitive dependency to address
[CVE-2023-43646](https://nvd.nist.gov/vuln/detail/CVE-2023-43646). It
came through `chai@4.3.7`; updating to `chai@4.5.0` was enough to get a
fixed version of `get-func-name`. Also did a few updates of transitive
dependency `loupe` from 2.3.6 to 2.3.7 (even though `chai` doesn't
strictly needed it), where they guarantee that their dependency ranges
now point to fixed versions of their dependencies.
## Description
- It's
[documented](https://learn.microsoft.com/en-us/azure/devops/pipelines/troubleshooting/troubleshooting?view=azure-devops#variables-having--single-quote-appended)
that ADO logging commands (`echo ##vso[<something>]`) don't interact
well with `set -x`, so undoing that part of
https://github.com/microsoft/FluidFramework/pull/22659 because we use
them in several places. I could have wrapped each use in `set +x;
<command>; set -x` but I think that would warrant a comment every time.
We have lived forever without having each command in a script printed
back to us, so I think it's fine to keep that behavior.
- Adds steps to install the correct node version in the job to upload
dev manifests. It wasn't using it, and since the build agent image made
node 18 available by default (by coincidence?) when installing the
several versions that we install in our image, everything worked. But in
the latest version of the image, re-generated automatically, node 16
seems to be the default (unsure why, separate topic I'm looking into),
and that uncovered this missing step.
The second bullet point is highlighted with a comment below. Everything
else in the PR is about the first one.
Disabling code coverage in internal pipelines to save time since we
don't use this data currently. (We can re-enable this if we end up
needing it later.) I made this change per Alex's suggestion here:
https://github.com/microsoft/FluidFramework/pull/22762/files#r1804966589.
These are the times for the Build stage in my test runs (9m 12s saved on
average):
1. With code coverage: 53m 41s, 58m 3s, 52m 8s, 50m 37s
2. Without code coverage: 46m 35s, 43m 42s, 44m 6s, 43m 16s
See runs for test/nocc branch (ignoring first one since CodeQL Finalize
step takes super long the first time in a branch).
## Description
While we should avoid inline scripts in `Bash@3` tasks in YML pipelines
as much as possible, wherever we have them, we should make them robust.
Adding `set -eux -o pipefail` to prevent common issues (like a
multi-command script failing in the middle but not causing the task to
fail because some other command after it succeeded) seems like a net
good.
Github copilot was pretty helpful to understand exactly what the line
does:
> - set -e: Exit immediately if a command exits with a non-zero status.
> - set -u: Treat unset variables as an error and exit immediately.
> - set -x: Print commands and their arguments as they are executed.
> - -o pipefail: The return value of a pipeline is the status of the
last command to exit with a non-zero status, or zero if no command
exited with a non-zero status.
>
> These options are often used in scripts to make them more robust and
easier to debug.
Motivated by [this pipeline
run](https://dev.azure.com/fluidframework/internal/_build/results?buildId=296239&view=logs&j=39ff69ce-e37a-5454-6dd7-938cc0c22dc6&t=b4f667a1-f59a-5f88-c7c4-0c8c8ea4419c)
(msft internal) where the reported error occurred way after the first
things started to fail but their tasks didn't.
Bumped client from 2.4.0 to 2.5.0.
Some new packages in the client had `workspace:^` deps that were
corrected to use tilde, so the lockfile had to be updated.
Command used:
```shell
flub release -g client -v
```
## Description
While looking at our pnpm version, I found two packages inside of client
other than the root specifying the pnpm version. This seems unnecessary.
This removes them, as well as the apparently unneeded workspace and
pnpm-lock from tools/markdown-magic.
## Description
### Changes Made
Validation Improvement: Updated the validation logic to use
Number.isNaN(Number.parseFloat(value)) instead of isNaN. This ensures
that any non-number data is correctly identified and handled.
### Error Handling
For app insights handlers, retained the try-catch mechanism to continue
processing subsequent metrics even if an error occurs.
_For non-app-insight handlers, implemented a strategy to throw an error
and terminate the process when non-number data is encountered. This
provides a clear indication of unexpected data and prevents the
submission of potentially invalid values._
### Unit Tests
Added unit tests to cover scenarios involving non-number data in
customData. These tests ensure that the new validation logic works as
expected and helps prevent future regressions.
## Test result
~~~
@chentong7 ➜ /workspaces/FluidFramework/tools/telemetry-generator
(telemetry) $ npm run test
> @fluid-tools/telemetry-generator@1.0.0 test
> mocha --recursive "dist/test/**/*.test.*js" --exit
Writing test results relative to package to nyc/junit-report.xml and
nyc/junit-report.json
appInsightsCustomBenchmarkHandler
✔ should emit metrics to Azure App Insights
appInsightsExecutionTimeTestHandler
✔ should emit valid metrics to Azure App Insights
✔ should skip logging for invalid metrics
appInsightsMemoryUsageTestHandler
✔ should emit heap used avg and std dev metrics to Azure App Insights
✔ should skip metrics if values are not numbers
executionTimeTestHandler
✔ should emit execution time avg and std dev metrics to logger
memoryUsageTestHandler
✔ should emit heap used avg and std dev metrics to logger
✔ should throw an error for invalid heap used avg metric
✔ should throw an error for invalid heap used std dev metric
9 passing (12ms)
~~~
---------
Co-authored-by: Alex Villarreal <716334+alexvy86@users.noreply.github.com>
Due to some bad pull/rebase, the PR
https://github.com/microsoft/FluidFramework/pull/21702 introduced
content that shouldn't exist in the main branch (based on the commit
c126bba051 (diff-5839274c7eb7f262c570edf94a75999ccf76913c6d7b9ec3d53152537a00b0cb)).
This PR is to address this issue.
## Reviewer Guidance
The review process is outlined on [this wiki
page](https://github.com/microsoft/FluidFramework/wiki/PR-Guidelines#guidelines).
> List any specific things you want to get reviewer opinions on, and
anything a reviewer would need to know to review this PR effectively.
> Things you might want to include:
>
> - Questions about how to properly make automated tests for your
changes.
> - Questions about design choices you made.
> - Descriptions of how to manually test the changes (and how much of
that you have done).
> - etc.
>
> If you have any questions in this section, consider making the PR a
draft until all questions have been resolved.
## Description
Updates transitive dependencies on `braces` from 3.0.2 to 3.0.3 to
address [CVE-2024-4068](https://nvd.nist.gov/vuln/detail/CVE-2024-4068).
A couple of applications of `flub modify lockfile --dependency braces
--version 3.0.3 --releaseGroup <release group>`, and some manual updates
in packages/release groups that we can't target with `flub`, basically
doing the same thing but manually (add an override in package.json,
install dependencies, remove override, install dependencies again to
clean up override from the lockfile).
In a few cases I got unrelated updates, mostly about node types, which I
reverted manually.
Server packages also got semver update from 7.6.0 to 7.6.3 which seems
fine.
[AB#2729](https://dev.azure.com/fluidframework/internal/_workitems/edit/2729),
[AB#2730](https://dev.azure.com/fluidframework/internal/_workitems/edit/2730)
This PR is a prototype for improving the Test - DDS Stress line by using
a numeric stressLevel instead of a boolean isStress. This change allows
for more granular control over whether to use stress runs and to what
degree.
The follow-up tasks include:
1. Integrating this functionality (potentially as a stage) into the CI
build pipeline with a light stress level mode, so it can be triggered in
the PR gateway.
2. Creating a long-running pipeline to execute the DDS stress tests with
a heavy stress level mode, scheduled to run periodically during off-peak
hours, similar to the test stability pipeline.
[Guidelines for Pull
Requests](https://github.com/microsoft/FluidFramework/wiki/PR-Guidelines#guidelines).
## Reviewer Guidance
The review process is outlined on [this wiki
page](https://github.com/microsoft/FluidFramework/wiki/PR-Guidelines#guidelines).
> List any specific things you want to get reviewer opinions on, and
anything a reviewer would need to know to review this PR effectively.
> Things you might want to include:
>
> - Questions about how to properly make automated tests for your
changes.
> - Questions about design choices you made.
> - Descriptions of how to manually test the changes (and how much of
that you have done).
> - etc.
>
> If you have any questions in this section, consider making the PR a
draft until all questions have been resolved.
---------
Co-authored-by: Abram Sanderson <Abram.sanderson@gmail.com>
The mocha-json-output-reporter is failing which I was able to determine
via trial and error by adjusting the reporters, it also appears to be
essentially unsupported. It seems we only use it for inline logs in our
ci build output, which shouldn't be necessary as we have the test
results separately in the test tab. I think the best solution is to just
remove this.
Note: requires changes from following PR
https://github.com/microsoft/FluidFramework/pull/22252
Updated build-docs to trigger on release branches.
Now includes a stage before deployment to check if triggering branch
version matches a latest minor version of a major release.
We handle such input gracefully, so it really isn't an error. A warning
is still appropriate, however, to ensure the caller is aware of the
situation.
Also updates the logged notice to include the tag that was encountered.
Removes A LOT of bespoke HTML rendering logic, replacing it with
[hast-util-to-html](https://www.npmjs.com/package/hast-util-to-html) and
[hast-util-format](https://www.npmjs.com/package/hast-util-format) to do
the rendering on our behalf.
Note that there are A LOT of test asset diffs here. The rendering
libraries mentioned above make different styling choices than we made in
our bespoke rendering. In most cases these changes are trivial, or
arguably readability improvements. Regardless, this PR makes the
formatting no longer a direct responsibility of this library, which is a
win IMO.
The Documentation domain supports hierarchical text formatting
overrides. This includes setting formatting at a position in the tree,
and disabling it at a lower level in the tree.
HTML does not support this kind of hierarchical format disabling. The
way we were transforming Documentation domain input to HTML ignored
this, and was fundamentally broken. This change fixes the issue by
tracking text formatting context while transforming the tree, and
injecting formatting tags only at the lowest levels.
See updated tests for examples of the changes.
Support for embedded HTML contents in TSDoc comments has been removed.
The TSDoc parser has some half-baked support for preserving HTML tags in
its output, despite the TSDoc spec making no claims about supporting
embedded HTML. But it does so in a structure that is difficult to handle
correctly, assuming that the output language can support arbitrary HTML
contents, and that it is safe to output the contents raw and
unsanitized.
As a result, this library's support for such contents was similarly
half-baked, and difficult to maintain.
VSCode Intellisense, as a comparison, chooses to completely ignore HTML
tags, and simply render the inner contents ignoring any HTML decorators.
This library has adopted the same policy.
### Breaking Change
If you depended on HTML content preservation, this change will affect
your output.
## Description
- Remove a `workingDir` node at the incorrect indentation level. The
script itself checks if that directory exists before `cd`ing into it if
applicable, so having it run in it from the start doesn't seem necessary
(and might even fail if it doesn't exist?)
- Replace a few `tag` strings with the more explicit `Tag image` in
`Docker@1` tasks, as listed [in the
docs](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/docker-v1?view=azure-pipelines).
`tag` works as an alias, but the explicit version helps so VSCode
doesn't report problems/unexpected values in yml files.
Allow users to invoke `toHtml` functions with either a partial config or
a full transform context. Also promotes `toHtml` APIs to `@public` so we
can use them in the website build.
Allows updating custom table renderer in `fluidframework.com`'s API docs
build to leverage the transformation and
[hast-util-to-html](https://www.npmjs.com/package/hast-util-to-html) to
simplify the logic as follows:
[AB#8866](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/8866)
The above bug seems to stem from our "pretty format" HTML rendering. A
cursory investigation didn't yield useful context on exactly what was
going wrong in that code. But the idea has been for quite a while to
eventually remove that code and replace it with the HAST-based `toHtml`
transformation logic anyways. I have confirmed that switching to this,
and using an existing open-source `hast-util-to-html` HAST renderer
resolves the issue.
dependabot[bot]
Michael Zhen
Craig Macomber (Microsoft)
Joshua Smithrud
Alex Villarreal
Jatin Garg
Abram Sanderson
jzaffiro
Tyler Butler
Scarlett Lee
Tong Chen
clarenceli-msft
Tony Murphy