Currently, we publish an artifact for the electron build that includes both the packed and unpacked versions of the product for each platform. This is useful for testing purposes, but we only need the packed version for the actual release process. This PR adds a build step that copies only the files we need to release into a separate directory for use in the release pipeline. The artifact is then published as usual.
Our electron update mechanism relies on a latest.yml metadata file to provide the package file name, version, and an accompanying checksum to validate the downloaded file. Currently, this metadata file is generated alongside the corresponding package file. In the build loop, we then sign that package file. This modifies the package, thus rendering the previously generated checksum invalid. This PR introduces a script to recalculate the checksum and update the latest.yml file accordingly. It then puts the script immediately after the signing steps for mac and windows.
* fix the version
* parameterize the electron build version to download
* Use conditional path in spectron e2e test
* renaming
* rename file
* rename file
* modify the discription
* test
* test env var
* fix typo
* replace for non linux
* use env vars in pipeline
* log err
* move the script
* remove extra script file
* renaming
* move replacing step as a separate template
* rephrase description
#### Description of changes
This updates all the existing E2E tests to use the new insightsUserConfiguration controller in the background page to set high-contrast mode for accessibility tests, rather than using the details page UI to do it. This:
* Cuts 2-3min off e2e runtime
* Removes a bunch of operations that would otherwise be opportunities for flakiness
* Cooperates better with upcoming permissions changes that would otherwise prevent the use of the details view UI without popping the popup UI first
Mechanically, this allows for a bunch of simplification of the existing tests, since they no longer need to spawn separate browser instances/pages in a lot of cases; there was a pattern of having 2 top level describe blocks for "normal mode" vs "high contrast" mode that I've eliminated and replaced with updating a11y tests to just do `it.each` over the 2 HC states.
As part of this, I updated the guidance content tests to work more similar to every other test case, where we do a11y scans in both HC/non-HC modes but only snapshot the components in one of the 2 modes. This is where most of the "30k lines deleted" comes from.
#### Pull request checklist
<!-- If a checklist item is not applicable to this change, write "n/a" in the checkbox -->
- [n/a] Addresses an existing issue: #0000
- [x] Ran `yarn fastpass`
- [n/a] Added/updated relevant unit test(s) (and ran `yarn test`)
- [n/a] Verified code coverage for the changes made. Check coverage report at: `<rootDir>/test-results/unit/coverage`
- [x] PR title *AND* final merge commit title both start with a semantic tag (`fix:`, `chore:`, `feat(feature-name):`, `refactor:`). Check workflow guide at: `<rootDir>/docs/workflow.md`
- [n/a] (UI changes only) Added screenshots/GIFs to description above
- [n/a] (UI changes only) Verified usability with NVDA/JAWS
Updates typescript to the latest stable version.
The notable ancillary changes this required were:
* This update increased the amount of memory used by webpack-all to a tipping point where it began occasionally crashing without finishing. Uses `grunt-concurrent` to have build:all run 3 instances of webpack rather than 1 instance of webpack building 3 configs
* Document.getElementsByTagName was updated to return `HTMLCollectionOf<Element>` instead of `NodeListOf<Element>`; there was a cascading series of type and test updates in response to this
* `MessageEvent`s were generalized to allow events from more than just `Window`s as sources; updated `window-message-handler` with an `e.source as Window`, since it knows it will only register to receive messages from `Window`s.
* Removed a few obsolete `tsconfig.json` entries that were tripping up the new version (in particular, `extendedDiagnostics: true` *actually* causes tsc to emit diagnostics way more verbose than we want in the updated version
To test build reliability with the changes, also introduces a new `build-reliability` pipeline similar to the existing `e2e-reliability` one.
PR #1001 added some electron e2e tests. This PR runs those tests on mac, linux, and windows as a part of the PR/CI builds. Since the linux agents appear to be non-interactive, we have to use xvfb to run tests on them.
Sets up the docker-specific parts of gathering failure screenshots to still execute when the tests fail.
Also updates the more general results publishing task to work consistently with them and not run if the pipeline is cancelled (that's the difference between always() and succeededOrFailed().
We were seeing our Windows hosted Pipelines agents hit "Page crashed!" errors somewhat reliably (~50% of e2e runs). I added functionality to collect detailed chrome logs and crash dumps and collect them as part of our published e2e results, but adding this actually fixed the issue. After some experimentation, I think the key to fixing it was in specifying a custom userDataDir launch parameter to chromium that uses a temporary userDataDir under our test-results folder, rather than deferring to Puppeteer's default behavior of putting it in the system %TEMP% directory. I'm not sure why this fixes the problem; maybe some sort of permissions issue, maybe some sort of overaggressive cleanup of the temp folder while the test is still running.
Validated that 15x runs on Windows with the userDataDir in place never saw the "Page crashed!" issue. Prior to this fix, we see it in about 50% of runs.
Out of scope: other buckets of e2e flakiness issues. In particular, it's expected that there will be some e2e-reliability failures still in the checks section of this PR (but none of them should be "Page crashed!" errors)
We tried updating to the latest agent version to see if it had any impact on reducing page crash rates compared to the older agents. It didn't impact page crash rate, but it improved performance 5-10%, so we'll take it anyway as we continue to test other options for fixing the page crashes.
Since the recent change to run them in linux/windows, the e2e test suite has proven too flaky to use as a PR/CI gate. This change temporarily moves the flaky tests out of the required suite in build.yaml and into a 5x-per-environment run in a separate e2e-reliability.yaml instead, which will be set up to run as a non-required gate only in those PRs/branches working on e2e updates/fixes.
We'll be prioritizing improving the flakiness and getting the runs re-enabled as required checks as part of #867.
Before this change, the windows E2E test logs showed the following warning:
```
warning From Yarn 1.0 onwards, scripts don't require "--" for options to be forwarded. In a future version, any explicit "--" will be forwarded as-is to the scripts.
```
This addresses it in the CI build and also in our README suggested commands (I searched the codebase for uses of the string ` -- `)
Noticed that the new e2e tests were accidentally using the same names for test run titles and failure screenshot artifacts, which made it hard to tell which was which in the pipelines results page. This refactors the common result publishing logic to a common place and makes the names/artifacts be based on job name so they'll be distinct between environments.
* separate out cosmetic code tests and add ubuntu e2e job
* remove code formatting test yaml and change a little bit in drop job
* one solution for avoid drop failure
* change the way i was running builds on different environment
* try to run headful on ubuntu run for build
* push executable path
* push executable path
* add mac and windows support to the build
* undo package.json change
* try changing the ubuntu version to 18.04
* revert to 16.04 ubuntu
* Merge master changes
* try scripting for linux
* try scripting for linux
* change some build structure to test more stuff
* reduce a little bit of parallelization and change windows hosted machine
* reduce a little bit of parallelization and change windows hosted machine
* change the build agent for unit test job for comparison
* change name of mac image - typo
* change name of mac image - typo
* change name of mac image - typo
* update macOS hosted agent version
* try docker
* try docker
* try docker
* Remove --network parameter from docker command
* Remove -t parameter from docker command
* changes to Dockerfile
* add dockerignore to speed up perf for docker steps
* copy test-results from docker back to application agent
* get container id from docker to run copy command
* give container id and docker cp another try
* try and figure out the correct path for the container test-results e2e folder
* stip down docker copy for even better perf
* separate out docker steps into a different yaml to be more clear in our build steps
* copy before yarn dev command
* minor changes in name and sequence of commands
* PR comment on better comments and better explanations
* change some descriptions for jobs
* make changes based on PR comments
* move stuff to yarn
* change build.yaml
* change build to work better
* change name of installation task
* change more stuff from npm to yarn
* make some more simple changes
* run npm upgrade
* remove unncessary changes
* update yarn.lock
* try -frozen-lockfile param
* testing with a minor change to trigger build again
* update readme
* fix formatting
* update documentation
* remove package-lock from prettierignore since the file doesn't exist anymore
* change version and other pr comments
* change occurance of npm
* change workflow.md
* update setup file
* update setting up vscode document
* Add prerequisites yaml file
* Add e2e test yaml file
* format yaml files
* Move copyright headers up
* update comment to clarify why we need to run build:all before running e2e test
Shobhit
John Alkire
Liu Haonan
Mark Reay
Dan Bjorge
AhmedAbdoOrtiga
Karan
Sebastian Morales
Peter Durham