There were a few things missing in the initial attempt at this workflow. I left off the file extension for the CMake cache file, and also forgot to initialize the submodules.
Additionally using make is... slow. Using make `-j` causes the VM to run out of memory. To resolve those issues I'm installing ninja and using ninja instead. I'm also setting `LLVM_USE_LINKER=lld` to reduce linker memory usage and `LLVM_PARALLEL_LINK_JOBS=1` to only run one link at a time.
I've also added `--rerun-failed` and `--output-on-failure` to the ctest invocation so that if it fails we get some hopefully helpful output.