Merged PR 678857: [DeveloperGuide] List Linux prerequisites

- update how to run tests on Linux
- add some docs about Linux sandbox
- add instructions from creating and prepping a Linux VM (copied from OneNote)

Documentation/Specs/Sandboxing.md

@@ -21,6 +21,16 @@ In terms of performance, this implementation adds 1-5% of time overhead to runni
 
 Technical note: The top-level process initiating Detours calls must be a 64-bit process. Detours bootstrapping code is hard-coded to start from 64-bit, matching the requirements for large memory needs for the BuildXL engine for parsing and tracking large repos.
 
+## Linux Sandboxing
+
+Akin to Detouring on Windows, the [dynamic loader](https://www.man7.org/linux/man-pages/man8/ld.so.8.html) allows for *library preloading* (a.k.a. function interposing) to hook various system calls.  Just like on Windows, the BuildXL sandbox leverages this feature to intercept all relevant filesystem-related system calls from the standard C library (`libc`) and handle them appropriately (e.g., report all requested accesses to the BuildXL process, block disallowed accesses, etc.).
+
+For a full list of all interposed system calls, see [syscalls.md](/Public/Src/Sandbox/Linux/syscalls.md).
+
+The semantics of how various high-level filesystem operations (e.g., absent probes, directory enumerations, reads, writes, etc.) are handled is expected to be the same on all supported operating systems.
+
+A clear **limitation** of this approach is that in only applies to executables that are **dynamically linked** to `libc`.  In Linux, that is the case for the vast majority of executables.  Notable exceptions, however, are programs written in [`Go`](https://go.dev/) which are by default statically linked.
+
 ## macOS Sandboxing
 Interposing system calls (akin to Detouring on Windows) is possible on macOS, but comes with a major restriction: it is not applicable to "protected" system processes.  Another drawback of this approach is making sure that all relevant system calls are interposed, the list of which may be huge and not readily available.  Our sandbox for macOS avoids those restrictions by being implemented as a Darwin kernel extension, producing similar data and blocking capabilities as noted above for Windows.
 

Documentation/Wiki/DeveloperGuide.md

@@ -3,6 +3,12 @@
 * Windows 10 is the minimum requirement for BuildXL. You do not need to install [Visual Studio](https://visualstudio.microsoft.com/vs/) to get a working build, but it can be very helpful and is recommended for Windows development.
 * You will also need to install the Windows development kit. When you build the repo, the build script will determine if you have a compatible version installed and provide an error message with a link if one needs to be installed
 * [Visual Studio 2019 Build Tools](https://visualstudio.microsoft.com/downloads/) build tools must be installed. Scroll down to the "Tools for Visual Studio 2019" section, download and run the installer for "Build Tools for Visual Studio 2019". Within the Visual Studio installer under "Individual Components", search for and install "MSVC (v142) - VS 2019 C++ x64/x86 Spectre-mitigated libs (v14.29-16.10)".
+
+
+## Linux
+See [Prepare Linux VM](/Documentation/Wiki/LinuxDevelopment/How_to_prep_VM.md)
+
+
 ## macOS
 To run BuildXL on macOS you need to install:
 

Documentation/Wiki/LinuxDevelopment/How_to_create_HyperV_VM.md

@@ -0,0 +1,16 @@
+# How to Set Up a Linux VM using HyperV
+
+* `Hyper-V Settings` -> Make sure that the locations where to store VMs and VHDs are pointing to drive with plenty of free space. SSD is strongly advised. 
+* `Virtual Switch Manager` -> Create a new External switch and call it 'External' 
+* `Quick Create` -> Select Ubuntu 20.04.x or newer (distros other than Ubuntu could also work, but the [VM prep steps](./How_to_prep_VM.md) might be different)
+* Once the VM is created, but before connecting to it, open its settings: 
+* Update memory, processor count, select 'External' Network Adapter. 
+* Uncheck 'Use automatic checkpoints' 
+* `Action` -> `Edit Disk` -> `Select downloaded VHD` -> `Expand` -> Set to 200GB at minimum 
+* The previous step can be repeated if more disk space is needed. 
+* Do not select the "Log me in automatically" option during set up as this will break enhanced session. 
+* Follow the steps below to update the volume size: 
+    * sudo apt install gparted && sudo gparted
+    * In the GUI, select the current drive at the top by clicking on it and then click on the expand button ("->"). Assign the entire Total space to the current drive. 
+    * Then click on the green check mark to expand the drive. 
+    * Then run sudo reboot to restart the VM for the changes to take effect. 

Documentation/Wiki/LinuxDevelopment/How_to_prep_VM.md

@@ -0,0 +1,20 @@
+# How to prepare a fresh Linux VM
+
+## Install Prerequisites
+```bash
+# install packages
+sudo apt-get update 
+sudo apt-get install –y git build-essential mono-devel mono-complete libc6-dev openssh-server curl 
+
+# install .NET
+wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
+sudo dpkg -i packages-microsoft-prod.deb
+rm packages-microsoft-prod.deb
+sudo apt-get install -y dotnet-sdk-6.0 
+```
+
+## Install/configure git credentials provider
+
+* follow https://github.com/GitCredentialManager/git-credential-manager#linux
+* run `git config --global credential.credentialStore secretservice` (might require GUI)
+* run `git config --global credential.helper cache && git config --global credential.helper 'cache --timeout=600000`

Documentation/Wiki/LinuxDevelopment/How_to_test.md

@@ -1,27 +1,46 @@
-# How to simply test *libDetours.so*
-* `cd Public/Src/Sandbox/Linux`
-* `make`
-* `echo > libDetours.log`
-* `env __BUILDXL_LOG_PATH=$(pwd)/libDetours.log LD_PRELOAD=$(pwd)/bin/debug/libDetours.so ls -l`
+# How to run unit tests
+To run all unit tests (i.e., all pips tagged 'test')
+```bash
+./bxl.sh --internal "/f:tag='test'"
+```
 
-# How to run BuildXL unit tests on Linux
-## Build unit test binaries on a Windows Machine
-We currently do not support building BuildXL banaires in Linux machines. BuildXL unit test binaries need to be built in Windows.
-* `bxl /q:DebugLinux /f:output='out/bin/linux-x64-tests/*'`
+Standard bxl pip filtering options apply.
 
-## Build Linux sandbox binaries on a Linux Machine/VM
-* `cd Public/Src/Sandbox/Linux`
-* `make`
+To conveniently filter by test class or method name, you can use use the shortcut `--test-class` and `--test-method` switches provided by the `bxl.sh` script.
 
-## Deploy test binaries and Linux sandbox binaries to a Linux Machine/VM
-*Example:*
-* `scp -r Out/Bin/linux-x64-tests/debug/* AnyBuildTestVM:~/bxl-tests-linux-x64/`
-* `scp -r Public/Src/Sandbox/Linux/bin/debug/*.so AnyBuildTestVM:~/bxl-tests-linux-x64/TestProj/tests/shared_bin`
+# How to debug a `bxl` process running on Linux from Visual Studio running on Windows
 
-## Run tests
-* `cd ~/bxl-tests-linux-x64/`
-* `./bash_runner.sh`
+* (Linux) set `BuildXLDebugOnStart` env var to `1` before starting your `bxl` process
+    * e.g., `env BuildXLDebugOnStart=1 ./bxl.sh ...`
+    * this won't automatically attach a debugger; instead it will pause and print out a 'press Enter when the debugger is attached' message
+* (Windows) follow [remote-debugging](https://docs.microsoft.com/en-us/visualstudio/debugger/remote-debugging-dotnet-core-linux-with-ssh?view=vs-2022) instructions to attach to your `bxl` process from Visual Studio over SSH
+    * make sure your local source code is in sync with what you are debugging and that symbols match
+    * set breakpoints and stuff before proceeding
+* (Linux) press Enter to resume the `bxl` process
+    * the process will break in your Visual Studio instance if it hits any of the set breakpoints
 
-## Logs
-* Unit test logs can be found in *~/bxl-tests-linux-x64/TestProj/tests/shared_bin/XunitLogs*
-* Linux sandbox log can be found in */tmp* with name similar to *bxl_Pip1234.31870.log*
+# How to debug unit tests
+The most straightforward (but not very convenient) method is running a test from the command line (see above) and then inspecting the produced log files.
+
+Some options for trying to get the VSCode debugger to work:
+* generate csproj files (e.g., `./bxl.sh --internal --minimal --vs`), then try to load the projects in VSCode, then try to run unit tests directly from VSCode
+* build a test assembly of interest with BuildXL then create a VSCode task (in `tasks.json`) to run/debug that assembly with XUnit
+
+Debugging from Visual Studio over SSH (see the previous section) could also be an option.
+
+# How to test/debug the sandbox as a standalone library
+```bash
+set -eu
+
+# build the sandbox binaries
+./bxl.sh --internal "/f:output='*/Out/Bin/debug/linux-x64/lib*'"
+
+# run any process (e.g., 'ls -l') with LD_PRELOAD set to newly build 'libDetours.so' and __BUILDXL_LOG_PATH to an empty log file
+echo > bxl.log
+env 
+  __BUILDXL_LOG_PATH="$(pwd)/bxl.log" \
+  LD_PRELOAD="$(pwd)/Out/Bin/debug/linux-x64/libDetours.so" \
+  ls -l
+
+# inspect the bxl.log file to see reported accesses
+```