From 26bcbaff4cb6a590a5c6d5a2ba8f3a78da3561c6 Mon Sep 17 00:00:00 2001 From: Billy Robert O'Neal III Date: Wed, 3 Feb 2021 12:52:52 -0800 Subject: [PATCH] Transfer docs. --- docs/benchmarking.md | 195 +++++++++++++++++++++++++++++++++++++++++++ docs/layout.md | 60 +++++++++++++ docs/testing.md | 152 +++++++++++++++++++++++++++++++++ 3 files changed, 407 insertions(+) create mode 100644 docs/benchmarking.md create mode 100644 docs/layout.md create mode 100644 docs/testing.md diff --git a/docs/benchmarking.md b/docs/benchmarking.md new file mode 100644 index 000000000..10bfccc69 --- /dev/null +++ b/docs/benchmarking.md @@ -0,0 +1,195 @@ +# Benchmarking + +Benchmarking new code against old code is extremely important whenever making +large changes to how something works. If you are attempting to make something +faster, and you end up slowing it down, you'll never know if you don't +benchmark! We have benchmarks in the `src/vcpkg-test` directory, just +like the tests -- they're treated as a special kind of test. + +## Running Benchmarks + +Unlike normal tests, benchmarks are hidden behind a special define -- `CATCH_CONFIG_ENABLE_BENCHMARKING` -- so that you never try to run benchmarks +unless you specifically want to. This is because benchmarks actually take quite +a long time! However, if you want to run benchmarks (and I recommend running +only specific benchmarks at a time), you can do so by passing the +`VCPKG_ENABLE_BENCHMARKING` option at cmake configure time. + +```sh +$ cmake -B out -S . -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DVCPKG_BUILD_BENCHMARKING=On + +-- The C compiler identification is MSVC 19.22.27905.0 +-- The CXX compiler identification is MSVC 19.22.27905.0 +-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe +-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe -- works +-- Detecting C compiler ABI info +-- Detecting C compiler ABI info - done +-- Detecting C compile features +-- Detecting C compile features - done +-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe +-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe -- works +-- Detecting CXX compiler ABI info +-- Detecting CXX compiler ABI info - done +-- Detecting CXX compile features +-- Detecting CXX compile features - done +-- Looking for pthread.h +-- Looking for pthread.h - not found +-- Found Threads: TRUE +-- Configuring done +-- Generating done +-- Build files have been written to: C:/Users/t-nimaz/src/vcpkg-tool/out + +$ cmake --build OUT + +[0/2] Re-checking globbed directories... +[80/80] Linking CXX executable vcpkg-test.exe +``` + +You can then run benchmarks easily with the following command (which run the +files benchmarks): + +```sh +$ ./out/vcpkg-test [!benchmark][file] +``` + +You can switch out `[file]` for a different set -- `[hash]`, for example. + +## Writing Benchmarks + +First, before anything else, I recommend reading the +[benchmarking documentation] at Catch2's repository. + +Now, after that, let's say that you wanted to benchmark, say, our ASCII +case-insensitive string compare against your new implementation. We place +benchmarks for code in the same file as their tests, so open +`vcpkg-test/strings.cpp`, and add the following at the bottom: + +```cpp +#if defined(CATCH_CONFIG_ENABLE_BENCHMARKING) +TEST_CASE ("case insensitive ascii equals: benchmark", "[strings][!benchmark]") +{ + BENCHMARK("qwertyuiop") { + return vcpkg::Strings::case_insensitive_ascii_equals("qwertyuiop", "QWERTYUIOP"); + }; +} +#endif +``` + +Remember the `;` at the end of the benchmark -- it's not required for +`TEST_CASE`s, but is for `BENCHMARK`s. + +Now, let's rebuild and run: + +```sh +$ cmake --build out +[0/2] Re-checking globbed directories... +[2/2] Linking CXX executable vcpkg-test.exe +$ ./out/vcpkg-test [strings][!benchmark] +Filters: [strings][!benchmark] + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +vcpkg-test.exe is a Catch v2.9.1 host application. +Run with -? for options + +------------------------------------------------------------------------------- +case insensitive ascii equals: benchmark +------------------------------------------------------------------------------- +C:\Users\t-nimaz\src\vcpkg\vcpkg-tool\src\vcpkg-test\strings.cpp(36) +............................................................................... + +benchmark name samples iterations estimated + mean low mean high mean + std dev low std dev high std dev +------------------------------------------------------------------------------- +qwertyuiop 100 2088 3.9672 ms + 25 ns 24 ns 26 ns + 6 ns 5 ns 8 ns + + +=============================================================================== +test cases: 1 | 1 passed +assertions: - none - +``` + +You've now written your first benchmark! + +But wait. This seems kind of silly. Benchmarking the comparison of literal +strings is great and all, but could we make it a little more realistic? + +This is where `BENCHMARK_ADVANCED` comes in. `BENCHMARK_ADVANCED` allows one to +write a benchmark that has a little setup to it without screwing up the numbers. +Let's try it now: + +```cpp +TEST_CASE ("case insensitive ascii equals: benchmark", "[strings][!benchmark]") +{ + BENCHMARK_ADVANCED("equal strings")(Catch::Benchmark::Chronometer meter) + { + std::vector strings; + strings.resize(meter.runs()); + std::mt19937_64 urbg; + std::uniform_int_distribution data_generator; + + std::generate(strings.begin(), strings.end(), [&] { + std::string result; + for (std::size_t i = 0; i < 1000; ++i) + { + result += vcpkg::Strings::b32_encode(data_generator(urbg)); + } + + return result; + }); + + meter.measure( + [&](int run) { return vcpkg::Strings::case_insensitive_ascii_equals(strings[run], strings[run]); }); + }; +} +``` + +Then, run it again! + +```sh +$ cmake --build out +[0/2] Re-checking globbed directories... +[2/2] Linking CXX executable vcpkg-test.exe +$ out/vcpkg-test [strings][!benchmark] +Filters: [strings][!benchmark] + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +vcpkg-test.exe is a Catch v2.9.1 host application. +Run with -? for options + +------------------------------------------------------------------------------- +case insensitive ascii equals: benchmark +------------------------------------------------------------------------------- +C:\Users\t-nimaz\src\vcpkg\vcpkg-tool\src\vcpkg-test\strings.cpp(36) +............................................................................... + +benchmark name samples iterations estimated + mean low mean high mean + std dev low std dev high std dev +------------------------------------------------------------------------------- +equal strings 100 2 5.4806 ms + 22.098 us 21.569 us 23.295 us + 3.842 us 2.115 us 7.41 us + + +=============================================================================== +test cases: 1 | 1 passed +assertions: - none - +``` + +And now you have a working benchmark to test the speed of the existing code, and +of new code! + +If you're writing a lot of benchmarks that follow the same sort of pattern, with +some differences in constants, look into `vcpkg-test/files.cpp`'s benchmarks -- +there are a lot of things one can do to make writing new benchmarks really easy. + +If you wish to add a benchmark for a piece of code that has not yet been tested, +please read the [testing documentation], and please write some unit tests. +The speed of your code isn't very important if it doesn't work at all! + +[benchmarking documentation]: https://github.com/catchorg/Catch2/blob/master/docs/benchmarks.md#top +[testing documentation]: ./testing.md#adding-new-test-files diff --git a/docs/layout.md b/docs/layout.md new file mode 100644 index 000000000..62bfa91ee --- /dev/null +++ b/docs/layout.md @@ -0,0 +1,60 @@ +# Layout of the vcpkg source tree + +If you'd like to contribute to the vcpkg tool itself, most of your code edits will be in this repo. + +## Build Files + +These are the files used to build and configure the project. + +### Top Level + +We have six files in this directory -- one `.clang-format` file, one +`CMakeLists.txt` file, three Visual Studio files, and `VERSION.txt`. + + - `.clang-format`: This is where we store the formatting settings of the + project. If you want to format the project, you can use the `format` target + with the CMake build system. + - `CMakeLists.txt`: This is where the CMake build system definition lives. If + you want to modify how one builds the project, or add a target, you can do + it here. + - The Visual Studio file: + - `vcpkg.natvis`: NATVIS files allow one to visualize objects of user + defined type in the debugger -- this one contains the definitions for + `vcpkg`'s types. + +## Source Files + +If you're modifying the project, it's likely that these are the directories that +you're going to deal with. + +### `include` + +There's one file in here -- `pch.h`. This contains most of the C++ standard +library, and acts as a [precompiled header]. You can read more at the link. + +There are three directories: + + - `catch2` -- This contains the single-header library [catch2]. We use this + library for both [testing] and [benchmarking]. + - `vcpkg` -- This contains the header files for the `vcpkg` project. All of + the interfaces for building, installing, and generally "port stuff" live + here. + - `vcpkg/base` -- This contains the interfaces for the + "vcpkg standard library" -- file handling, hashing, strings, + `Span`, printing, etc. + - `vcpkg-test` -- This contains the interfaces for any common utilities + required by the tests. + +### `src` + +The source files live here. `pch.cpp` is the source file for the +[precompiled header]; `vcpkg.cpp` is where the `vcpkg` binary lives. + +The interesting files live in the `vcpkg` and `vcpkg-test` directories. In +`vcpkg`, you have the implementation for the interfaces that live in +`include/vcpkg`; and in `vcpkg-test`, you have the tests and benchmarks. + +[precompiled header]: https://en.wikipedia.org/wiki/Precompiled_header +[catch2]: https://github.com/catchorg/Catch2 +[testing]: ./testing.md +[benchmarking]: ./benchmarking.md diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 000000000..8598b2e91 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,152 @@ +# Testing + +Testing vcpkg is important whenever one makes changes to the tool itself, and +writing new tests and keeping them up to date is also very important. If one's +code is subtly broken, we'd rather find it out right away than a few weeks down +the line when someone complains! + +## Running Tests + +Before anything else, we should know whether you can actually run the tests! +All you should need is a way to build vcpkg -- anything will do! All you have to +do is follow the guide 😄 + +With `$VCPKG_DIRECTORY` being the directory where you have cloned vcpkg-tool, +create a build directory in (commonly named `out`), and +`cd` into it. Make sure to clean it out if it already exists! + +```sh +$ cmake out -DCMAKE_BUILD_TYPE=Debug -G Ninja +$ cmake --build out +$ ./out/vcpkg-test # ./out/vcpkg-test [$SPECIFIC_TEST] for a specific set of tests +$ # i.e., ./out/vcpkg-test [arguments] +``` + +If you make any modifications to `vcpkg`, you'll have to do the +`cmake --build .` step again. + +## Writing Tests + +In your journey to write new tests, and to modify existing tests, reading the +[Catch2 documentation] will be very helpful! Come back after reading those 😀 + +You'll want to place your tests in one of the existing files, or, if it doesn't +belong in any of those, in a [new file](#adding-new-test-files). + +The layout of these tests is as follows: + +```cpp +// ... includes + +TEST_CASE("Name of test", "[filename without the .cpp]") { + // setup and the like + REQUIRE(some boolean expression); +} + +// etc. +``` + +You want to give these test cases good, descriptive, unique names, like +`SourceParagraph construct minimum` -- it doesn't need to be extremely clear +english, and shorthand is good, but make sure it's clear what the test is from +the name. For the latter parameter, known as "tags", you should at least put the +name of the file which the test case is in -- e.g., in `arguments.cpp`, you'd +tag all of the test cases with `[arguments]`. + +If you wish to add helper functions, make sure to place them in an anonymous +namespace -- this will ensure that they don't trample over anybody else's +space. Additionally, there are a few helper functions that live in +`` and `src/vcpkg-test/util.cpp` -- make sure to look into +them so that you're not rewriting functionality. + +That should be all you need to know to start writing your own tests! +Remember to check out the [Catch2 documentation] +if you'd like to get more advanced with your tests, +and good luck on your testing journey! + +## Adding New Test Files + +Adding new test files should be easy and straightforward. All it requires is +creating a new source file in `src/vcpkg-test`. + +### Example + +Let's try writing a new test file called `example` (very creative, I know). + +First, we should create a file, `example.cpp`, in `src/vcpkg-test`: + +```cpp +// vcpkg-test/example.cpp +#include +``` + +This is the minimum file needed for tests; let's rebuild! + +```sh +$ cmake --build . +[80/80] Linking CXX executable vcpkg.exe +``` + +Okay, now let's make sure this worked; add a test case to `example.cpp`: + +```cpp +TEST_CASE("Example 1 - fail", "[example]") { + REQUIRE(false); +} +``` + +Now build the tests again, and run them: + +```sh +$ cmake --build . +[2/2] Linking CXX executable vcpkg-test.exe +$ ./vcpkg-test + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +vcpkg-test.exe is a Catch v2.9.1 host application. +Run with -? for options + +------------------------------------------------------------------------------- +Example 1 - fail +------------------------------------------------------------------------------- +$VCPKG_DIRECTORY/src/vcpkg-test/example.cpp(3) +............................................................................... + +$VCPKG_DIRECTORY/src/vcpkg-test/example.cpp(14): FAILED: + REQUIRE( false ) + +=============================================================================== +test cases: 102 | 101 passed | 1 failed +assertions: 3611 | 3610 passed | 1 failed +``` + +Hopefully, that worked! It should compile correctly, and have one failing test. +Now let's try a more complex test, after deleting the old one; + +```cpp +// add #include to the top of the file +namespace Strings = vcpkg::Strings; + +TEST_CASE("Example 2 - success", "[example]") { + std::string hello = "Hello"; + REQUIRE(Strings::case_insensitive_ascii_equals(hello, "hELLo")); + REQUIRE_FALSE(Strings::case_insensitive_ascii_starts_with(hello, "E")); +} +``` + +Now compile and build the tests, and this time let's only run our example tests: + +```sh +$ cmake --build . +[2/2] Linking CXX executable vcpkg-test.exe +$ ./vcpkg-test [example] +Filters: [example] +=============================================================================== +All tests passed (2 assertions in 1 test case) +``` + +Hopefully you have one test running and succeeding! If you have that, you have +succeeded at adding a new file to vcpkg's tests. Congratulations! Have fun on +the rest of your journey 🐱‍👤😁 + +[Catch2 documentation]: https://github.com/catchorg/Catch2/blob/master/docs/tutorial.md#top