vcpkg-tool/docs/testing.md

Testing

Testing vcpkg is important whenever one makes changes to the tool itself, and writing new tests and keeping them up to date is also very important. If one's code is subtly broken, we'd rather find it out right away than a few weeks down the line when someone complains!

Running end to end (e2e) tests

To run the e2e test run the following command:
pwsh azure-pipelines/end-to-end-tests.ps1
Parameters:
WorkingRoot: The dir in which the tests are executed and temporary files are created. Default work
VcpkgRoot: The path to the vcpkg root. Default $VCPKG_ROOT
VCPKGExe: The path to the vcpkg executable. Default ./vcpkg
Filter: The name of the tests. The names are the filenames in the folder azure-pipelines/end-to-end-tests-dir.

Environment Variables

You will need to set the environment variable VCPKG_ROOT to the location of a copy of the main vcpkg repo on your machine, in order to run the tests.

Failure to do so will result in this error, when some of the tests are run:

Error: Could not detect vcpkg-root.

Running Tests

Before anything else, we should know whether you can actually run the tests! All you should need is a way to build vcpkg -- anything will do! All you have to do is follow the guide 😄

With $VCPKG_DIRECTORY being the directory where you have cloned vcpkg-tool, create a build directory in (commonly named out), and cd into it. Make sure to clean it out if it already exists!

$ cmake out -DCMAKE_BUILD_TYPE=Debug -G Ninja
$ cmake --build out
$ ./out/vcpkg-test # ./out/vcpkg-test [$SPECIFIC_TEST] for a specific set of tests
$ # i.e., ./out/vcpkg-test [arguments]

If you make any modifications to vcpkg, you'll have to do the cmake --build . step again.

Writing Tests

In your journey to write new tests, and to modify existing tests, reading the Catch2 documentation will be very helpful! Come back after reading those 😀

You'll want to place your tests in one of the existing files, or, if it doesn't belong in any of those, in a new file.

The layout of these tests is as follows:

// ... includes

TEST_CASE("Name of test", "[filename without the .cpp]") {
    // setup and the like
    REQUIRE(some boolean expression);
}

// etc.

You want to give these test cases good, descriptive, unique names, like SourceParagraph construct minimum -- it doesn't need to be extremely clear english, and shorthand is good, but make sure it's clear what the test is from the name. For the latter parameter, known as "tags", you should at least put the name of the file which the test case is in -- e.g., in arguments.cpp, you'd tag all of the test cases with [arguments].

If you wish to add helper functions, make sure to place them in an anonymous namespace -- this will ensure that they don't trample over anybody else's space. Additionally, there are a few helper functions that live in <vcpkg-test/util.h> and src/vcpkg-test/util.cpp -- make sure to look into them so that you're not rewriting functionality.

That should be all you need to know to start writing your own tests! Remember to check out the Catch2 documentation if you'd like to get more advanced with your tests, and good luck on your testing journey!

Adding New Test Files

Adding new test files should be easy and straightforward. All it requires is creating a new source file in src/vcpkg-test.

Example

Let's try writing a new test file called example (very creative, I know).

First, we should create a file, example.cpp, in src/vcpkg-test:

// vcpkg-test/example.cpp
#include <catch2/catch.hpp>

This is the minimum file needed for tests; let's rebuild!

$ cmake --build .
[80/80] Linking CXX executable vcpkg.exe

Okay, now let's make sure this worked; add a test case to example.cpp:

TEST_CASE("Example 1 - fail", "[example]") {
    REQUIRE(false);
}

Now build the tests again, and run them:

$ cmake --build .
[2/2] Linking CXX executable vcpkg-test.exe
$ ./vcpkg-test

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vcpkg-test.exe is a Catch v2.9.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
Example 1 - fail
-------------------------------------------------------------------------------
$VCPKG_DIRECTORY/src/vcpkg-test/example.cpp(3)
...............................................................................

$VCPKG_DIRECTORY/src/vcpkg-test/example.cpp(14): FAILED:
    REQUIRE( false )

===============================================================================
test cases:  102 |  101 passed | 1 failed
assertions: 3611 | 3610 passed | 1 failed

Hopefully, that worked! It should compile correctly, and have one failing test. Now let's try a more complex test, after deleting the old one;

// add #include <vcpkg/base/strings.h> to the top of the file
namespace Strings = vcpkg::Strings;

TEST_CASE("Example 2 - success", "[example]") {
    std::string hello = "Hello";
    REQUIRE(Strings::case_insensitive_ascii_equals(hello, "hELLo"));
    REQUIRE_FALSE(Strings::case_insensitive_ascii_starts_with(hello, "E"));
}

Now compile and build the tests, and this time let's only run our example tests:

$ cmake --build .
[2/2] Linking CXX executable vcpkg-test.exe
$ ./vcpkg-test [example]
Filters: [example]
===============================================================================
All tests passed (2 assertions in 1 test case)

Hopefully you have one test running and succeeding! If you have that, you have succeeded at adding a new file to vcpkg's tests. Congratulations! Have fun on the rest of your journey 🐱‍👤😁