Data Pipelines

Data Pipelines are used to convert from arbitrary data into a matrix/tensor that can be consumed by machine learning algorithms.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Developer Quick Start

Note that these commands should be invoked within the root of the repository.

Linux

1. Run sudo bootstrap.sh ..

   This will need to happen once per machine or after a repository has moved to a different directory. Note that sudo is not necessary when running within a Docker container.

2. Run source ./Activate.sh x64

   This will need to happen each time a new terminal window is opened.

Windows

1. Run bootstrap.cmd ..

   This will need to happen once per machine or after a repository has moved to a different directory.

2. Run Activate.cmd x64

   This will need to happen each time a new terminal window is opened.

Useful Development Commands

The following commands are available within an activated terminal window.

Name	Command	Description
DevEnvScripts	`DevEnvScripts.sh	.cmd`
Builder	`Builder.sh	.cmd`
Tester	`Tester.sh	.cmd`
Formatter	`Formatter.sh	.cmd`

Invoking CMake

Native code is built via CMake. The following commands can be used to build any folder that contains a CMakeLists.txt file. In each example, create a build directory that will contain the generated CMake content and eventual binaries.

Assumed directory structure:

| - <Workspace dir>
    | - CMakeLists.txt
    | - ...
    | - build
        | - <Initially empty>

Within <Workspace Dir>/build run...

• [Debug] cmake -G Ninja ..
• [Debug with Code Coverage] cmake -G Ninja -D CppCommon_CODE_COVERAGE=ON ..
• [Release] cmake -G Ninja -D CMAKE_BUILD_TYPE=Release ..

Once CMake has generated the build files, within <WorkSpace Dir>/build run...

• [Build] cmake --build . or ninja -v (if the build generator was Ninja)
• [Test] ctest --parallel

Featurizers

Information for setup/configuration of the Featurizers library can be found here