Stas
GitHub
Родитель
cb9c0ecf90
Коммит
1dd0979612
|
|
@ -325,7 +325,7 @@ upload-utils - Uploads the tool definitions to the service
|
|||
allowed_skus = [
|
||||
'B1', 'B2', 'B3', 'D1', 'F1',
|
||||
'I1', 'I2', 'I3', 'P1V2', 'P2V2', 'P3V2',
|
||||
'PC2', 'PC3', 'PC4', 'S1', 'S2', 'S3', 'SHARED']
|
||||
'PC2', 'PC3', 'PC4', 'S1', 'S2', 'S3']
|
||||
|
||||
service_parser.add_argument(
|
||||
'--sku', default='B2',
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
"duration": "00:10:00",
|
||||
"rootFileshare": "sample-fuzz",
|
||||
"webhook": {
|
||||
"name": "sample-fuzz",
|
||||
"name": "LogicApp",
|
||||
"metadata": {}
|
||||
},
|
||||
"readonlyFileShareMounts": [
|
||||
|
|
|
|||
|
|
@ -24,7 +24,12 @@ you've cloned the RAFT repo to `/home/git/rest-api-fuzz-testing` on your local m
|
|||
Please adjust this command as necessary:
|
||||
|
||||
```bash
|
||||
$ alias raft='python /home/git/rest-api-fuzz-testing/cli/raft.py --defaults-context-path /home/git/rest-api-fuzz-testing/cli/defaults.json $*'
|
||||
$ alias raft='python /home/git/rest-api-fuzz-testing/cli/raft.py $*'
|
||||
```
|
||||
|
||||
Azure Portal Shell
|
||||
```bash
|
||||
$ alias raft='/opt/az/bin/python3 /home/git/rest-api-fuzz-testing/cli/raft.py'
|
||||
```
|
||||
|
||||
For PowerShell, you may create an alias with the following command. Note that this assumes
|
||||
|
|
@ -32,7 +37,7 @@ you've cloned the RAFT repo to `D:\git\rest-api-fuzz-testing` on your local mach
|
|||
adjust this command as necessary:
|
||||
|
||||
```powershell
|
||||
> function InvokeRaftCLI { python d:\git\rest-api-fuzz-testing\cli\raft.py --defaults-context-path d:\git\rest-api-fuzz-testing\cli\defaults.json $args}
|
||||
> function InvokeRaftCLI { python d:\git\rest-api-fuzz-testing\cli\raft.py $args}
|
||||
> set-alias raft InvokeRaftCLI
|
||||
```
|
||||
|
||||
|
|
@ -46,8 +51,8 @@ For interactive authentication, all you need to do is execute a command from an
|
|||
unauthenticated state; this will cause the system to prompt you to authenticate.
|
||||
All subsequent commands are authenticated until you call the `logout` command.
|
||||
|
||||
For unattended scripting scenarios, you must include an API key in each call via
|
||||
the `-secret` argument. This can be obtained either by using the Azure Portal or the `az` cli.
|
||||
For unattended scripting scenarios, you must include an Service Principal secret in each call via
|
||||
the `--secret` argument. This can be obtained either by using the Azure Portal or the `az` cli.
|
||||
|
||||
In the portal, navigate to your deployment's App Registration under the Azure Active
|
||||
Directory resource. The app registration will be named **[deploymentName]-raft**.
|
||||
|
|
@ -58,21 +63,21 @@ To create a secret using the `az` cli, see the documentation for [az ad sp crede
|
|||
##### Usage
|
||||
|
||||
```
|
||||
$ raft [-secret SECRET_VALUE]
|
||||
$ raft [--secret SECRET_VALUE]
|
||||
[--skip-sp-deployment]
|
||||
command subcommand ...
|
||||
```
|
||||
|
||||
##### Authentication arguments
|
||||
|
||||
- `-secret SECRET_VALUE` Used for scripting, where each individual call must be authenticated.
|
||||
- `--secret SECRET_VALUE` Used for scripting, where each individual call must be authenticated.
|
||||
|
||||
##### Optional arguments
|
||||
|
||||
- `--skip-sp-deployment` is used, new secret generation is not executed.
|
||||
- `--skip-sp-deployment` is used, creation of new Service Principal is not executed.
|
||||
However, the deployment will overwrite configuration settings for the APIService and the Orchestrator.
|
||||
These services need to know the service principal secret.
|
||||
Use this parameter to pass the secret to the deployment process.
|
||||
Use this parameter to pass the secret to the service re-deployment process.
|
||||
|
||||
##### Available commands
|
||||
|
||||
|
|
@ -80,7 +85,7 @@ $ raft [-secret SECRET_VALUE]
|
|||
|
||||
job create Creates a new job
|
||||
job status Returns the status of the given job
|
||||
job list Returns the status of all jobs started in the last _n_ hours
|
||||
job list Returns the status of all jobs started in the last 24 hours
|
||||
job update Deploys a job definition to an existing, idling job
|
||||
job delete Deletes a running job
|
||||
job results Returns a url to where the job's results are stored
|
||||
|
|
@ -182,57 +187,52 @@ $ raft job status
|
|||
##### Example
|
||||
|
||||
```bash
|
||||
$ raft job status --job-id ffc4a296-f85d-4122-b49b-8074b88c9755
|
||||
[
|
||||
{
|
||||
"tool": "RESTler",
|
||||
"jobId": "9ef309ab-df0a-4a98-a930-be6df04da47e",
|
||||
"state": "Completed",
|
||||
"metrics": {
|
||||
"totalRequestCount": 905,
|
||||
"responseCodeCounts": {
|
||||
"200": 99,
|
||||
"400": 683,
|
||||
"404": 82,
|
||||
"500": 41
|
||||
},
|
||||
"totalBugBucketsCount": 8
|
||||
},
|
||||
"utcEventTime": "2020-10-12T17:18:09.2568235Z",
|
||||
"details": [],
|
||||
"agentName": "0-restler-fuzz-1"
|
||||
},
|
||||
{
|
||||
"tool": "RESTler",
|
||||
"jobId": "9ef309ab-df0a-4a98-a930-be6df04da47e",
|
||||
"state": "Completed",
|
||||
"metrics": {
|
||||
"totalRequestCount": 899,
|
||||
"responseCodeCounts": {
|
||||
"200": 79,
|
||||
"400": 674,
|
||||
"404": 106,
|
||||
"500": 40
|
||||
},
|
||||
"totalBugBucketsCount": 8
|
||||
},
|
||||
"utcEventTime": "2020-10-12T17:18:13.9189848Z",
|
||||
"details": [],
|
||||
"agentName": "1-restler-3"
|
||||
},
|
||||
{
|
||||
"tool": "",
|
||||
"jobId": "9ef309ab-df0a-4a98-a930-be6df04da47e",
|
||||
"state": "Completed",
|
||||
"utcEventTime": "2020-10-12T17:19:00.5900998Z",
|
||||
"details": [
|
||||
"CPU Average 19.409091",
|
||||
"Network total bytes received: 944423",
|
||||
"Network total bytes sent: 728473"
|
||||
],
|
||||
"agentName": "9ef309ab-df0a-4a98-a930-be6df04da47e"
|
||||
}
|
||||
]
|
||||
$ raft job status --job-id e3405ee2-c451-4dc0-b72a-14da2cafc715
|
||||
|
||||
e3405ee2-c451-4dc0-b72a-14da2cafc715 Completed
|
||||
Details:
|
||||
cpuAverage : 11.166667
|
||||
networkTotalBytesReceived : 946815
|
||||
networkTotalBytesSent : 918968
|
||||
Agent: 0-fuzz-1 Tool: RESTler State: Completed Total Request Count: 224
|
||||
Response Code Count
|
||||
--------------- -------
|
||||
200 4
|
||||
201 7
|
||||
400 105
|
||||
409 106
|
||||
500 2
|
||||
|
||||
Details:
|
||||
finalSpecCoverage : 6 / 27
|
||||
numberOfBugsFound : 1
|
||||
======================
|
||||
Agent: 1-fuzz-2 Tool: RESTler State: Completed Total Request Count: 247
|
||||
Response Code Count
|
||||
--------------- -------
|
||||
200 4
|
||||
201 19
|
||||
400 101
|
||||
409 121
|
||||
500 2
|
||||
|
||||
Details:
|
||||
finalSpecCoverage : 5 / 27
|
||||
numberOfBugsFound : 1
|
||||
======================
|
||||
Agent: 2-fuzz-3 Tool: RESTler State: Completed Total Request Count: 488
|
||||
Response Code Count
|
||||
--------------- -------
|
||||
200 48
|
||||
201 175
|
||||
400 230
|
||||
409 33
|
||||
500 2
|
||||
|
||||
Details:
|
||||
finalSpecCoverage : 6 / 27
|
||||
numberOfBugsFound : 1
|
||||
======================
|
||||
```
|
||||
|
||||
<br/>
|
||||
|
|
@ -256,133 +256,29 @@ $ raft job list
|
|||
##### Example
|
||||
|
||||
```bash
|
||||
$ raft job list --look-back-hours 8
|
||||
[
|
||||
{
|
||||
"tool": "",
|
||||
"jobId": "02c7b598c-bce2-4a19-ab2d-5ee7fc7a0028",
|
||||
"state": "Error",
|
||||
"utcEventTime": "2020-10-12T16:59:52.504944Z",
|
||||
"details": [
|
||||
"Failed to get configuration for unsupported tool: \"fuzzstring\""
|
||||
],
|
||||
"agentName": "02c7b598c-bce2-4a19-ab2d-5ee7fc7a0028"
|
||||
},
|
||||
{
|
||||
"tool": "",
|
||||
"jobId": "039b42d54-84a4-4413-9ab0-cd1351284d37",
|
||||
"state": "Error",
|
||||
"utcEventTime": "2020-10-12T17:11:19.0013263Z",
|
||||
"details": [
|
||||
"Failed to get configuration for unsupported tool: \"fuzzstring\""
|
||||
],
|
||||
"agentName": "039b42d54-84a4-4413-9ab0-cd1351284d37"
|
||||
},
|
||||
{
|
||||
"tool": "",
|
||||
"jobId": "06bdc1817-4512-47d5-8fce-348ff4ffcb06",
|
||||
"state": "Error",
|
||||
"utcEventTime": "2020-10-12T17:09:14.0202546Z",
|
||||
"details": [
|
||||
"Failed to get configuration for unsupported tool: \"fuzzstring\""
|
||||
],
|
||||
"agentName": "06bdc1817-4512-47d5-8fce-348ff4ffcb06"
|
||||
},
|
||||
{
|
||||
"tool": "",
|
||||
"jobId": "091379c3e-d146-4f5d-9357-32b566af03f3",
|
||||
"state": "Error",
|
||||
"utcEventTime": "2020-10-12T16:52:30.9707085Z",
|
||||
"details": [
|
||||
"Failed to get configuration for unsupported tool: \"fuzzstring\""
|
||||
],
|
||||
"agentName": "091379c3e-d146-4f5d-9357-32b566af03f3"
|
||||
},
|
||||
{
|
||||
"tool": "RESTler",
|
||||
"jobId": "61a50b73-02cf-4b07-9208-fab9dc7f08ad",
|
||||
"state": "Completed",
|
||||
"metrics": {
|
||||
"totalRequestCount": 900,
|
||||
"responseCodeCounts": {
|
||||
"200": 65,
|
||||
"400": 690,
|
||||
"404": 105,
|
||||
"500": 40
|
||||
},
|
||||
"totalBugBucketsCount": 8
|
||||
},
|
||||
"utcEventTime": "2020-10-12T17:06:37.9648198Z",
|
||||
"details": [],
|
||||
"agentName": "0-restler-test-fuzz-lean-1"
|
||||
},
|
||||
{
|
||||
"tool": "RESTler",
|
||||
"jobId": "61a50b73-02cf-4b07-9208-fab9dc7f08ad",
|
||||
"state": "Completed",
|
||||
"metrics": {
|
||||
"totalRequestCount": 16,
|
||||
"responseCodeCounts": {
|
||||
"200": 7,
|
||||
"400": 1,
|
||||
"404": 4,
|
||||
"500": 4
|
||||
},
|
||||
"totalBugBucketsCount": 2
|
||||
},
|
||||
"utcEventTime": "2020-10-12T16:56:07.621813Z",
|
||||
"details": [],
|
||||
"agentName": "1-restler-test-2"
|
||||
},
|
||||
{
|
||||
"tool": "",
|
||||
"jobId": "61a50b73-02cf-4b07-9208-fab9dc7f08ad",
|
||||
"state": "Completed",
|
||||
"utcEventTime": "2020-10-12T17:07:00.7120105Z",
|
||||
"details": [
|
||||
"CPU Average 16.989583",
|
||||
"Network total bytes received: 603156",
|
||||
"Network total bytes sent: 410004"
|
||||
],
|
||||
"agentName": "61a50b73-02cf-4b07-9208-fab9dc7f08ad"
|
||||
},
|
||||
{
|
||||
"tool": "RESTler",
|
||||
"jobId": "9ef309ab-df0a-4a98-a930-be6df04da47e",
|
||||
"state": "Completed",
|
||||
"metrics": {
|
||||
"totalRequestCount": 905,
|
||||
"responseCodeCounts": {
|
||||
"200": 99,
|
||||
"400": 683,
|
||||
"404": 82,
|
||||
"500": 41
|
||||
},
|
||||
"totalBugBucketsCount": 8
|
||||
},
|
||||
"utcEventTime": "2020-10-12T17:18:09.2568235Z",
|
||||
"details": [],
|
||||
"agentName": "0-restler-fuzz-1"
|
||||
},
|
||||
{
|
||||
"tool": "RESTler",
|
||||
"jobId": "9ef309ab-df0a-4a98-a930-be6df04da47e",
|
||||
"state": "Completed",
|
||||
"metrics": {
|
||||
"totalRequestCount": 899,
|
||||
"responseCodeCounts": {
|
||||
"200": 79,
|
||||
"400": 674,
|
||||
"404": 106,
|
||||
"500": 40
|
||||
},
|
||||
"totalBugBucketsCount": 8
|
||||
},
|
||||
"utcEventTime": "2020-10-12T17:18:13.9189848Z",
|
||||
"details": [],
|
||||
"agentName": "1-restler-3"
|
||||
}
|
||||
]
|
||||
$ raft job list --look-back-hours 1
|
||||
|
||||
sample-compile-28e72091-3c49-47fc-8343-ca15fb084456 Completed
|
||||
Details:
|
||||
cpuAverage : 0.000000
|
||||
networkTotalBytesReceived : 0
|
||||
networkTotalBytesSent : 0
|
||||
Agent: 0-compile-1 Tool: RESTler State: Completed
|
||||
======================
|
||||
Agent: 1-compile-2 Tool: RESTler State: Completed
|
||||
======================
|
||||
|
||||
sample-compile-8d993113-d283-446a-a49f-e520309498a7 Completed
|
||||
Details:
|
||||
cpuAverage : 0.000000
|
||||
networkTotalBytesReceived : 0
|
||||
networkTotalBytesSent : 0
|
||||
Agent: 0-compile-1 Tool: RESTler State: Completed
|
||||
======================
|
||||
Agent: 1-compile-2 Tool: RESTler State: Completed
|
||||
======================
|
||||
|
||||
Total number of jobs: 2
|
||||
```
|
||||
|
||||
<br/>
|
||||
|
|
@ -391,12 +287,11 @@ $ raft job list --look-back-hours 8
|
|||
|
||||
The `job update` command deploys a job definition to an existing job that
|
||||
was created with the `isIdling` flag set to true, which tells the service
|
||||
to not delete the container when the job has completed. This lets you
|
||||
quickly deploy a new job without waiting for container creation.
|
||||
to not run the execution command but instead keep container in an idling state. This lets you
|
||||
quickly run a new job config over and over without waiting for container creation.
|
||||
|
||||
> [!NOTE]
|
||||
> It is also possible to use `ssh` to log into the container if manual exploration of the container is needed.
|
||||
> If the container is not running for some reason, the job will be created as normal.
|
||||
> If the job container creation failed for some reason, the job will not be created. You can check the application insights log for failures.
|
||||
|
||||
##### Usage
|
||||
|
|
@ -453,7 +348,7 @@ $ raft job delete --job-id 76d90e69-86ec-4f86-8870-f3d733f833e0
|
|||
## job results
|
||||
|
||||
The `job results` command returns the path to the storage account where the job results
|
||||
are stored. You may browse to this URL to see results using a browser.
|
||||
are stored. You may browse to this URL to see results using a web browser.
|
||||
|
||||
##### Usage
|
||||
|
||||
|
|
@ -509,9 +404,9 @@ indicates the App Service Plan size; the default is `B2`. Note that these are L
|
|||
##### Examples
|
||||
|
||||
```javascript
|
||||
$ raft service deploy --sku SHARED
|
||||
$ raft service deploy --sku B2
|
||||
|
||||
$ raft service deploy --skip-sp-deployment --secret PYqL447A7j.VYURqIrvIE2ur_ITI984r
|
||||
$ raft service deploy --skip-sp-deployment --secret PYqL447A7jVYURqIrvIE2ur_ITI984r
|
||||
```
|
||||
|
||||
<br/>
|
||||
|
|
@ -520,6 +415,7 @@ $ raft service deploy --skip-sp-deployment --secret PYqL447A7j.VYURqIrvIE2ur_ITI
|
|||
|
||||
The `service restart` command restarts the API service and the orchestrator. If there
|
||||
is a new version of RAFT available, it will be automatically be downloaded and installed.
|
||||
Also any changes that are made to the keyvault secrets will require a service restart in order for the service to start using the new secrets.
|
||||
|
||||
##### Usage
|
||||
|
||||
|
|
|
|||
|
|
@ -6,8 +6,7 @@
|
|||
**Why use file shares?**
|
||||
</br>File shares can be mounted to the containers running tools. This provides an easy way to share data.
|
||||
|
||||
The use of file shares makes it easy to upload your custom swagger files, and authentication configurations files.
|
||||
You can also view real time log changes and results in the files share during the job run,
|
||||
The use of file shares makes it easy to upload your custom swagger files and also passing data between different jobs. For example RESTler requires an initial compile step that is consumed by the Test and Fuzz steps. The output of the compilation step can be mounted to multiple containers. Each of these containers (tasks) may be configured differently to get different behavior from the tool. You can also view real time log changes and results in the files share during the job run,
|
||||
and mount these file shares to your computer for a better user experience.
|
||||
|
||||
To mount the file share, use the connect button on the azure portal.
|
||||
|
|
|
|||
|
|
@ -55,6 +55,8 @@ Here's a breakdown of the estimated Azure spend for running RAFT:
|
|||
In summary, a RAFT instance that ran a series of daily tests against ten REST
|
||||
APIs would cost less than $100 a month.
|
||||
|
||||
NOTE: `defaults.json` provides a flag to disable deployment of Application Insights during the initial service deployment. This will reduce Azure costs without affecting service functionality.
|
||||
|
||||
<br/>
|
||||
|
||||
## How a job executes
|
||||
|
|
@ -69,7 +71,7 @@ Job configurations are submitted via a JSON blob to the RAFT service front-end v
|
|||
CLI or other client app (e.g., Postman or some code you've written calling the REST API).
|
||||
|
||||
Note that upon running `job create` or calling `POST /jobs/v1` with the job
|
||||
definition JSON, you're handed back a Job ID GUID; you'll need this to identify
|
||||
definition JSON, you're handed back a Job ID; you'll need this to identify
|
||||
the specific Azure resources created for this job.
|
||||
|
||||
Further note that this step causes the `jobStatus` webhook to fire.
|
||||
|
|
|
|||
|
|
@ -4,20 +4,25 @@ The following guide should get you up and running with an instance of RAFT.
|
|||
|
||||
<br/>
|
||||
|
||||
### The first option is to setup all the dependencies on your workstation and use the RAFT CLI from there. The second option is to use the Azure Portal Shell. When using the portal's shell, you will only need to upload the CLI package as all required dependencies are already installed.
|
||||
|
||||
## Step 1: Enable the RAFT Command Line Interface (CLI)
|
||||
|
||||
Let's start out by getting the RAFT command line interface (CLI from now on)
|
||||
up and running. It functions just the same on Windows and Linux clients.
|
||||
|
||||
These two steps are required if you've decided to run the CLI from your workstation:
|
||||
- First, you'll need to [install Python](https://www.python.org/downloads/) if
|
||||
you don't have it installed already; RAFT requires at least **version 3.6**.
|
||||
|
||||
- Next, you'll need to [install the Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
|
||||
if you haven't already; RAFT requires at least **version 2.12**.
|
||||
|
||||
If you've decided to use the Azure Portal Shell, keep in mind that the path to Python is `/opt/az/bin/python3`
|
||||
|
||||
- Now download the RAFT CLI, either just the binaries or the source tree if you intend to build them from source:
|
||||
|
||||
- Get the RAFT CLI from [releases](https://github.com/microsoft/raft/releases)
|
||||
- Get the RAFT CLI from [releases](https://github.com/microsoft/rest-api-fuzz-testing/releases)
|
||||
- Clone the repo at https://github.com/microsoft/raft
|
||||
|
||||
- At this point, you're able to run a the one-time prep script using Python's
|
||||
|
|
@ -94,25 +99,17 @@ Note that only four of these are required.
|
|||
| `metricsOptIn`* | Yes | Whether you want the service to send us anonymized usage data; we use this to improve the service and respond to errors and other problems proactively (Note: to change you choice, just update the field and redeploy) |
|
||||
| `isDevelop` | No | Is this deployment for developing the RAFT service? Setting this value to true will generate yaml variables for use in your build pipelines |
|
||||
| `useAppInsights` | No | deploy AppInsights and use it to write all service logs |
|
||||
| `containerRegistryServiceConnection`** | No | service connection to the azure container registry which stores images |
|
||||
| `registry`** | No | registry which stores service images. Default: mcr.microsoft.com |
|
||||
|
||||
*By default, we collect anonymous usage data from your RAFT instance, which helps
|
||||
us understand how users use RAFT and the problems they experience, which in turn
|
||||
helps us improve the quality of the offering over time. Specifically, We do **not**
|
||||
collect any data about the targets and results of tools you might run. The data
|
||||
fields we collect are defined in the `telemetry.fs` source file. To opt-out of
|
||||
fields we collect are defined in the `src/Contracts/Telemetry.fs` source file. To opt-out of
|
||||
sending this data to Microsoft, simply set the `metricsOptIn` field in the `defaults.json`
|
||||
file set to false. You may also manually opt out by clearing the value from the setting
|
||||
`RAFT_METRICS_APP_INSIGHTS_KEY` in the apiservice and the orchestrator function app.
|
||||
(Do not delete the setting; simply clear the value.)
|
||||
|
||||
**If your deployment uses a private registry, then this value should reference a
|
||||
Service Connection in your project (see Project Settings -> Pipelines -> Service connections).
|
||||
This allows the build pipeline to access the registry. Also, you will need to manually
|
||||
[add a service principal](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-service-principal#create-a-service-principal)
|
||||
and [grant it access](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aci)
|
||||
to your private registry access control, granting the AcrPull role to it.
|
||||
|
||||
<br/>
|
||||
|
||||
## Step 4: Run the Deployment Script
|
||||
|
|
@ -143,23 +140,8 @@ metricsOptIn - allow Microsoft collect anonymized metrics from the deployment.
|
|||
|
||||
useAppInsights - deploy AppInsights and use it to write all service logs
|
||||
|
||||
containerRegistryServiceConnection - service connection to the azure container
|
||||
registry which stores images.
|
||||
|
||||
If your deployment uses a private registry, then this value should
|
||||
reference a Service Connection in your project
|
||||
(see Project Settings -> Pipelines -> Service connections).
|
||||
This allows the build pipeline to access the registry.
|
||||
|
||||
Also, you will need to manually add the newly Service Principal that is
|
||||
created when the deployment script is run, to your private registry
|
||||
access control, granting the AcrPull role to it.
|
||||
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-service-principal#create-a-service-principal
|
||||
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aci
|
||||
|
||||
getToken - Needed because we don't have images in public repositories yet.
|
||||
Only needed for Microsoft internal deployments.
|
||||
|
||||
registry - registry which stores service images.
|
||||
Default: mcr.microsoft.com
|
||||
-------------------------
|
||||
To apply any changes made to the defaults.json file,
|
||||
please run 'raft.py service deploy'
|
||||
|
|
|
|||
|
|
@ -1,15 +1,15 @@
|
|||
# How to Onboard a Tool, Step by Step
|
||||
|
||||
REST API Fuzz Testing (RAFT) is a platform designed to automate the execution
|
||||
of security tools against REST APIs, typically from a CI/CD platform. But it
|
||||
of security tools against REST APIs, typically from a CI/CD pipeline. But it
|
||||
in theory can run any tool you'd like, subject to some conditions, and in this
|
||||
doc we'll walk you through the process of onboarding a new tools to the platform.
|
||||
|
||||
<br/>
|
||||
|
||||
## Step One: Prep the Tool Container
|
||||
## Step One: Prep the Tool Docker Container
|
||||
|
||||
RAFT allows you to onboard any tool that is packaged into a docker container.
|
||||
RAFT allows you to onboard any tool that is packaged into a Docker container.
|
||||
This means that any public tool (or private tool you develop) is available to
|
||||
include in the test suite you design to test your services.
|
||||
|
||||
|
|
@ -19,22 +19,18 @@ include in the test suite you design to test your services.
|
|||
Tools are packaged in containers. These containers are then deployed into a
|
||||
[container instance](https://azure.microsoft.com/en-us/services/container-instances/)
|
||||
in Microsoft Azure. How many tools you may run simultaneously and the cost of doing
|
||||
so is determined by your subscription's
|
||||
so is determined by your subscription limits
|
||||
[quotas and limits](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-quotas#service-quotas-and-limits).
|
||||
|
||||
#### Installing a command-line tool in a container
|
||||
|
||||
TODO: point to instructions on docker.com,
|
||||
|
||||
TODO-clarify: The only output method that RAFT currently provides to extract data is through a
|
||||
mounted file share. That said, you may export data from the docker container
|
||||
(e.g., write to a DB, call and API, etc.) in any way you choose -- we do not
|
||||
block this.
|
||||
If you are new to Docker than this is a good starting point:
|
||||
[getting started with Docker](https://www.docker.com/get-started)
|
||||
|
||||
You may want to consider writing an **agent**, or a daemon if you prefer, that
|
||||
starts up and provides continual monitoring of the security tool you're running;
|
||||
|
||||
- negotiating authentication to the target
|
||||
- retrieving authentication token
|
||||
- sending messages to the Deployment's Service Hub to:
|
||||
- Emit a progress indicator (percent covered, items found, etc.)
|
||||
- Register a bug found (which would in turn trigger a web hook)
|
||||
|
|
@ -45,13 +41,17 @@ an agent we wrote to get ZAP to run inside RAFT. The `report_status` function,
|
|||
for example, notifies the service bus of ZAP event of note.
|
||||
|
||||
Note that the agent can only send outbound messages; we do not support inbound
|
||||
connections to the agent at this time.
|
||||
connections to the agent.
|
||||
|
||||
RAFT starts Docker containers (tasks) with "no-restart" policy. Tasks run until termination, either success or failure. When all tasks have terminated then the job is considered to be complete.
|
||||
|
||||
|
||||
RAFT sets environment variables, and mounts file shares before the container is started. **RAFT_WORK_DIRECTORY** is the file share that is writable and will persist after container termination. This is meant for writing output from test tools runs.
|
||||
|
||||
#### Making the container available to RAFT
|
||||
|
||||
The container that you want to add must either be present in [Docker Hub](https://hub.docker.com/)
|
||||
or in a private repository; RAFT will automatically download it to the machine from
|
||||
which you're executing RAFT if it is missing or your local copy is outdated.
|
||||
or in a private repository.
|
||||
|
||||
To enable RAFT to retrieve a container from a private repository, you'll need
|
||||
to create a secret in your Deployment's key vault, which is named
|
||||
|
|
@ -90,21 +90,24 @@ the existing tools:
|
|||
RAFT expects each tool it launches to have a designated **config.json** file inside
|
||||
the tool folder created in Step Two that maps the RAFT orchestration steps to the
|
||||
corresponding commands your tool expects, and defines the resources that the tool's
|
||||
container needs to run.
|
||||
container needs to run. RAFT will always deploy the latest version that matches the container tag specified in **config.json** file.
|
||||
|
||||
Here's a simple example:
|
||||
|
||||
```json
|
||||
{
|
||||
"container" : "demoTool/demoTool-stable",
|
||||
"container" : "demoTool/demoTool-stable:v.1.0",
|
||||
"run" : {
|
||||
"command" : "bash",
|
||||
"arguments" : ["-c",
|
||||
"cd $RAFT_RUN_DIRECTORY; ln -s $RAFT_WORK_DIRECTORY /demoTool; python3 run.py install; python3 run.py" ]
|
||||
"cd $RAFT_RUN_DIRECTORY; ln -s $RAFT_WORK_DIRECTORY /demoTool; python3 run.py install; python3 run.py --run-faster $RUN_FASTER" ]
|
||||
},
|
||||
"idle" : {
|
||||
"command" : "bash",
|
||||
"arguments" : ["-c", "echo DebugMode; while true; do sleep 100000; done;"]
|
||||
},
|
||||
"environmentVariables" : {
|
||||
"RUN_FASTER" : "1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -117,20 +120,15 @@ We'll discuss each of the configuration sections in the next steps.
|
|||
|
||||
In this example, the `container` field defines the docker image. As explained
|
||||
in Step One, RAFT will download this container if it is not present or outdated locally.
|
||||
|
||||
For private Docker registries, when the orchestrator starts it will search the key vault
|
||||
for secrets that begin with `PrivateRegistry` and register them in alphabetical order
|
||||
so that each is available for use. Registries will be searched for the requested image
|
||||
in the same order, starting with the private registries and then iterating through the
|
||||
public Docker Hub.
|
||||
|
||||
For private Docker registries, when the orchestrator starts it will search the RAFT service key vault
|
||||
for secrets that begin with `PrivateRegistry` and enable container deployment from those registries.
|
||||
</br>
|
||||
|
||||
## Step Five: Define Run and Idle Sections
|
||||
|
||||
The `run` and `idle` sections define how RAFT will execute a security tool under normal
|
||||
use and for debugging tool runs, respectively. When tool execution completes in the former
|
||||
scenario, the parent container is marked for deletion and recycles; in the latter scenario,
|
||||
The `run` and `idle` sections define how RAFT will start the specified container under normal
|
||||
or debug use, respectively. When tool execution completes in the former
|
||||
scenario, the parent container is marked for deletion and garbage collected; in the latter scenario,
|
||||
the container is not deleted and must be removed manually, providing operators the ability
|
||||
to connect to it to investigate issues.
|
||||
|
||||
|
|
@ -139,16 +137,18 @@ or absolute path to an executable image on the container image that will be run.
|
|||
`arguments` string is passed to STDIN as the command line argument string.
|
||||
|
||||
Whether the `run` or `idle` sections are executed upon the `create job` command is
|
||||
determined by the `isIdle` setting in the submitted job description JSON blob. If the
|
||||
determined by the `isIdle` setting in the submitted job definition JSON blob. If the
|
||||
`update job` command is executed, for example while you're SSH'd to the container for
|
||||
debugging purposes, the `run` command is executed.
|
||||
|
||||
`environmentVariables` is a map of environment variables that will be set on the container when it starts. They are accessible from `command` or `arguments` parameters of the **config.json**
|
||||
|
||||
#### Referencing Environment Variables
|
||||
|
||||
The `arguments` parameter may reference the following environment variables that have
|
||||
been populated for you by the orchestration:
|
||||
|
||||
| Environment Variable | Value |
|
||||
| Predefined Environment Variable | Value |
|
||||
|----------|------------------------|
|
||||
| RAFT_JOB_ID | The Job ID string of the currently executing job; you'll need this if you plan to generate events |
|
||||
| RAFT_CONTAINER_GROUP_NAME | The name of the parent's container's Container Group |
|
||||
|
|
@ -157,8 +157,10 @@ been populated for you by the orchestration:
|
|||
| RAFT_WORK_DIRECTORY | The absolute path to the file share mounted to the container (contains the **task-config.json** file) |
|
||||
| RAFT_RUN_DIRECTORY | The absolute path to the tool folder created in Step Two that gets mounted as read-only to the container as well* |
|
||||
| RAFT_RUN_CMD | The command text provided in the `command` parameter |
|
||||
| RAFT_TASK_INDEX | Array index of the task defined in job definition JSON blob
|
||||
| RAFT_SITE_HASH | RAFT deployment hash
|
||||
|
||||
*This folder TODO-Stas: When the tool is uploaded to the file share via the cli command `python raft.py service upload-utils` a unique file share is created and mounted to the container as read-only. This gives you an easy way to reference any scripts or executables you launch
|
||||
*When the tool is uploaded to the file share via the cli command `python raft.py service upload-utils` a unique file share is created and mounted to the container as read-only. The path to the tool folder running the task within the file share is set in **RAFT_RUN_DIRECTORY** environment variable.
|
||||
|
||||
#### Referencing the task-config.json file
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
// Copyright (c) Microsoft Corporation.
|
||||
// Licensed under the MIT License.
|
||||
|
||||
//Copied from RESTler repo (need to keep in sync)
|
||||
//https://github.com/microsoft/restler-fuzzer/blob/main/src/compiler/Restler.Compiler/Telemetry.fs
|
||||
|
||||
module Restler.Telemetry
|
||||
|
||||
//Instrumentation key is from app insights resource in Azure Portal
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче