зеркало из https://github.com/Azure/benchpress.git
update the main README and add a Manually Testing the Test Engine doc (#27)
* update the main README and add a Manually Testing the Test Engine doc * linting and spelling errors * Apply suggestions from code review Co-authored-by: Omeed Musavi <omusavi@users.noreply.github.com> * addressing PR comments * add sample code to samples Co-authored-by: Omeed Musavi <omusavi@users.noreply.github.com>
This commit is contained in:
jessica-ern
GitHub
Родитель
59a2644e76
Коммит
6788879f35
37
README.md
37
README.md
|
|
@ -1,11 +1,10 @@
|
|||
# Bicep testing framework
|
||||
|
||||
This framework is intended to work as a testing framework for Azure deployment features by using [Bicep](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview?tabs=bicep).
|
||||
This is a testing framework for Azure deployments using [Bicep](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview?tabs=bicep).
|
||||
|
||||
In order to see how you can work with this framework you can find one sample bicep file in the folder [samples](./samples/dotnet/samples/pwsh/resourceGroup.bicep)
|
||||
that will be deployed by using one PowerShell script.
|
||||
An example of how to use this framework can be found in the [Powershell Test Sample](docs/powershell_test_sample.md) guide.
|
||||
|
||||
Process is the following:
|
||||
The process of the tests is the following:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
|
|
@ -14,13 +13,37 @@ A[Creation] -->|Bicep| B[Verification]
|
|||
B --> C[Remove]
|
||||
```
|
||||
|
||||
**Creation**: New Features are gonna be deployed through Bicep files
|
||||
**Verification**: Test is going to confirm the resource exists and also assert if it matches the expected value
|
||||
**Remove**: Optionally resources can be removed after being tested
|
||||
**Creation**: New Features are deployed using Bicep files
|
||||
**Verification**: Tests confirm that the resource exists and that it matches the expected values
|
||||
**Remove**: Optionally, resources can be removed after being tested
|
||||
|
||||
## Benchpress Architecture
|
||||
|
||||
BenchPress uses [gRPC](https://grpc.io/docs/what-is-grpc/introduction/) to create a multi-language testing framework.
|
||||
|
||||
The BenchPress Test Engine is a C# gRPC Server located under `/engine/BenchPress.TestEngine`. The Test Engine is responsible
|
||||
for the business logic of deploying Bicep files, obtaining information about deployed resources in Azure, and cleaning up the
|
||||
deployment afterward.
|
||||
|
||||
The BenchPress Test Frameworks (located under `/framework/`) are gRPC Clients of multiple languages. The Test Frameworks are
|
||||
responsible interfacing between the user's tests (written in their chosen language) and the Test Engine. They are also
|
||||
responsible for managing the life cycle of the Test Engine.
|
||||
|
||||
gRPC uses [protocol buffers](https://developers.google.com/protocol-buffers/docs/overview). The `/protos/` folder contains BenchPress's .proto files. These define the API that the gRPC
|
||||
Server and Clients use to communicate.
|
||||
|
||||
|
||||
|
||||
## Getting started
|
||||
|
||||
See [Getting Started](docs/getting_started.md) guide on how to start development on *Benchpress*.
|
||||
|
||||
## Deveplopment Tips
|
||||
|
||||
See [Github Actions Lint Workflow](docs/github_actions_lint_workflow.md) for how to maintain the CI/CD pipeline.
|
||||
|
||||
See [Manually Testing the Test Engine](docs/manually_testing_the_test_engine.md) for guidance on exploring the gRPC endpoints.
|
||||
|
||||
## Contributing
|
||||
|
||||
This project welcomes contributions and suggestions. Most contributions require you to agree to a
|
||||
|
|
|
|||
|
|
@ -30,7 +30,9 @@
|
|||
"msrc",
|
||||
"Benchpress",
|
||||
"BenchPress",
|
||||
"pwsh"
|
||||
"pwsh",
|
||||
"proto",
|
||||
"protos"
|
||||
],
|
||||
"version": "0.2",
|
||||
"patterns": [
|
||||
|
|
|
|||
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 69 KiB |
|
|
@ -0,0 +1,199 @@
|
|||
# Manually Testing the Test Engine
|
||||
|
||||
Ideally, when a Test Engine method needs to be E2E tested, we would run one of the Test Frameworks against the Engine. The Test Framework would already know how to spin up the Engine, make gRPC calls, and kill the Engine. So, testing the Engine would be as simple as calling the Engine method from within the Framework.
|
||||
|
||||
Unfortunately, a Test Framework might not be ready to successfully call the Test Engine method that you want to test. In this case, we can write our own gRPC Client to call the Engine's gRPC endpoint and verify that it works.
|
||||
|
||||
## Running the Engine
|
||||
|
||||
First, ensure that your machine is authenticated with Azure by running `az login`.
|
||||
|
||||
From the `/engine/BenchPress.TestEngine/` folder, start the server with `dotnet run`. When the gRPC Server starts, it will output what port it is running on (in this case, `5152`).
|
||||
|
||||
```bash
|
||||
info: Microsoft.Hosting.Lifetime[14]
|
||||
Now listening on: http://localhost:5152
|
||||
info: Microsoft.Hosting.Lifetime[0]
|
||||
Application started. Press Ctrl+C to shut down.
|
||||
info: Microsoft.Hosting.Lifetime[0]
|
||||
Hosting environment: Production
|
||||
info: Microsoft.Hosting.Lifetime[0]
|
||||
Content root path: /workspaces/benchpress-private/engine/BenchPress.TestEngine/
|
||||
```
|
||||
|
||||
## Writing a Client
|
||||
|
||||
The following Client code examples can be found under `/samples/manual-testers/`.
|
||||
|
||||
### C#
|
||||
|
||||
Create a new C# Project to make calls to the Test Engine.
|
||||
|
||||
Ensure that the `.csproj` file references the necessary gRPC packages...
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Google.Protobuf" Version="3.21.6" />
|
||||
<PackageReference Include="Grpc.Net.Client" Version="2.49.0" />
|
||||
<PackageReference Include="Grpc.Tools" Version="2.49.1">
|
||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
</PackageReference>
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
... and the `.proto` files. Make sure they are marked as Client, not Server.
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<Protobuf Include="../../../protos/deployment.proto" GrpcServices="Client" />
|
||||
<Protobuf Include="../../../protos/resource_group.proto" GrpcServices="Client" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
In the `.cs` file that will be calling the Test Engine method, include the following `using` statements:
|
||||
|
||||
```csharp
|
||||
using Grpc.Net.Client;
|
||||
using BenchPress.TestEngine;
|
||||
```
|
||||
|
||||
Your editor's intellisense should now be able to pick up on the objects and interfaces defined in the .proto files.
|
||||
|
||||
Start by creating the gRPC Channel using the port that the Test Engine is running on:
|
||||
|
||||
```csharp
|
||||
using var channel = GrpcChannel.ForAddress("http://localhost:5152");
|
||||
```
|
||||
|
||||
Then, create the client(s) that define the Test Engine method(s) you wish to call:
|
||||
|
||||
```csharp
|
||||
var deployClient = new Deployment.DeploymentClient(channel);
|
||||
var rgClient = new ResourceGroup.ResourceGroupClient(channel);
|
||||
```
|
||||
|
||||
Now you can build your Request objects...
|
||||
|
||||
```csharp
|
||||
var request = new DeploymentGroupRequest {
|
||||
BicepFilePath = "main.bicep",
|
||||
ParameterFilePath = "params.json",
|
||||
ResourceGroupName = "rg-jsmith-benchpress-test",
|
||||
SubscriptionNameOrId = "John-Smiths-Subscription"
|
||||
};
|
||||
```
|
||||
|
||||
... and make calls to the Test Engine
|
||||
|
||||
```csharp
|
||||
var result = await deployClient.DeploymentGroupCreateAsync(request);
|
||||
```
|
||||
|
||||
Altogether, you may have a simple console app `Program.cs` that looks like this:
|
||||
|
||||
```csharp
|
||||
using Grpc.Net.Client;
|
||||
using BenchPress.TestEngine;
|
||||
|
||||
using var channel = GrpcChannel.ForAddress("http://localhost:5152");
|
||||
var client = new Deployment.DeploymentClient(channel);
|
||||
|
||||
var request = new DeploymentSubRequest {
|
||||
BicepFilePath = "<path to your bicep file>",
|
||||
ParameterFilePath = "<path to your params file, if needed>",
|
||||
Location = "eastus",
|
||||
SubscriptionNameOrId = "<your subscription id>"
|
||||
};
|
||||
|
||||
var result = await client.DeploymentSubCreateAsync(request);
|
||||
|
||||
Console.WriteLine($"Success? {result.Success}");
|
||||
Console.WriteLine(result.ErrorMessage);
|
||||
```
|
||||
|
||||
While the Test Engine is running in a separate terminal, run your Client code. Using this Client, you can manually test the Test Engine.
|
||||
|
||||
### Python
|
||||
|
||||
If needed, generate the `pb2` scripts for the `.proto` files you will be using. (They might already exist in under `/framework/python/src/benchpress`).
|
||||
|
||||
```bash
|
||||
python -m grpc_tools.protoc -I./protos --python_out=. --grpc_python_out=. ./protos/deployment.proto
|
||||
```
|
||||
|
||||
The above example ran at the root of the project directory will create a `deployment_pb2.py` and `deployment_pb2_grpc.py` file.
|
||||
|
||||
In the python script you will be using for your tests, include the following import statements (for whatever `pb2` files you are going to use):
|
||||
|
||||
```python
|
||||
import grpc
|
||||
import deployment_pb2
|
||||
import deployment_pb2_grpc
|
||||
import resource_group_pb2
|
||||
import resource_group_pb2_grpc
|
||||
```
|
||||
|
||||
Start by creating the gRPC Channel using the port that the Test Engine is running on:
|
||||
|
||||
```python
|
||||
with grpc.insecure_channel('localhost:5152') as channel:
|
||||
```
|
||||
|
||||
Then, create the client(s) that define the Test Engine method(s) you wish to call:
|
||||
|
||||
```python
|
||||
deployStub = deployment_pb2_grpc.DeploymentStub(channel)
|
||||
rgStub = resource_group_pb2_grpc.ResourceGroupStub(channel)
|
||||
```
|
||||
|
||||
Now you can build your Request objects...
|
||||
|
||||
```python
|
||||
req = deployment_pb2.DeploymentGroupRequest(
|
||||
bicep_file_path = 'main.bicep',
|
||||
parameter_file_path = 'params.json',
|
||||
resource_group_name = 'rg-jsmith-benchpress-test',
|
||||
subscription_name_or_id = 'John-Smiths-Subscription'
|
||||
)
|
||||
```
|
||||
|
||||
... and make calls to the Test Engine
|
||||
|
||||
```python
|
||||
response = deployStub.DeploymentGroupCreate(req)
|
||||
```
|
||||
|
||||
Altogether, you may have a python script like this:
|
||||
|
||||
```python
|
||||
from __future__ import print_function
|
||||
|
||||
import logging
|
||||
|
||||
import grpc
|
||||
import deployment_pb2
|
||||
import deployment_pb2_grpc
|
||||
|
||||
|
||||
def run():
|
||||
with grpc.insecure_channel('localhost:5152') as channel:
|
||||
stub = deployment_pb2_grpc.DeploymentStub(channel)
|
||||
req = deployment_pb2.DeploymentSubRequest(
|
||||
bicep_file_path = '<path to your bicep file>',
|
||||
parameter_file_path = '<path to your params file, if needed>',
|
||||
location = 'eastus',
|
||||
subscription_name_or_id = '<your subscription id>'
|
||||
)
|
||||
response = stub.DeploymentSubCreate(req)
|
||||
print("Success? " + response.success)
|
||||
print(response.error_message)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
logging.basicConfig()
|
||||
run()
|
||||
|
||||
```
|
||||
|
||||
While the Test Engine is running in a separate terminal, run your Client script. Using this Client, you can manually test the Test Engine.
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
using Grpc.Net.Client;
|
||||
using BenchPress.TestEngine;
|
||||
|
||||
using var channel = GrpcChannel.ForAddress("http://localhost:5152");
|
||||
var client = new Deployment.DeploymentClient(channel);
|
||||
|
||||
var request = new DeploymentSubRequest {
|
||||
BicepFilePath = "<path to your bicep file>",
|
||||
ParameterFilePath = "<path to your params file, if needed>",
|
||||
Location = "eastus",
|
||||
SubscriptionNameOrId = "<your subscription id>"
|
||||
};
|
||||
|
||||
var result = await client.DeploymentSubCreateAsync(request);
|
||||
|
||||
Console.WriteLine($"Success? {result.Success}");
|
||||
Console.WriteLine(result.ErrorMessage);
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net6.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Protobuf Include="../../../protos/deployment.proto" GrpcServices="Client" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Google.Protobuf" Version="3.21.6" />
|
||||
<PackageReference Include="Grpc.Net.Client" Version="2.49.0" />
|
||||
<PackageReference Include="Grpc.Tools" Version="2.49.1">
|
||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
</PackageReference>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Generated by the protocol buffer compiler. DO NOT EDIT!
|
||||
# source: deployment.proto
|
||||
"""Generated protocol buffer code."""
|
||||
from google.protobuf.internal import builder as _builder
|
||||
from google.protobuf import descriptor as _descriptor
|
||||
from google.protobuf import descriptor_pool as _descriptor_pool
|
||||
from google.protobuf import symbol_database as _symbol_database
|
||||
# @@protoc_insertion_point(imports)
|
||||
|
||||
_sym_db = _symbol_database.Default()
|
||||
|
||||
|
||||
|
||||
|
||||
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x10\x64\x65ployment.proto\x12\nbenchpress\"\x8c\x01\n\x16\x44\x65ploymentGroupRequest\x12\x17\n\x0f\x62icep_file_path\x18\x01 \x01(\t\x12\x1b\n\x13parameter_file_path\x18\x02 \x01(\t\x12\x1b\n\x13resource_group_name\x18\x03 \x01(\t\x12\x1f\n\x17subscription_name_or_id\x18\x04 \x01(\t\"R\n\x12\x44\x65leteGroupRequest\x12\x1b\n\x13resource_group_name\x18\x01 \x01(\t\x12\x1f\n\x17subscription_name_or_id\x18\x02 \x01(\t\":\n\x10\x44\x65ploymentResult\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x15\n\rerror_message\x18\x02 \x01(\t2\xb4\x01\n\nDeployment\x12Y\n\x15\x44\x65ploymentGroupCreate\x12\".benchpress.DeploymentGroupRequest\x1a\x1c.benchpress.DeploymentResult\x12K\n\x0b\x44\x65leteGroup\x12\x1e.benchpress.DeleteGroupRequest\x1a\x1c.benchpress.DeploymentResultB\x18\xaa\x02\x15\x42\x65nchPress.TestEngineb\x06proto3')
|
||||
|
||||
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
|
||||
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'deployment_pb2', globals())
|
||||
if _descriptor._USE_C_DESCRIPTORS == False:
|
||||
|
||||
DESCRIPTOR._options = None
|
||||
DESCRIPTOR._serialized_options = b'\252\002\025BenchPress.TestEngine'
|
||||
_DEPLOYMENTGROUPREQUEST._serialized_start=33
|
||||
_DEPLOYMENTGROUPREQUEST._serialized_end=173
|
||||
_DELETEGROUPREQUEST._serialized_start=175
|
||||
_DELETEGROUPREQUEST._serialized_end=257
|
||||
_DEPLOYMENTRESULT._serialized_start=259
|
||||
_DEPLOYMENTRESULT._serialized_end=317
|
||||
_DEPLOYMENT._serialized_start=320
|
||||
_DEPLOYMENT._serialized_end=500
|
||||
# @@protoc_insertion_point(module_scope)
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
|
||||
"""Client and server classes corresponding to protobuf-defined services."""
|
||||
import grpc
|
||||
|
||||
import deployment_pb2 as deployment__pb2
|
||||
|
||||
|
||||
class DeploymentStub(object):
|
||||
"""Currently only supports deployments with the target scope of resource group.
|
||||
Other scopes: subscription, management group, and tenant.
|
||||
"""
|
||||
|
||||
def __init__(self, channel):
|
||||
"""Constructor.
|
||||
|
||||
Args:
|
||||
channel: A grpc.Channel.
|
||||
"""
|
||||
self.DeploymentGroupCreate = channel.unary_unary(
|
||||
'/benchpress.Deployment/DeploymentGroupCreate',
|
||||
request_serializer=deployment__pb2.DeploymentGroupRequest.SerializeToString,
|
||||
response_deserializer=deployment__pb2.DeploymentResult.FromString,
|
||||
)
|
||||
self.DeleteGroup = channel.unary_unary(
|
||||
'/benchpress.Deployment/DeleteGroup',
|
||||
request_serializer=deployment__pb2.DeleteGroupRequest.SerializeToString,
|
||||
response_deserializer=deployment__pb2.DeploymentResult.FromString,
|
||||
)
|
||||
|
||||
|
||||
class DeploymentServicer(object):
|
||||
"""Currently only supports deployments with the target scope of resource group.
|
||||
Other scopes: subscription, management group, and tenant.
|
||||
"""
|
||||
|
||||
def DeploymentGroupCreate(self, request, context):
|
||||
"""Missing associated documentation comment in .proto file."""
|
||||
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
|
||||
context.set_details('Method not implemented!')
|
||||
raise NotImplementedError('Method not implemented!')
|
||||
|
||||
def DeleteGroup(self, request, context):
|
||||
"""Missing associated documentation comment in .proto file."""
|
||||
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
|
||||
context.set_details('Method not implemented!')
|
||||
raise NotImplementedError('Method not implemented!')
|
||||
|
||||
|
||||
def add_DeploymentServicer_to_server(servicer, server):
|
||||
rpc_method_handlers = {
|
||||
'DeploymentGroupCreate': grpc.unary_unary_rpc_method_handler(
|
||||
servicer.DeploymentGroupCreate,
|
||||
request_deserializer=deployment__pb2.DeploymentGroupRequest.FromString,
|
||||
response_serializer=deployment__pb2.DeploymentResult.SerializeToString,
|
||||
),
|
||||
'DeleteGroup': grpc.unary_unary_rpc_method_handler(
|
||||
servicer.DeleteGroup,
|
||||
request_deserializer=deployment__pb2.DeleteGroupRequest.FromString,
|
||||
response_serializer=deployment__pb2.DeploymentResult.SerializeToString,
|
||||
),
|
||||
}
|
||||
generic_handler = grpc.method_handlers_generic_handler(
|
||||
'benchpress.Deployment', rpc_method_handlers)
|
||||
server.add_generic_rpc_handlers((generic_handler,))
|
||||
|
||||
|
||||
# This class is part of an EXPERIMENTAL API.
|
||||
class Deployment(object):
|
||||
"""Currently only supports deployments with the target scope of resource group.
|
||||
Other scopes: subscription, management group, and tenant.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def DeploymentGroupCreate(request,
|
||||
target,
|
||||
options=(),
|
||||
channel_credentials=None,
|
||||
call_credentials=None,
|
||||
insecure=False,
|
||||
compression=None,
|
||||
wait_for_ready=None,
|
||||
timeout=None,
|
||||
metadata=None):
|
||||
return grpc.experimental.unary_unary(request, target, '/benchpress.Deployment/DeploymentGroupCreate',
|
||||
deployment__pb2.DeploymentGroupRequest.SerializeToString,
|
||||
deployment__pb2.DeploymentResult.FromString,
|
||||
options, channel_credentials,
|
||||
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
|
||||
|
||||
@staticmethod
|
||||
def DeleteGroup(request,
|
||||
target,
|
||||
options=(),
|
||||
channel_credentials=None,
|
||||
call_credentials=None,
|
||||
insecure=False,
|
||||
compression=None,
|
||||
wait_for_ready=None,
|
||||
timeout=None,
|
||||
metadata=None):
|
||||
return grpc.experimental.unary_unary(request, target, '/benchpress.Deployment/DeleteGroup',
|
||||
deployment__pb2.DeleteGroupRequest.SerializeToString,
|
||||
deployment__pb2.DeploymentResult.FromString,
|
||||
options, channel_credentials,
|
||||
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
# Copyright 2015 gRPC authors.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""The Python implementation of the GRPC helloworld.Greeter client."""
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import logging
|
||||
|
||||
import grpc
|
||||
import deployment_pb2
|
||||
import deployment_pb2_grpc
|
||||
|
||||
|
||||
def run():
|
||||
print("Will try to greet world ...")
|
||||
with grpc.insecure_channel('localhost:5152') as channel:
|
||||
stub = deployment_pb2_grpc.DeploymentStub(channel)
|
||||
req = deployment_pb2.DeploymentGroupRequest(
|
||||
bicep_file_path = '/Users/jessicaern/Projects/benchpress-private/engine/BenchPress.TestEngine.Tests/SampleFiles/storageAccount.bicep',
|
||||
resource_group_name = 'jern-benchpress-playground',
|
||||
subscription_name_or_id = '519c3e33-0884-4604-bad7-6964e6ef55f8'
|
||||
)
|
||||
response = stub.DeploymentGroupCreate(req)
|
||||
print("Success? " + response.success)
|
||||
print(response.error_message)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
logging.basicConfig()
|
||||
run()
|
||||
Загрузка…
Ссылка в новой задаче