microsoft
/
restler-fuzzer
зеркало из https://github.com/microsoft/restler-fuzzer.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Add the ability to track parameter values in the spec coverage file. (#253) 

For large-scale testing, it is useful to have the specific values of
parameters fuzzed in the spec coverage file, in order to facilitate
root cause failure analysis.

This change adds an option to the compiler that produces a grammar which
contains the parameter names to track for every fuzzable value.

Testing:
- manual testing
- added/fixed engine tests to make sure new code path is exercised
- Note: the payload body checker code is not modified in this change because
it does not require this feature.
This commit is contained in:
marina-p 2021-06-30 15:32:03 -07:00 коммит произвёл GitHub
Родитель 89279e0869
Коммит e2b25ebe16
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
16 изменённых файлов: 289 добавлений и 113 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

5
docs/user-guide/CompilerConfig.md
Просмотреть файл

@ -96,4 +96,7 @@ catch consistency bugs in the specification because producer-consumer dependenci
PascalCase PascalCase
HyphenSeparator HyphenSeparator
UnderscoreSeparator UnderscoreSeparator
``` ```
* *TrackFuzzedParameterNames* False by default. When true, every fuzzable primitive will
include an additional parameter `param_name` which is the name of the property or
parameter being fuzzed. These will be used to capture fuzzed parameters in ```tracked_parameters``` in the spec coverage file.

7
docs/user-guide/Testing.md
Просмотреть файл

@ -105,7 +105,8 @@ During each Test run a `speccov.json` file will be created in the logs directory
"response_body": "{\n \"errors\": {\n \"id\": \"'5882' is not of type 'integer'\"\n },\n \"message\": \"Input payload validation failed\"\n}\n" "response_body": "{\n \"errors\": {\n \"id\": \"'5882' is not of type 'integer'\"\n },\n \"message\": \"Input payload validation failed\"\n}\n"
}, },
"tracked_parameters": { "tracked_parameters": {
"per_page_9": "2" "per_page": ["2"],
"page": ["1"]
} }
}, },
``` ```
@ -135,7 +136,9 @@ the coverage data is being reported.
* The __"tracked_parameters"__ property is optional and generated only when using * The __"tracked_parameters"__ property is optional and generated only when using
`Test` mode with ```test-all-combinations```. `Test` mode with ```test-all-combinations```.
This property contains key-value pairs of all of the parameters This property contains key-value pairs of all of the parameters
for which more than one value is being tested. for which more than one value is being tested. By default, enums and custom payloads
are always tracked. In addition, when the specification was compiled with
`TrackFuzzedParameterNames` set to `true`, all fuzzable parameters will be tracked.
#### Postprocessing Scripts: #### Postprocessing Scripts:
The `utilities` directory contains a sub-directory called `speccovparsing` that contains scripts for postprocessing speccov files. The `utilities` directory contains a sub-directory called `speccovparsing` that contains scripts for postprocessing speccov files.

36
restler/engine/core/requests.py
Просмотреть файл

@ -602,11 +602,10 @@ class Request(object):
quoted = request_block[2] quoted = request_block[2]
examples = request_block[3] examples = request_block[3]
else: else:
field_name = None
default_val = request_block[1] default_val = request_block[1]
quoted = request_block[2] quoted = request_block[2]
examples = request_block[3] examples = request_block[3]
field_name = request_block[4]
values = [] values = []
# Handling dynamic primitives that need fresh rendering every time # Handling dynamic primitives that need fresh rendering every time
if primitive_type == primitives.FUZZABLE_UUID4: if primitive_type == primitives.FUZZABLE_UUID4:
@ -696,9 +695,10 @@ class Request(object):
# Only track the parameter if there are multiple values being combined # Only track the parameter if there are multiple values being combined
if len(values) > 1: if len(values) > 1:
if not field_name: if not field_name:
field_name = "tracked_param" field_name = f"tracked_param_{param_idx}"
field_name = f"{field_name}_{param_idx}" if field_name not in tracked_parameters:
tracked_parameters[field_name] = param_idx tracked_parameters[field_name] = []
tracked_parameters[field_name].append(param_idx)
fuzzable.append(values) fuzzable.append(values)
@ -718,9 +718,10 @@ class Request(object):
values = request_utilities.resolve_dynamic_primitives(values, candidate_values_pool) values = request_utilities.resolve_dynamic_primitives(values, candidate_values_pool)
tracked_parameter_values = {} tracked_parameter_values = {}
for (k, idx) in tracked_parameters.items(): for (k, idx_list) in tracked_parameters.items():
tracked_parameter_values[k] = values[idx] tracked_parameter_values[k] = []
for idx in idx_list:
tracked_parameter_values[k].append(values[idx])
rendered_data = "".join(values) rendered_data = "".join(values)
yield rendered_data, parser, tracked_parameter_values yield rendered_data, parser, tracked_parameter_values
@ -764,6 +765,25 @@ class Request(object):
return self._total_feasible_combinations return self._total_feasible_combinations
def update_tracked_parameters(self, tracked_parameters):
""" Updates tracked parameters for the request by merging with the
existing parameters.
The tracked parameters are currently a set of names, with arrays of
values that were observed during sequence rendering.
@param tracked_parameters: The tracked parameters and their values.
@type tracked_parameters: Dict
@return: None
@rtype : None
"""
for param_name, param_val_list in tracked_parameters.items():
for param_val in param_val_list:
if param_name not in self._tracked_parameters:
self._tracked_parameters[param_name] = []
self._tracked_parameters[param_name].append(param_val)
def GrammarRequestCollection(): def GrammarRequestCollection():
""" Accessor for the global request collection singleton """ """ Accessor for the global request collection singleton """
return GlobalRequestCollection.Instance()._req_collection return GlobalRequestCollection.Instance()._req_collection

5
restler/engine/core/sequences.py
Просмотреть файл

@ -350,7 +350,8 @@ class Sequence(object):
dependencies.reset_tlb() dependencies.reset_tlb()
sequence_failed = False sequence_failed = False
request._tracked_parameters = tracked_parameters request._tracked_parameters = {}
request.update_tracked_parameters(tracked_parameters)
# Step A: Static template rendering # Step A: Static template rendering
# Render last known valid combination of primitive type values # Render last known valid combination of primitive type values
# for every request until the last # for every request until the last
@ -360,7 +361,7 @@ class Sequence(object):
prev_request.render_current(candidate_values_pool, prev_request.render_current(candidate_values_pool,
preprocessing=preprocessing) preprocessing=preprocessing)
request._tracked_parameters.update(tracked_parameters) request.update_tracked_parameters(tracked_parameters)
# substitute reference placeholders with resolved values # substitute reference placeholders with resolved values
if not Settings().ignore_dependencies: if not Settings().ignore_dependencies:

2
restler/engine/fuzzing_parameters/body_schema.py
Просмотреть файл

@ -201,10 +201,10 @@ class BodySchema():
examples = request_block[3] examples = request_block[3]
value = None value = None
else: else:
field_name = None
value = request_block[1] value = request_block[1]
quoted = request_block[2] quoted = request_block[2]
examples = request_block[3] examples = request_block[3]
field_name = request_block[4]
# accumulate # accumulate
if primitive_type == primitives.STATIC_STRING: if primitive_type == primitives.STATIC_STRING:

65
restler/engine/primitives.py
Просмотреть файл

@ -51,6 +51,11 @@ UNQUOTED_STR = '_unquoted'
# Optional argument passed to fuzzable primitive definition function that can # Optional argument passed to fuzzable primitive definition function that can
# provide an example value. This value is used instead of the default value if present. # provide an example value. This value is used instead of the default value if present.
EXAMPLES_ARG = 'examples' EXAMPLES_ARG = 'examples'
# Optional argument passed to fuzzable primitive definition function that can
# provide the name of the parameter being fuzzed.
# This value is used in test-all-combinations mode to allow the user to analyze spec coverage
# for particular parameter values.
PARAM_NAME_ARG = 'param_name'
class CandidateValues(object): class CandidateValues(object):
def __init__(self): def __init__(self):
self.unquoted_values = [] self.unquoted_values = []
@ -374,7 +379,8 @@ def restler_static_string(*args, **kwargs):
if QUOTED_ARG in kwargs: if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG] quoted = kwargs[QUOTED_ARG]
examples = None examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_string(*args, **kwargs): def restler_fuzzable_string(*args, **kwargs):
@ -400,7 +406,10 @@ def restler_fuzzable_string(*args, **kwargs):
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_int(*args, **kwargs): def restler_fuzzable_int(*args, **kwargs):
""" Integer primitive. """ Integer primitive.
@ -425,7 +434,10 @@ def restler_fuzzable_int(*args, **kwargs):
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_bool(*args, **kwargs): def restler_fuzzable_bool(*args, **kwargs):
@ -451,7 +463,10 @@ def restler_fuzzable_bool(*args, **kwargs):
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_number(*args, **kwargs): def restler_fuzzable_number(*args, **kwargs):
@ -477,7 +492,10 @@ def restler_fuzzable_number(*args, **kwargs):
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_delim(*args, **kwargs): def restler_fuzzable_delim(*args, **kwargs):
@ -503,7 +521,10 @@ def restler_fuzzable_delim(*args, **kwargs):
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_group(*args, **kwargs): def restler_fuzzable_group(*args, **kwargs):
@ -536,7 +557,8 @@ def restler_fuzzable_group(*args, **kwargs):
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, enum_vals, quoted, examples param_name = None
return sys._getframe().f_code.co_name, field_name, enum_vals, quoted, examples, param_name
def restler_fuzzable_uuid4(*args, **kwargs): def restler_fuzzable_uuid4(*args, **kwargs):
@ -562,7 +584,10 @@ def restler_fuzzable_uuid4(*args, **kwargs):
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_datetime(*args, **kwargs) : def restler_fuzzable_datetime(*args, **kwargs) :
@ -589,7 +614,10 @@ def restler_fuzzable_datetime(*args, **kwargs) :
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_object(*args, **kwargs) : def restler_fuzzable_object(*args, **kwargs) :
""" object primitive ({}) """ object primitive ({})
@ -614,7 +642,9 @@ def restler_fuzzable_object(*args, **kwargs) :
examples=None examples=None
if EXAMPLES_ARG in kwargs: if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG] examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_multipart_formdata(*args, **kwargs): def restler_multipart_formdata(*args, **kwargs):
""" Multipart/formdata primitive """ Multipart/formdata primitive
@ -638,7 +668,8 @@ def restler_multipart_formdata(*args, **kwargs):
if QUOTED_ARG in kwargs: if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG] quoted = kwargs[QUOTED_ARG]
examples = None examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_custom_payload(*args, **kwargs): def restler_custom_payload(*args, **kwargs):
@ -662,7 +693,8 @@ def restler_custom_payload(*args, **kwargs):
if QUOTED_ARG in kwargs: if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG] quoted = kwargs[QUOTED_ARG]
examples = None examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_custom_payload_header(*args, **kwargs): def restler_custom_payload_header(*args, **kwargs):
@ -686,7 +718,8 @@ def restler_custom_payload_header(*args, **kwargs):
if QUOTED_ARG in kwargs: if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG] quoted = kwargs[QUOTED_ARG]
examples = None examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_custom_payload_uuid4_suffix(*args, **kwargs): def restler_custom_payload_uuid4_suffix(*args, **kwargs):
@ -710,7 +743,8 @@ def restler_custom_payload_uuid4_suffix(*args, **kwargs):
if QUOTED_ARG in kwargs: if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG] quoted = kwargs[QUOTED_ARG]
examples = None examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_refreshable_authentication_token(*args, **kwargs): def restler_refreshable_authentication_token(*args, **kwargs):
@ -734,4 +768,5 @@ def restler_refreshable_authentication_token(*args, **kwargs):
if QUOTED_ARG in kwargs: if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG] quoted = kwargs[QUOTED_ARG]
examples = None examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name

4
restler/unit_tests/log_baseline_test_files/test_grammar.json
Просмотреть файл

@ -137,7 +137,9 @@
"payload": { "payload": {
"Fuzzable": [ "Fuzzable": [
"Int", "Int",
"1000" "1000",
null,
"population"
] ]
}, },
"isRequired": false, "isRequired": false,

10
restler/unit_tests/log_baseline_test_files/test_grammar.py
Просмотреть файл

@ -223,8 +223,12 @@ request = requests.Request([
primitives.restler_static_string("\r\n"), primitives.restler_static_string("\r\n"),
primitives.restler_static_string("{"), primitives.restler_static_string("{"),
primitives.restler_static_string('"population":'), primitives.restler_static_string('"population":'),
primitives.restler_fuzzable_int('10000', quoted=True), primitives.restler_fuzzable_int('10000', quoted=True, param_name="population"),
primitives.restler_static_string(', "area": "5000",'), primitives.restler_static_string(', "area": "5000",'),
# Note: There is a discrepancy here due to
# manual creation of this grammar - the fuzzable string
# does not match the grammar.json, which contains
# a constant string for this value.
primitives.restler_fuzzable_string('strtest', quoted=True), primitives.restler_fuzzable_string('strtest', quoted=True),
primitives.restler_static_string(':'), primitives.restler_static_string(':'),
primitives.restler_fuzzable_bool('true', quoted=True), primitives.restler_fuzzable_bool('true', quoted=True),
@ -626,7 +630,7 @@ request = requests.Request([
primitives.restler_static_string("{"), primitives.restler_static_string("{"),
primitives.restler_static_string('testbool', quoted=True), primitives.restler_static_string('testbool', quoted=True),
primitives.restler_static_string(':'), primitives.restler_static_string(':'),
primitives.restler_fuzzable_bool("testval", quoted=True), primitives.restler_fuzzable_bool("testval", quoted=True, param_name="testbool"),
primitives.restler_static_string(',"location":'), primitives.restler_static_string(',"location":'),
primitives.restler_custom_payload("location", quoted=True), primitives.restler_custom_payload("location", quoted=True),
primitives.restler_static_string("}"), primitives.restler_static_string("}"),
@ -849,7 +853,7 @@ request = requests.Request([
primitives.restler_static_string("\r\n"), primitives.restler_static_string("\r\n"),
primitives.restler_static_string("{"), primitives.restler_static_string("{"),
primitives.restler_static_string('"datetest":'), primitives.restler_static_string('"datetest":'),
primitives.restler_fuzzable_datetime("2020-1-1", quoted=True), primitives.restler_fuzzable_datetime("2020-1-1", quoted=True, example=["2020-2-2"], param_name="datetest"),
primitives.restler_static_string(',"id":'), primitives.restler_static_string(',"id":'),
primitives.restler_static_string('"/testparts/'), primitives.restler_static_string('"/testparts/'),
primitives.restler_custom_payload("testcustomparts", quoted=False), primitives.restler_custom_payload("testcustomparts", quoted=False),

2
restler/utils/logger.py
Просмотреть файл

@ -480,10 +480,10 @@ def custom_network_logging(sequence, candidate_values_pool, **kwargs):
quoted = request_block[2] quoted = request_block[2]
examples = request_block[3] examples = request_block[3]
else: else:
field_name = None
default_val = request_block[1] default_val = request_block[1]
quoted = request_block[2] quoted = request_block[2]
examples = request_block[3] examples = request_block[3]
field_name = request_block[4]
# Handling dynamic primitives that need fresh rendering every time # Handling dynamic primitives that need fresh rendering every time
if primitive == "restler_fuzzable_uuid4": if primitive == "restler_fuzzable_uuid4":

36
src/compiler/Restler.Compiler.Test/CodeGeneratorTests.fs
Просмотреть файл

@ -5,6 +5,7 @@ namespace Restler.Test
open System.IO open System.IO
open Xunit open Xunit
open System
open Restler.Grammar open Restler.Grammar
open Restler.CodeGenerator.Python.Types open Restler.CodeGenerator.Python.Types
open Tree open Tree
@ -23,7 +24,7 @@ module CodeGenerator =
/// This is useful because the grammar json is validated /// This is useful because the grammar json is validated
/// when deserialized by the RESTler compiler, avoiding typos made directly in python. /// when deserialized by the RESTler compiler, avoiding typos made directly in python.
/// This also sanity checks that the grammar generation is deterministic. /// This also sanity checks that the grammar generation is deterministic.
member __.``generate code from modified grammar`` () = let ``generate code from modified grammar`` () =
let grammarOutputDirectory1 = ctx.testRootDirPath let grammarOutputDirectory1 = ctx.testRootDirPath
Restler.Workflow.generateRestlerGrammar None Restler.Workflow.generateRestlerGrammar None
@ -42,7 +43,7 @@ module CodeGenerator =
[<Fact>] [<Fact>]
member __.``python grammar request sanity test`` () = let ``python grammar request sanity test`` () =
let q1 = LeafNode { LeafProperty.name = "page"; payload = Constant (Int, "1") ;isRequired = true ; isReadOnly = false } let q1 = LeafNode { LeafProperty.name = "page"; payload = Constant (Int, "1") ;isRequired = true ; isReadOnly = false }
let q2 = LeafNode { LeafProperty.name = "page"; payload = Constant (Int, "1") ;isRequired = true ; isReadOnly = false } let q2 = LeafNode { LeafProperty.name = "page"; payload = Constant (Int, "1") ;isRequired = true ; isReadOnly = false }
let b1 = LeafNode { LeafProperty.name = "payload"; payload = Constant (PrimitiveType.String, "hello") ; let b1 = LeafNode { LeafProperty.name = "payload"; payload = Constant (PrimitiveType.String, "hello") ;
@ -88,7 +89,7 @@ module CodeGenerator =
Assert.True(elements |> Seq.length > 0) Assert.True(elements |> Seq.length > 0)
[<Fact>] [<Fact>]
member __.``python grammar parameter sanity test`` () = let ``python grammar parameter sanity test`` () =
let p1 = { LeafProperty.name = "region"; payload = Constant (PrimitiveType.String, "WestUS"); let p1 = { LeafProperty.name = "region"; payload = Constant (PrimitiveType.String, "WestUS");
isRequired = true ; isReadOnly = false} isRequired = true ; isReadOnly = false}
let p2 = { LeafProperty.name = "maxResources"; payload = Constant (Int, "10") ; let p2 = { LeafProperty.name = "maxResources"; payload = Constant (Int, "10") ;
@ -112,5 +113,34 @@ module CodeGenerator =
let hasResources = result |> Seq.tryFind (fun s -> s = (Restler_static_string_constant "\"maxResources\":")) let hasResources = result |> Seq.tryFind (fun s -> s = (Restler_static_string_constant "\"maxResources\":"))
Assert.True(hasResources.IsSome, "resources not found") Assert.True(hasResources.IsSome, "resources not found")
// Test that the generated python and json grammars for tracked parameters are correct
[<Fact>]
let ``tracked parameters tests`` () =
let grammarDirectoryPath = ctx.testRootDirPath
let config = { Restler.Config.SampleConfig with
IncludeOptionalParameters = true
GrammarOutputDirectoryPath = Some grammarDirectoryPath
ResolveBodyDependencies = true
UseBodyExamples = Some false
SwaggerSpecFilePath = Some [(Path.Combine(Environment.CurrentDirectory, @"swagger\example_demo1.json"))]
CustomDictionaryFilePath = Some (Path.Combine(Environment.CurrentDirectory, @"swagger\example_demo_dictionary.json"))
}
Restler.Workflow.generateRestlerGrammar None config
let grammarFilePath = Path.Combine(grammarDirectoryPath, "grammar.py")
let grammar = File.ReadAllText(grammarFilePath)
// The grammar should not contain gracked parameters by default
Assert.False(grammar.Contains("param_name="))
// Now turn on parameter tracking
let config = { config with TrackFuzzedParameterNames = true }
Restler.Workflow.generateRestlerGrammar None config
let grammarFilePath = Path.Combine(grammarDirectoryPath, "grammar.py")
let grammar = File.ReadAllText(grammarFilePath)
Assert.True(grammar.Contains("param_name=\"storeId\""))
Assert.True(grammar.Contains("param_name=\"bannedBrands\""))
Assert.True(grammar.Contains("param_name=\"groceryItemTags\""))
interface IClassFixture<Fixtures.TestSetupAndCleanup> interface IClassFixture<Fixtures.TestSetupAndCleanup>

55
src/compiler/Restler.Compiler/CodeGenerator.fs
Просмотреть файл

@ -21,6 +21,7 @@ module Types =
defaultValue : string defaultValue : string
isQuoted : bool isQuoted : bool
exampleValue : string option exampleValue : string option
trackedParameterName: string option
} }
/// RESTler grammar built-in types /// RESTler grammar built-in types
@ -62,18 +63,18 @@ let rec getRestlerPythonPayload (payload:FuzzingPayload) (isQuoted:bool) : Reque
match p with match p with
| Constant (t,v) -> | Constant (t,v) ->
Restler_static_string_constant v Restler_static_string_constant v
| Fuzzable (t,v,exv) -> | Fuzzable (t,v,exv,parameterName) ->
match t with match t with
| Bool -> Restler_fuzzable_bool { defaultValue = v ; isQuoted = false ; exampleValue = exv } | Bool -> Restler_fuzzable_bool { defaultValue = v ; isQuoted = false ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.DateTime -> | PrimitiveType.DateTime ->
Restler_fuzzable_datetime { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv } Restler_fuzzable_datetime { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.String -> | PrimitiveType.String ->
Restler_fuzzable_string { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv } Restler_fuzzable_string { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.Object -> Restler_fuzzable_object { defaultValue = v ; isQuoted = false ; exampleValue = exv } | PrimitiveType.Object -> Restler_fuzzable_object { defaultValue = v ; isQuoted = false ; exampleValue = exv ; trackedParameterName = parameterName }
| Int -> Restler_fuzzable_int { defaultValue = v ; isQuoted = false; exampleValue = exv } | Int -> Restler_fuzzable_int { defaultValue = v ; isQuoted = false; exampleValue = exv ; trackedParameterName = parameterName }
| Number -> Restler_fuzzable_number { defaultValue = v ; isQuoted = false ; exampleValue = exv } | Number -> Restler_fuzzable_number { defaultValue = v ; isQuoted = false ; exampleValue = exv ; trackedParameterName = parameterName }
| Uuid -> | Uuid ->
Restler_fuzzable_uuid4 { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv } Restler_fuzzable_uuid4 { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.Enum (enumPropertyName, _, enumeration, defaultValue) -> | PrimitiveType.Enum (enumPropertyName, _, enumeration, defaultValue) ->
let defaultStr = let defaultStr =
match defaultValue with match defaultValue with
@ -86,11 +87,11 @@ let rec getRestlerPythonPayload (payload:FuzzingPayload) (isQuoted:bool) : Reque
defaultStr defaultStr
) )
Restler_fuzzable_group Restler_fuzzable_group
{ defaultValue = groupValue ; isQuoted = isQuoted ; exampleValue = exv } { defaultValue = groupValue ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| Custom c -> | Custom c ->
match c.payloadType with match c.payloadType with
| CustomPayloadType.String -> | CustomPayloadType.String ->
Restler_custom_payload { defaultValue = c.payloadValue ; isQuoted = isQuoted ; exampleValue = None } Restler_custom_payload { defaultValue = c.payloadValue ; isQuoted = isQuoted ; exampleValue = None ; trackedParameterName = None }
| CustomPayloadType.UuidSuffix -> | CustomPayloadType.UuidSuffix ->
Restler_custom_payload_uuid4_suffix c.payloadValue Restler_custom_payload_uuid4_suffix c.payloadValue
| CustomPayloadType.Header -> | CustomPayloadType.Header ->
@ -335,7 +336,7 @@ let generatePythonParameter includeOptionalParameters parameterKind (requestPara
| FuzzingPayload.Constant (primitiveType, v) -> | FuzzingPayload.Constant (primitiveType, v) ->
isPrimitiveTypeQuoted primitiveType (isNull v), isPrimitiveTypeQuoted primitiveType (isNull v),
false false
| FuzzingPayload.Fuzzable (primitiveType, _, _) -> | FuzzingPayload.Fuzzable (primitiveType, _, _,_) ->
// Note: this is a current RESTler limitation - // Note: this is a current RESTler limitation -
// fuzzable values may not be set to null without changing the grammar. // fuzzable values may not be set to null without changing the grammar.
isPrimitiveTypeQuoted primitiveType false, isPrimitiveTypeQuoted primitiveType false,
@ -427,7 +428,7 @@ let generatePythonFromRequestElement includeOptionalParameters (e:RequestElement
) )
// Handle the case of '/' // Handle the case of '/'
if x |> List.isEmpty then if x |> List.isEmpty then
[ Restler_static_string_constant "/" ] [ Restler_static_string_constant "/" ]
else else
x x
| HeaderParameters hp -> | HeaderParameters hp ->
@ -826,6 +827,14 @@ let getRequests(requests:Request list) includeOptionalParameters =
let quotedStr = sprintf "%s%s%s" exDelim exStr exDelim let quotedStr = sprintf "%s%s%s" exDelim exStr exDelim
sprintf ", examples=[%s]" quotedStr sprintf ", examples=[%s]" quotedStr
let getTrackedParamPrimitiveParameter paramName =
match paramName with
| None -> ""
| Some str ->
let exStr, exDelim = quoteStringForPythonGrammar str
let quotedStr = sprintf "%s%s%s" exDelim exStr exDelim
sprintf ", param_name=%s" quotedStr
let str = let str =
match p with match p with
| Restler_static_string_jtoken_delim s -> | Restler_static_string_jtoken_delim s ->
@ -856,32 +865,38 @@ let getRequests(requests:Request list) includeOptionalParameters =
let quotedDefaultString = let quotedDefaultString =
sprintf "%s%s%s" delim str delim sprintf "%s%s%s" delim str delim
let exampleParameter = getExamplePrimitiveParameter s.exampleValue let exampleParameter = getExamplePrimitiveParameter s.exampleValue
sprintf "primitives.restler_fuzzable_string(%s, quoted=%s%s)" let trackedParamName = getTrackedParamPrimitiveParameter s.trackedParameterName
sprintf "primitives.restler_fuzzable_string(%s, quoted=%s%s%s)"
quotedDefaultString quotedDefaultString
(if s.isQuoted then "True" else "False") (if s.isQuoted then "True" else "False")
exampleParameter exampleParameter
trackedParamName
| Restler_fuzzable_group s -> | Restler_fuzzable_group s ->
sprintf "primitives.restler_fuzzable_group(%s,quoted=%s%s)" sprintf "primitives.restler_fuzzable_group(%s,quoted=%s%s)"
s.defaultValue s.defaultValue
(if s.isQuoted then "True" else "False") (if s.isQuoted then "True" else "False")
(getExamplePrimitiveParameter s.exampleValue) (getExamplePrimitiveParameter s.exampleValue)
| Restler_fuzzable_int s -> | Restler_fuzzable_int s ->
sprintf "primitives.restler_fuzzable_int(\"%s\"%s)" sprintf "primitives.restler_fuzzable_int(\"%s\"%s%s)"
s.defaultValue s.defaultValue
(getExamplePrimitiveParameter s.exampleValue) (getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_number s -> | Restler_fuzzable_number s ->
sprintf "primitives.restler_fuzzable_number(\"%s\"%s)" sprintf "primitives.restler_fuzzable_number(\"%s\"%s%s)"
s.defaultValue s.defaultValue
(getExamplePrimitiveParameter s.exampleValue) (getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_bool s -> | Restler_fuzzable_bool s ->
sprintf "primitives.restler_fuzzable_bool(\"%s\"%s)" sprintf "primitives.restler_fuzzable_bool(\"%s\"%s%s)"
s.defaultValue s.defaultValue
(getExamplePrimitiveParameter s.exampleValue) (getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_datetime s -> | Restler_fuzzable_datetime s ->
sprintf "primitives.restler_fuzzable_datetime(\"%s\", quoted=%s%s)" sprintf "primitives.restler_fuzzable_datetime(\"%s\", quoted=%s%s%s)"
s.defaultValue s.defaultValue
(if s.isQuoted then "True" else "False") (if s.isQuoted then "True" else "False")
(getExamplePrimitiveParameter s.exampleValue) (getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_object s -> | Restler_fuzzable_object s ->
if String.IsNullOrEmpty s.defaultValue then if String.IsNullOrEmpty s.defaultValue then
printfn "ERROR: fuzzable objects should not be empty. Skipping." printfn "ERROR: fuzzable objects should not be empty. Skipping."
@ -891,14 +906,16 @@ let getRequests(requests:Request list) includeOptionalParameters =
let quotedDefaultString = let quotedDefaultString =
sprintf "%s%s%s" delim str delim sprintf "%s%s%s" delim str delim
let exampleParameter = getExamplePrimitiveParameter s.exampleValue let exampleParameter = getExamplePrimitiveParameter s.exampleValue
sprintf "primitives.restler_fuzzable_object(%s%s)" sprintf "primitives.restler_fuzzable_object(%s%s%s)"
quotedDefaultString quotedDefaultString
exampleParameter exampleParameter
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_uuid4 s -> | Restler_fuzzable_uuid4 s ->
sprintf "primitives.restler_fuzzable_uuid4(\"%s\", quoted=%s%s)" sprintf "primitives.restler_fuzzable_uuid4(\"%s\", quoted=%s%s%s)"
s.defaultValue s.defaultValue
(if s.isQuoted then "True" else "False") (if s.isQuoted then "True" else "False")
(getExamplePrimitiveParameter s.exampleValue) (getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_custom_payload p -> | Restler_custom_payload p ->
sprintf "primitives.restler_custom_payload(\"%s\", quoted=%s)" sprintf "primitives.restler_custom_payload(\"%s\", quoted=%s)"
p.defaultValue p.defaultValue

41
src/compiler/Restler.Compiler/Compiler.fs
Просмотреть файл

@ -156,7 +156,8 @@ module private Parameters =
| _ -> raise (UnsupportedType "Complex path parameters are not supported") | _ -> raise (UnsupportedType "Complex path parameters are not supported")
let private getParametersFromExample (examplePayload:ExampleRequestPayload) let private getParametersFromExample (examplePayload:ExampleRequestPayload)
(parameterList:seq<OpenApiParameter>) = (parameterList:seq<OpenApiParameter>)
(trackParameters:bool) =
parameterList parameterList
|> Seq.choose (fun declaredParameter -> |> Seq.choose (fun declaredParameter ->
// If the declared parameter isn't in the example, skip it. Here, the example is used to // If the declared parameter isn't in the example, skip it. Here, the example is used to
@ -169,7 +170,7 @@ module private Parameters =
| PayloadFormat.JToken payloadValue -> | PayloadFormat.JToken payloadValue ->
let parameterGrammarElement = let parameterGrammarElement =
generateGrammarElementForSchema declaredParameter.ActualSchema generateGrammarElementForSchema declaredParameter.ActualSchema
(Some payloadValue, false) [] id (Some payloadValue, false) trackParameters [] id
Some { name = declaredParameter.Name Some { name = declaredParameter.Name
payload = parameterGrammarElement payload = parameterGrammarElement
serialization = getParameterSerialization declaredParameter } serialization = getParameterSerialization declaredParameter }
@ -205,7 +206,8 @@ module private Parameters =
None None
let pathParameters (swaggerMethodDefinition:OpenApiOperation) (endpoint:string) let pathParameters (swaggerMethodDefinition:OpenApiOperation) (endpoint:string)
(exampleConfig: ExampleRequestPayload list option) = (exampleConfig: ExampleRequestPayload list option)
(trackParameters:bool) =
let declaredPathParameters = swaggerMethodDefinition.ActualParameters let declaredPathParameters = swaggerMethodDefinition.ActualParameters
|> Seq.filter (fun p -> p.Kind = NSwag.OpenApiParameterKind.Path) |> Seq.filter (fun p -> p.Kind = NSwag.OpenApiParameterKind.Path)
@ -250,19 +252,22 @@ module private Parameters =
(tryGetEnumeration schema) (tryGetEnumeration schema)
(tryGetDefault schema) (tryGetDefault schema)
specExampleValue specExampleValue
trackParameters
Some { name = parameterName Some { name = parameterName
payload = LeafNode leafProperty payload = LeafNode leafProperty
serialization = serialization } serialization = serialization }
| Some (firstExample::remainingExamples) -> | Some (firstExample::remainingExamples) ->
// Use the first example specified to determine the parameter value. // Use the first example specified to determine the parameter value.
getParametersFromExample firstExample (parameter |> stn) getParametersFromExample firstExample (parameter |> stn) trackParameters
|> Seq.head |> Seq.head
|> Some |> Some
) )
ParameterList parameterList ParameterList parameterList
let private getParameters (parameterList:seq<OpenApiParameter>) let private getParameters (parameterList:seq<OpenApiParameter>)
(exampleConfig:ExampleRequestPayload list option) (dataFuzzing:bool) = (exampleConfig:ExampleRequestPayload list option)
(dataFuzzing:bool)
(trackParameters:bool) =
// When data fuzzing is specified, both the full schema and examples should be available for analysis. // When data fuzzing is specified, both the full schema and examples should be available for analysis.
// Otherwise, use the first example if it exists, or the schema, and return a single schema. // Otherwise, use the first example if it exists, or the schema, and return a single schema.
@ -272,9 +277,9 @@ module private Parameters =
| Some [] -> None | Some [] -> None
| Some (firstExample::remainingExamples) -> | Some (firstExample::remainingExamples) ->
// Use the first example specified to determine the parameter schema and values. // Use the first example specified to determine the parameter schema and values.
let firstPayload = getParametersFromExample firstExample parameterList let firstPayload = getParametersFromExample firstExample parameterList trackParameters
let restOfPayloads = let restOfPayloads =
remainingExamples |> List.map (fun e -> getParametersFromExample e parameterList) remainingExamples |> List.map (fun e -> getParametersFromExample e parameterList trackParameters)
Some (firstPayload::restOfPayloads) Some (firstPayload::restOfPayloads)
let schemaPayload = let schemaPayload =
@ -289,15 +294,19 @@ module private Parameters =
let parameterPayload = generateGrammarElementForSchema let parameterPayload = generateGrammarElementForSchema
p.ActualSchema p.ActualSchema
(specExampleValue, true) [] id (specExampleValue, true)
trackParameters
[] id
// Add the name to the parameter payload // Add the name to the parameter payload
let parameterPayload = let parameterPayload =
match parameterPayload with match parameterPayload with
| LeafNode leafProperty -> | LeafNode leafProperty ->
let leafNodePayload = let leafNodePayload =
match leafProperty.payload with match leafProperty.payload with
| Fuzzable (Enum(propertyName, propertyType, values, defaultValue), x, y) -> | Fuzzable (Enum(propertyName, propertyType, values, defaultValue), x, y, z) ->
Fuzzable (Enum(p.Name, propertyType, values, defaultValue), x, y) Fuzzable (Enum(p.Name, propertyType, values, defaultValue), x, y, z)
| Fuzzable (a, b, c, _) ->
Fuzzable (a, b, c, if trackParameters then Some p.Name else None)
| _ -> leafProperty.payload | _ -> leafProperty.payload
LeafNode { leafProperty with payload = leafNodePayload } LeafNode { leafProperty with payload = leafNodePayload }
| InternalNode (internalNode, children) -> | InternalNode (internalNode, children) ->
@ -333,7 +342,8 @@ module private Parameters =
let getAllParameters (swaggerMethodDefinition:OpenApiOperation) let getAllParameters (swaggerMethodDefinition:OpenApiOperation)
(parameterKind:NSwag.OpenApiParameterKind) (parameterKind:NSwag.OpenApiParameterKind)
exampleConfig dataFuzzing = exampleConfig dataFuzzing
trackParameters =
let localParameters = swaggerMethodDefinition.ActualParameters let localParameters = swaggerMethodDefinition.ActualParameters
|> Seq.filter (fun p -> p.Kind = parameterKind) |> Seq.filter (fun p -> p.Kind = parameterKind)
// add shared parameters for the endpoint, if any // add shared parameters for the endpoint, if any
@ -342,7 +352,7 @@ module private Parameters =
let allParameters = let allParameters =
[localParameters ; declaredSharedParameters ] |> Seq.concat [localParameters ; declaredSharedParameters ] |> Seq.concat
getParameters allParameters exampleConfig dataFuzzing getParameters allParameters exampleConfig dataFuzzing trackParameters
let generateRequestPrimitives (requestId:RequestId) let generateRequestPrimitives (requestId:RequestId)
(responseParser:ResponseParser option) (responseParser:ResponseParser option)
@ -615,6 +625,7 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
Parameters.pathParameters Parameters.pathParameters
m.Value ep m.Value ep
(if usePathExamples then exampleConfig else None) (if usePathExamples then exampleConfig else None)
config.TrackFuzzedParameterNames
RequestParameters.header = RequestParameters.header =
let useHeaderExamples = let useHeaderExamples =
config.UseHeaderExamples |> Option.defaultValue false config.UseHeaderExamples |> Option.defaultValue false
@ -623,7 +634,7 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
OpenApiParameterKind.Header OpenApiParameterKind.Header
(if useHeaderExamples then exampleConfig else None) (if useHeaderExamples then exampleConfig else None)
config.DataFuzzing config.DataFuzzing
config.TrackFuzzedParameterNames
RequestParameters.query = RequestParameters.query =
let useQueryExamples = let useQueryExamples =
config.UseQueryExamples |> Option.defaultValue false config.UseQueryExamples |> Option.defaultValue false
@ -632,6 +643,7 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
OpenApiParameterKind.Query OpenApiParameterKind.Query
(if useQueryExamples then exampleConfig else None) (if useQueryExamples then exampleConfig else None)
config.DataFuzzing config.DataFuzzing
config.TrackFuzzedParameterNames
RequestParameters.body = RequestParameters.body =
let useBodyExamples = let useBodyExamples =
config.UseBodyExamples |> Option.defaultValue false config.UseBodyExamples |> Option.defaultValue false
@ -640,12 +652,13 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
OpenApiParameterKind.Body OpenApiParameterKind.Body
(if useBodyExamples then exampleConfig else None) (if useBodyExamples then exampleConfig else None)
config.DataFuzzing config.DataFuzzing
config.TrackFuzzedParameterNames
} }
let allResponseProperties = seq { let allResponseProperties = seq {
for r in m.Value.Responses do for r in m.Value.Responses do
if validResponseCodes |> List.contains r.Key && not (isNull r.Value.ActualResponse.Schema) then if validResponseCodes |> List.contains r.Key && not (isNull r.Value.ActualResponse.Schema) then
yield generateGrammarElementForSchema r.Value.ActualResponse.Schema (None, false) [] id yield generateGrammarElementForSchema r.Value.ActualResponse.Schema (None, false) false [] id
} }
// 'allResponseProperties' contains the schemas of all possible responses // 'allResponseProperties' contains the schemas of all possible responses

7
src/compiler/Restler.Compiler/Config.fs
Просмотреть файл

@ -102,6 +102,11 @@ type Config =
// When specified, use only this naming convention to infer // When specified, use only this naming convention to infer
// producer-consumer dependencies. // producer-consumer dependencies.
ApiNamingConvention : NamingConvention option ApiNamingConvention : NamingConvention option
// When this switch is on, the generated grammar will contain
// parameter names for all fuzzable values. For example:
// restler_fuzzable_string("1", param_name="num_items")
TrackFuzzedParameterNames : bool
} }
let convertToAbsPath (currentDirPath:string) (filePath:string) = let convertToAbsPath (currentDirPath:string) (filePath:string) =
@ -209,6 +214,7 @@ let SampleConfig =
EngineSettingsFilePath = None EngineSettingsFilePath = None
DataFuzzing = true DataFuzzing = true
ApiNamingConvention = None ApiNamingConvention = None
TrackFuzzedParameterNames = false
} }
/// The default config used for unit tests. Most of these should also be the defaults for /// The default config used for unit tests. Most of these should also be the defaults for
@ -238,4 +244,5 @@ let DefaultConfig =
EngineSettingsFilePath = None EngineSettingsFilePath = None
DataFuzzing = false DataFuzzing = false
ApiNamingConvention = None ApiNamingConvention = None
TrackFuzzedParameterNames = false
} }

6
src/compiler/Restler.Compiler/Dependencies.fs
Просмотреть файл

@ -734,7 +734,7 @@ let getParameterDependencies parameterKind globalAnnotations
let resourceAccessPath = PropertyAccessPaths.getLeafAccessPath parentAccessPath p let resourceAccessPath = PropertyAccessPaths.getLeafAccessPath parentAccessPath p
let primitiveType = let primitiveType =
match p.payload with match p.payload with
| FuzzingPayload.Fuzzable (pt, _, _) -> Some pt | FuzzingPayload.Fuzzable (pt, _, _,_) -> Some pt
| FuzzingPayload.Constant (pt, _) -> Some pt | FuzzingPayload.Constant (pt, _) -> Some pt
| FuzzingPayload.Custom c -> Some c.primitiveType | FuzzingPayload.Custom c -> Some c.primitiveType
| _ -> None | _ -> None
@ -1218,7 +1218,7 @@ module DependencyLookup =
else else
let defaultPayload = let defaultPayload =
match p.payload with match p.payload with
| None -> Fuzzable (PrimitiveType.String, "", None) | None -> Fuzzable (PrimitiveType.String, "", None, None)
| Some p -> p | Some p -> p
let propertyAccessPath = let propertyAccessPath =
{ path = PropertyAccessPaths.getInnerAccessPath resourceAccessPath p { path = PropertyAccessPaths.getInnerAccessPath resourceAccessPath p
@ -1235,7 +1235,7 @@ module DependencyLookup =
// First, check if the parameter itself has a dependency // First, check if the parameter itself has a dependency
let (parameterName, properties) = requestParameter.name, requestParameter.payload let (parameterName, properties) = requestParameter.name, requestParameter.payload
let defaultPayload = (Fuzzable (PrimitiveType.String, "", None)) let defaultPayload = (Fuzzable (PrimitiveType.String, "", None, None))
let dependencyPayload = getConsumerPayload dependencies pathPayload requestId parameterName EmptyAccessPath defaultPayload let dependencyPayload = getConsumerPayload dependencies pathPayload requestId parameterName EmptyAccessPath defaultPayload
let payloadWithDependencies = let payloadWithDependencies =

4
src/compiler/Restler.Compiler/Grammar.fs
Просмотреть файл

@ -135,9 +135,9 @@ type FuzzingPayload =
/// Example: (Int "1") /// Example: (Int "1")
| Constant of PrimitiveType * string | Constant of PrimitiveType * string
/// (data type, default value, example value) /// (data type, default value, example value, parameter name)
/// Example: (Int "1", "2") /// Example: (Int "1", "2")
| Fuzzable of PrimitiveType * string * string option | Fuzzable of PrimitiveType * string * string option * string option
/// The custom payload, as specified in the fuzzing dictionary /// The custom payload, as specified in the fuzzing dictionary
| Custom of CustomPayload | Custom of CustomPayload

117
src/compiler/Restler.Compiler/SwaggerVisitors.fs
Просмотреть файл

@ -65,8 +65,10 @@ module SchemaUtilities =
let getGrammarPrimitiveTypeWithDefaultValue (objectType:NJsonSchema.JsonObjectType) let getGrammarPrimitiveTypeWithDefaultValue (objectType:NJsonSchema.JsonObjectType)
(format:string) (format:string)
(exampleValue:string option) : (exampleValue:string option)
(PrimitiveType * string * string option) = (propertyName : string option)
(trackParameters:bool) :
(PrimitiveType * string * string option * string option) =
let defaultTypeWithValue = let defaultTypeWithValue =
match objectType with match objectType with
| NJsonSchema.JsonObjectType.String -> | NJsonSchema.JsonObjectType.String ->
@ -106,10 +108,13 @@ module SchemaUtilities =
raise (UnsupportedType (sprintf "%A is not a fuzzable primitive type. Please make sure your Swagger file is valid." objectType)) raise (UnsupportedType (sprintf "%A is not a fuzzable primitive type. Please make sure your Swagger file is valid." objectType))
let (primitiveType, defaultValue) = defaultTypeWithValue let (primitiveType, defaultValue) = defaultTypeWithValue
primitiveType, defaultValue, exampleValue let propertyName =
if trackParameters then propertyName else None
primitiveType, defaultValue, exampleValue, propertyName
let getFuzzableValueForObjectType (objectType:NJsonSchema.JsonObjectType) (format:string) (exampleValue: string option) = let getFuzzableValueForObjectType (objectType:NJsonSchema.JsonObjectType) (format:string) (exampleValue: string option) (propertyName: string option)
Fuzzable (getGrammarPrimitiveTypeWithDefaultValue objectType format exampleValue) (trackParameters:bool) =
Fuzzable (getGrammarPrimitiveTypeWithDefaultValue objectType format exampleValue propertyName trackParameters)
/// Get a boolean property from 'ExtensionData', if it exists. /// Get a boolean property from 'ExtensionData', if it exists.
let getExtensionDataBooleanPropertyValue (extensionData:System.Collections.Generic.IDictionary<string, obj>) (extensionDataKeyName:string) = let getExtensionDataBooleanPropertyValue (extensionData:System.Collections.Generic.IDictionary<string, obj>) (extensionDataKeyName:string) =
@ -129,7 +134,8 @@ open SchemaUtilities
module SwaggerVisitors = module SwaggerVisitors =
let getFuzzableValueForProperty propertyName (propertySchema:NJsonSchema.JsonSchema) isRequired isReadOnly enumeration defaultValue (exampleValue:string option) = let getFuzzableValueForProperty propertyName (propertySchema:NJsonSchema.JsonSchema) isRequired isReadOnly enumeration defaultValue (exampleValue:string option)
(trackParameters:bool) =
let payload = let payload =
match propertySchema.Type with match propertySchema.Type with
| NJsonSchema.JsonObjectType.String | NJsonSchema.JsonObjectType.String
@ -137,23 +143,23 @@ module SwaggerVisitors =
| NJsonSchema.JsonObjectType.Integer | NJsonSchema.JsonObjectType.Integer
| NJsonSchema.JsonObjectType.Boolean -> | NJsonSchema.JsonObjectType.Boolean ->
match enumeration with match enumeration with
| None -> getFuzzableValueForObjectType propertySchema.Type propertySchema.Format exampleValue | None -> getFuzzableValueForObjectType propertySchema.Type propertySchema.Format exampleValue (Some propertyName) trackParameters
| Some ev -> | Some ev ->
let enumValues = ev |> Seq.map (fun e -> string e) |> Seq.toList let enumValues = ev |> Seq.map (fun e -> string e) |> Seq.toList
let grammarPrimitiveType,_,exv = getGrammarPrimitiveTypeWithDefaultValue propertySchema.Type propertySchema.Format exampleValue let grammarPrimitiveType,_,exv,_ = getGrammarPrimitiveTypeWithDefaultValue propertySchema.Type propertySchema.Format exampleValue (Some propertyName) trackParameters
let defaultFuzzableEnumValue = let defaultFuzzableEnumValue =
match enumValues with match enumValues with
| [] -> "null" | [] -> "null"
| h::rest -> h | h::rest -> h
Fuzzable (PrimitiveType.Enum (propertyName, grammarPrimitiveType, enumValues, defaultValue), defaultFuzzableEnumValue, exv) Fuzzable (PrimitiveType.Enum (propertyName, grammarPrimitiveType, enumValues, defaultValue), defaultFuzzableEnumValue, exv, None)
| NJsonSchema.JsonObjectType.Object | NJsonSchema.JsonObjectType.Object
| NJsonSchema.JsonObjectType.None -> | NJsonSchema.JsonObjectType.None ->
// Example of JsonObjectType.None: "content": {} without a type specified in Swagger. // Example of JsonObjectType.None: "content": {} without a type specified in Swagger.
// Let's treat these the same as Object. // Let's treat these the same as Object.
getFuzzableValueForObjectType NJsonSchema.JsonObjectType.Object propertySchema.Format exampleValue getFuzzableValueForObjectType NJsonSchema.JsonObjectType.Object propertySchema.Format exampleValue (Some propertyName) trackParameters
| NJsonSchema.JsonObjectType.File -> | NJsonSchema.JsonObjectType.File ->
// Fuzz it as a string. // Fuzz it as a string.
Fuzzable (PrimitiveType.String, "file object", None) Fuzzable (PrimitiveType.String, "file object", None, if trackParameters then Some propertyName else None)
| nst -> | nst ->
raise (UnsupportedType (sprintf "Unsupported type formatting: %A" nst)) raise (UnsupportedType (sprintf "Unsupported type formatting: %A" nst))
{ LeafProperty.name = propertyName; payload = payload ;isRequired = isRequired ; isReadOnly = isReadOnly } { LeafProperty.name = propertyName; payload = payload ;isRequired = isRequired ; isReadOnly = isReadOnly }
@ -168,7 +174,23 @@ module SwaggerVisitors =
then None then None
else Some (property.Default.ToString()) else Some (property.Default.ToString())
/// Add property name to the fuzzable payload
/// This is used to track fuzzed parameter values
let addTrackedParameterName (tree:Tree<LeafProperty, InnerProperty>) paramName trackParameters =
if trackParameters then
match tree with
| LeafNode leafProperty ->
let payload = leafProperty.payload
match payload with
| Fuzzable(a, b, c, _) ->
let payload = Fuzzable (a, b, c, Some paramName)
LeafNode { leafProperty with LeafProperty.payload = payload }
| x -> tree
| InternalNode (_,_) -> tree
else tree
module GenerateGrammarElements = module GenerateGrammarElements =
/// Returns the specified property when the object contains it. /// Returns the specified property when the object contains it.
/// Note: if the example object does not contain the property, /// Note: if the example object does not contain the property,
let extractPropertyFromObject propertyName (objectType:NJsonSchema.JsonObjectType) let extractPropertyFromObject propertyName (objectType:NJsonSchema.JsonObjectType)
@ -258,6 +280,7 @@ module SwaggerVisitors =
let rec processProperty (propertyName, property:NJsonSchema.JsonSchemaProperty) let rec processProperty (propertyName, property:NJsonSchema.JsonSchemaProperty)
(propertyPayloadExampleValue: JToken option, generateFuzzablePayload:bool) (propertyPayloadExampleValue: JToken option, generateFuzzablePayload:bool)
(trackParameters:bool)
(parents:NJsonSchema.JsonSchema list) (parents:NJsonSchema.JsonSchema list)
(cont: Tree<LeafProperty, InnerProperty> -> Tree<LeafProperty, InnerProperty>) = (cont: Tree<LeafProperty, InnerProperty> -> Tree<LeafProperty, InnerProperty>) =
@ -285,6 +308,7 @@ module SwaggerVisitors =
(tryGetEnumeration propertySchema) (tryGetEnumeration propertySchema)
(tryGetDefault propertySchema) (tryGetDefault propertySchema)
schemaExampleValue schemaExampleValue
trackParameters
let propertyPayload = let propertyPayload =
match propertyPayloadExampleValue with match propertyPayloadExampleValue with
| None -> | None ->
@ -293,13 +317,13 @@ module SwaggerVisitors =
| Some v -> | Some v ->
let examplePropertyPayload = let examplePropertyPayload =
match fuzzablePropertyPayload.payload with match fuzzablePropertyPayload.payload with
| Fuzzable (primitiveType, defaultValue, _) -> | Fuzzable (primitiveType, defaultValue, _, propertyName) ->
let payloadValue = GenerateGrammarElements.formatJTokenProperty primitiveType v let payloadValue = GenerateGrammarElements.formatJTokenProperty primitiveType v
// Replace the default payload with the example payload, preserving type information. // Replace the default payload with the example payload, preserving type information.
// 'generateFuzzablePayload' is specified a schema example is found for the parent // 'generateFuzzablePayload' is specified a schema example is found for the parent
// object (e.g. an array). // object (e.g. an array).
if generateFuzzablePayload then if generateFuzzablePayload then
Fuzzable (primitiveType, defaultValue, Some payloadValue) Fuzzable (primitiveType, defaultValue, Some payloadValue, propertyName)
else else
Constant (primitiveType, payloadValue) Constant (primitiveType, payloadValue)
| _ -> raise (invalidOp(sprintf "invalid payload %A, expected fuzzable" fuzzablePropertyPayload)) | _ -> raise (invalidOp(sprintf "invalid payload %A, expected fuzzable" fuzzablePropertyPayload))
@ -312,7 +336,8 @@ module SwaggerVisitors =
// The example property value needs to be examined according to the 'ActualSchema', which // The example property value needs to be examined according to the 'ActualSchema', which
// is passed through below. // is passed through below.
generateGrammarElementForSchema property.ActualSchema (propertyPayloadExampleValue, generateFuzzablePayload) parents (fun tree -> generateGrammarElementForSchema property.ActualSchema (propertyPayloadExampleValue, generateFuzzablePayload)
trackParameters parents (fun tree ->
let innerProperty = { InnerProperty.name = propertyName let innerProperty = { InnerProperty.name = propertyName
payload = None payload = None
propertyType = Property propertyType = Property
@ -330,36 +355,47 @@ module SwaggerVisitors =
let arrayWithElements = let arrayWithElements =
generateGrammarElementForSchema property (propertyPayloadExampleValue, generateFuzzablePayload) generateGrammarElementForSchema property (propertyPayloadExampleValue, generateFuzzablePayload)
parents (fun tree -> tree) trackParameters parents (fun tree -> tree)
|> cont |> cont
// For tracked parameters, it is required to add the name to the fuzzable primitives
// This name will not be added if the array elements are leaf nodes that do not have inner properties.
match arrayWithElements with match arrayWithElements with
| InternalNode (n, tree) -> | InternalNode (n, tree) ->
let tree =
tree
|> Seq.map (fun elem -> addTrackedParameterName elem propertyName trackParameters)
InternalNode (innerArrayProperty, tree) InternalNode (innerArrayProperty, tree)
| LeafNode _ -> | LeafNode _ ->
raise (invalidOp("An array should be an internal node.")) raise (invalidOp("An array should be an internal node."))
| NJsonSchema.JsonObjectType.Object -> | NJsonSchema.JsonObjectType.Object ->
// This object may or may not have nested properties. let objTree =
// Similar to type 'None', just pass through the object and it will be taken care of downstream. // This object may or may not have nested properties.
generateGrammarElementForSchema property.ActualSchema (None, false) parents (fun tree -> // Similar to type 'None', just pass through the object and it will be taken care of downstream.
// If the object has no properties, it should be set to its primitive type. generateGrammarElementForSchema property.ActualSchema (None, false) trackParameters parents (fun tree ->
match tree with // If the object has no properties, it should be set to its primitive type.
| LeafNode l -> match tree with
LeafNode { l with name = propertyName } | LeafNode l ->
|> cont LeafNode { l with name = propertyName }
| InternalNode _ -> |> cont
let innerProperty = { InnerProperty.name = propertyName | InternalNode _ ->
payload = None let innerProperty = { InnerProperty.name = propertyName
propertyType = Property // indicates presence of nested properties payload = None
isRequired = true propertyType = Property // indicates presence of nested properties
isReadOnly = (propertyIsReadOnly property) } isRequired = true
InternalNode (innerProperty, stn tree) isReadOnly = (propertyIsReadOnly property) }
|> cont InternalNode (innerProperty, stn tree)
) |> cont
)
// For tracked parameters, it is required to add the name to the fuzzable primitives
// This name will not be added if the object elements are leaf nodes that do not have inner properties.
let objTree = addTrackedParameterName objTree propertyName trackParameters
objTree
| nst -> | nst ->
raise (UnsupportedType (sprintf "Found unsupported type in body parameters: %A" nst)) raise (UnsupportedType (sprintf "Found unsupported type in body parameters: %A" nst))
and generateGrammarElementForSchema (schema:NJsonSchema.JsonSchema) and generateGrammarElementForSchema (schema:NJsonSchema.JsonSchema)
(exampleValue:JToken option, generateFuzzablePayloadsForExamples:bool) (exampleValue:JToken option, generateFuzzablePayloadsForExamples:bool)
(trackParameters:bool)
(parents:NJsonSchema.JsonSchema list) (parents:NJsonSchema.JsonSchema list)
(cont: Tree<LeafProperty, InnerProperty> -> Tree<LeafProperty, InnerProperty>) = (cont: Tree<LeafProperty, InnerProperty> -> Tree<LeafProperty, InnerProperty>) =
@ -380,7 +416,7 @@ module SwaggerVisitors =
// TODO: need test case for this example. Raise an exception to flag these cases. // TODO: need test case for this example. Raise an exception to flag these cases.
// Most likely, the current example value should simply be used in the leaf property // Most likely, the current example value should simply be used in the leaf property
raise (UnsupportedRecursiveExample (sprintf "%A" exampleValue)) raise (UnsupportedRecursiveExample (sprintf "%A" exampleValue))
let leafProperty = { LeafProperty.name = ""; payload = Fuzzable (PrimitiveType.String, "", None); isRequired = true ; isReadOnly = false } let leafProperty = { LeafProperty.name = ""; payload = Fuzzable (PrimitiveType.String, "", None, None); isRequired = true ; isReadOnly = false }
LeafNode leafProperty LeafNode leafProperty
|> cont |> cont
else else
@ -395,6 +431,7 @@ module SwaggerVisitors =
else else
Some (processProperty (name, item.Value) Some (processProperty (name, item.Value)
(exValue, generateFuzzablePayloadsForExamples) (exValue, generateFuzzablePayloadsForExamples)
trackParameters
(schema::parents) id)) (schema::parents) id))
let arrayProperties = let arrayProperties =
if schema.IsArray then if schema.IsArray then
@ -414,12 +451,13 @@ module SwaggerVisitors =
let schemaArrayExamples = ExampleHelpers.tryGetArraySchemaExample schema let schemaArrayExamples = ExampleHelpers.tryGetArraySchemaExample schema
match schemaArrayExamples with match schemaArrayExamples with
| None -> | None ->
generateGrammarElementForSchema schema.Item.ActualSchema (None, false) (schema::parents) id generateGrammarElementForSchema schema.Item.ActualSchema (None, false) trackParameters (schema::parents) id
|> stn |> stn
| Some sae -> | Some sae ->
sae |> Seq.map (fun (schemaExampleValue, generateFuzzablePayload) -> sae |> Seq.map (fun (schemaExampleValue, generateFuzzablePayload) ->
generateGrammarElementForSchema schema.Item.ActualSchema generateGrammarElementForSchema schema.Item.ActualSchema
(schemaExampleValue, generateFuzzablePayload) (schemaExampleValue, generateFuzzablePayload)
trackParameters
(schema::parents) (schema::parents)
id id
) )
@ -432,6 +470,7 @@ module SwaggerVisitors =
generateGrammarElementForSchema generateGrammarElementForSchema
schema.Item.ActualSchema schema.Item.ActualSchema
(Some example, false) (Some example, false)
trackParameters
(schema::parents) (schema::parents)
id |> stn) id |> stn)
|> Seq.concat |> Seq.concat
@ -440,7 +479,7 @@ module SwaggerVisitors =
let allOfParameterSchemas = let allOfParameterSchemas =
schema.AllOf schema.AllOf
|> Seq.map (fun ao -> ao, generateGrammarElementForSchema ao.ActualSchema (exampleValue, false) (schema::parents) id) |> Seq.map (fun ao -> ao, generateGrammarElementForSchema ao.ActualSchema (exampleValue, false) trackParameters (schema::parents) id)
let allOfProperties = let allOfProperties =
allOfParameterSchemas allOfParameterSchemas
@ -496,6 +535,7 @@ module SwaggerVisitors =
(tryGetEnumeration schema) (*enumeration*) (tryGetEnumeration schema) (*enumeration*)
(tryGetDefault schema) (*defaultValue*) (tryGetDefault schema) (*defaultValue*)
specExampleValue specExampleValue
trackParameters
LeafNode leafProperty LeafNode leafProperty
|> cont |> cont
| Some v -> | Some v ->
@ -509,13 +549,14 @@ module SwaggerVisitors =
let schemaType = let schemaType =
if schema.Type = JsonObjectType.None then JsonObjectType.Object if schema.Type = JsonObjectType.None then JsonObjectType.Object
else schema.Type else schema.Type
// Note: since the paramName is not available here, set it to 'None'.
let primitiveType, defaultValue, _ = getGrammarPrimitiveTypeWithDefaultValue schemaType schema.Format None // This will be fixed up from the parent level.
let primitiveType, defaultValue, _ , _ = getGrammarPrimitiveTypeWithDefaultValue schemaType schema.Format None None trackParameters
let leafPayload = let leafPayload =
let exampleValue = GenerateGrammarElements.formatJTokenProperty primitiveType v let exampleValue = GenerateGrammarElements.formatJTokenProperty primitiveType v
if generateFuzzablePayloadsForExamples then if generateFuzzablePayloadsForExamples then
FuzzingPayload.Fuzzable (primitiveType, defaultValue, Some exampleValue) FuzzingPayload.Fuzzable (primitiveType, defaultValue, Some exampleValue, None)
else else
FuzzingPayload.Constant (primitiveType, exampleValue) FuzzingPayload.Constant (primitiveType, exampleValue)