microsoft
/
restler-fuzzer
зеркало из https://github.com/microsoft/restler-fuzzer.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Add the ability to track parameter values in the spec coverage file. (#253) 

For large-scale testing, it is useful to have the specific values of
parameters fuzzed in the spec coverage file, in order to facilitate
root cause failure analysis.

This change adds an option to the compiler that produces a grammar which
contains the parameter names to track for every fuzzable value.

Testing:
- manual testing
- added/fixed engine tests to make sure new code path is exercised
- Note: the payload body checker code is not modified in this change because
it does not require this feature.
This commit is contained in:
marina-p 2021-06-30 15:32:03 -07:00 коммит произвёл GitHub
Родитель 89279e0869
Коммит e2b25ebe16
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
16 изменённых файлов: 289 добавлений и 113 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

3
docs/user-guide/CompilerConfig.md
Просмотреть файл

@ -97,3 +97,6 @@ catch consistency bugs in the specification because producer-consumer dependenci
HyphenSeparator
UnderscoreSeparator
```
* *TrackFuzzedParameterNames* False by default. When true, every fuzzable primitive will
include an additional parameter `param_name` which is the name of the property or
parameter being fuzzed. These will be used to capture fuzzed parameters in ```tracked_parameters``` in the spec coverage file.

7
docs/user-guide/Testing.md
Просмотреть файл

@ -105,7 +105,8 @@ During each Test run a `speccov.json` file will be created in the logs directory
"response_body": "{\n \"errors\": {\n \"id\": \"'5882' is not of type 'integer'\"\n },\n \"message\": \"Input payload validation failed\"\n}\n"
},
"tracked_parameters": {
"per_page_9": "2"
"per_page": ["2"],
"page": ["1"]
}
},
```
@ -135,7 +136,9 @@ the coverage data is being reported.
* The __"tracked_parameters"__ property is optional and generated only when using
`Test` mode with ```test-all-combinations```.
This property contains key-value pairs of all of the parameters
for which more than one value is being tested.
for which more than one value is being tested. By default, enums and custom payloads
are always tracked. In addition, when the specification was compiled with
`TrackFuzzedParameterNames` set to `true`, all fuzzable parameters will be tracked.
#### Postprocessing Scripts:
The `utilities` directory contains a sub-directory called `speccovparsing` that contains scripts for postprocessing speccov files.

36
restler/engine/core/requests.py
Просмотреть файл

@ -602,11 +602,10 @@ class Request(object):
quoted = request_block[2]
examples = request_block[3]
else:
field_name = None
default_val = request_block[1]
quoted = request_block[2]
examples = request_block[3]
field_name = request_block[4]
values = []
# Handling dynamic primitives that need fresh rendering every time
if primitive_type == primitives.FUZZABLE_UUID4:
@ -696,9 +695,10 @@ class Request(object):
# Only track the parameter if there are multiple values being combined
if len(values) > 1:
if not field_name:
field_name = "tracked_param"
field_name = f"{field_name}_{param_idx}"
tracked_parameters[field_name] = param_idx
field_name = f"tracked_param_{param_idx}"
if field_name not in tracked_parameters:
tracked_parameters[field_name] = []
tracked_parameters[field_name].append(param_idx)
fuzzable.append(values)
@ -718,9 +718,10 @@ class Request(object):
values = request_utilities.resolve_dynamic_primitives(values, candidate_values_pool)
tracked_parameter_values = {}
for (k, idx) in tracked_parameters.items():
tracked_parameter_values[k] = values[idx]
for (k, idx_list) in tracked_parameters.items():
tracked_parameter_values[k] = []
for idx in idx_list:
tracked_parameter_values[k].append(values[idx])
rendered_data = "".join(values)
yield rendered_data, parser, tracked_parameter_values
@ -764,6 +765,25 @@ class Request(object):
return self._total_feasible_combinations
def update_tracked_parameters(self, tracked_parameters):
""" Updates tracked parameters for the request by merging with the
existing parameters.
The tracked parameters are currently a set of names, with arrays of
values that were observed during sequence rendering.
@param tracked_parameters: The tracked parameters and their values.
@type tracked_parameters: Dict
@return: None
@rtype : None
"""
for param_name, param_val_list in tracked_parameters.items():
for param_val in param_val_list:
if param_name not in self._tracked_parameters:
self._tracked_parameters[param_name] = []
self._tracked_parameters[param_name].append(param_val)
def GrammarRequestCollection():
""" Accessor for the global request collection singleton """
return GlobalRequestCollection.Instance()._req_collection

5
restler/engine/core/sequences.py
Просмотреть файл

@ -350,7 +350,8 @@ class Sequence(object):
dependencies.reset_tlb()
sequence_failed = False
request._tracked_parameters = tracked_parameters
request._tracked_parameters = {}
request.update_tracked_parameters(tracked_parameters)
# Step A: Static template rendering
# Render last known valid combination of primitive type values
# for every request until the last
@ -360,7 +361,7 @@ class Sequence(object):
prev_request.render_current(candidate_values_pool,
preprocessing=preprocessing)
request._tracked_parameters.update(tracked_parameters)
request.update_tracked_parameters(tracked_parameters)
# substitute reference placeholders with resolved values
if not Settings().ignore_dependencies:

2
restler/engine/fuzzing_parameters/body_schema.py
Просмотреть файл

@ -201,10 +201,10 @@ class BodySchema():
examples = request_block[3]
value = None
else:
field_name = None
value = request_block[1]
quoted = request_block[2]
examples = request_block[3]
field_name = request_block[4]
# accumulate
if primitive_type == primitives.STATIC_STRING:

65
restler/engine/primitives.py
Просмотреть файл

@ -51,6 +51,11 @@ UNQUOTED_STR = '_unquoted'
# Optional argument passed to fuzzable primitive definition function that can
# provide an example value. This value is used instead of the default value if present.
EXAMPLES_ARG = 'examples'
# Optional argument passed to fuzzable primitive definition function that can
# provide the name of the parameter being fuzzed.
# This value is used in test-all-combinations mode to allow the user to analyze spec coverage
# for particular parameter values.
PARAM_NAME_ARG = 'param_name'
class CandidateValues(object):
def __init__(self):
self.unquoted_values = []
@ -374,7 +379,8 @@ def restler_static_string(*args, **kwargs):
if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG]
examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_string(*args, **kwargs):
@ -400,7 +406,10 @@ def restler_fuzzable_string(*args, **kwargs):
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_int(*args, **kwargs):
""" Integer primitive.
@ -425,7 +434,10 @@ def restler_fuzzable_int(*args, **kwargs):
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_bool(*args, **kwargs):
@ -451,7 +463,10 @@ def restler_fuzzable_bool(*args, **kwargs):
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_number(*args, **kwargs):
@ -477,7 +492,10 @@ def restler_fuzzable_number(*args, **kwargs):
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_delim(*args, **kwargs):
@ -503,7 +521,10 @@ def restler_fuzzable_delim(*args, **kwargs):
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_group(*args, **kwargs):
@ -536,7 +557,8 @@ def restler_fuzzable_group(*args, **kwargs):
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, enum_vals, quoted, examples
param_name = None
return sys._getframe().f_code.co_name, field_name, enum_vals, quoted, examples, param_name
def restler_fuzzable_uuid4(*args, **kwargs):
@ -562,7 +584,10 @@ def restler_fuzzable_uuid4(*args, **kwargs):
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_datetime(*args, **kwargs) :
@ -589,7 +614,10 @@ def restler_fuzzable_datetime(*args, **kwargs) :
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_fuzzable_object(*args, **kwargs) :
""" object primitive ({})
@ -614,7 +642,9 @@ def restler_fuzzable_object(*args, **kwargs) :
examples=None
if EXAMPLES_ARG in kwargs:
examples = kwargs[EXAMPLES_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples
if PARAM_NAME_ARG in kwargs:
param_name = kwargs[PARAM_NAME_ARG]
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_multipart_formdata(*args, **kwargs):
""" Multipart/formdata primitive
@ -638,7 +668,8 @@ def restler_multipart_formdata(*args, **kwargs):
if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG]
examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_custom_payload(*args, **kwargs):
@ -662,7 +693,8 @@ def restler_custom_payload(*args, **kwargs):
if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG]
examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_custom_payload_header(*args, **kwargs):
@ -686,7 +718,8 @@ def restler_custom_payload_header(*args, **kwargs):
if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG]
examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_custom_payload_uuid4_suffix(*args, **kwargs):
@ -710,7 +743,8 @@ def restler_custom_payload_uuid4_suffix(*args, **kwargs):
if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG]
examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name
def restler_refreshable_authentication_token(*args, **kwargs):
@ -734,4 +768,5 @@ def restler_refreshable_authentication_token(*args, **kwargs):
if QUOTED_ARG in kwargs:
quoted = kwargs[QUOTED_ARG]
examples = None
return sys._getframe().f_code.co_name, field_name, quoted, examples
param_name = None
return sys._getframe().f_code.co_name, field_name, quoted, examples, param_name

4
restler/unit_tests/log_baseline_test_files/test_grammar.json
Просмотреть файл

@ -137,7 +137,9 @@
"payload": {
"Fuzzable": [
"Int",
"1000"
"1000",
null,
"population"
]
},
"isRequired": false,

10
restler/unit_tests/log_baseline_test_files/test_grammar.py
Просмотреть файл

@ -223,8 +223,12 @@ request = requests.Request([
primitives.restler_static_string("\r\n"),
primitives.restler_static_string("{"),
primitives.restler_static_string('"population":'),
primitives.restler_fuzzable_int('10000', quoted=True),
primitives.restler_fuzzable_int('10000', quoted=True, param_name="population"),
primitives.restler_static_string(', "area": "5000",'),
# Note: There is a discrepancy here due to
# manual creation of this grammar - the fuzzable string
# does not match the grammar.json, which contains
# a constant string for this value.
primitives.restler_fuzzable_string('strtest', quoted=True),
primitives.restler_static_string(':'),
primitives.restler_fuzzable_bool('true', quoted=True),
@ -626,7 +630,7 @@ request = requests.Request([
primitives.restler_static_string("{"),
primitives.restler_static_string('testbool', quoted=True),
primitives.restler_static_string(':'),
primitives.restler_fuzzable_bool("testval", quoted=True),
primitives.restler_fuzzable_bool("testval", quoted=True, param_name="testbool"),
primitives.restler_static_string(',"location":'),
primitives.restler_custom_payload("location", quoted=True),
primitives.restler_static_string("}"),
@ -849,7 +853,7 @@ request = requests.Request([
primitives.restler_static_string("\r\n"),
primitives.restler_static_string("{"),
primitives.restler_static_string('"datetest":'),
primitives.restler_fuzzable_datetime("2020-1-1", quoted=True),
primitives.restler_fuzzable_datetime("2020-1-1", quoted=True, example=["2020-2-2"], param_name="datetest"),
primitives.restler_static_string(',"id":'),
primitives.restler_static_string('"/testparts/'),
primitives.restler_custom_payload("testcustomparts", quoted=False),

2
restler/utils/logger.py
Просмотреть файл

@ -480,10 +480,10 @@ def custom_network_logging(sequence, candidate_values_pool, **kwargs):
quoted = request_block[2]
examples = request_block[3]
else:
field_name = None
default_val = request_block[1]
quoted = request_block[2]
examples = request_block[3]
field_name = request_block[4]
# Handling dynamic primitives that need fresh rendering every time
if primitive == "restler_fuzzable_uuid4":

36
src/compiler/Restler.Compiler.Test/CodeGeneratorTests.fs
Просмотреть файл

@ -5,6 +5,7 @@ namespace Restler.Test
open System.IO
open Xunit
open System
open Restler.Grammar
open Restler.CodeGenerator.Python.Types
open Tree
@ -23,7 +24,7 @@ module CodeGenerator =
/// This is useful because the grammar json is validated
/// when deserialized by the RESTler compiler, avoiding typos made directly in python.
/// This also sanity checks that the grammar generation is deterministic.
member __.``generate code from modified grammar`` () =
let ``generate code from modified grammar`` () =
let grammarOutputDirectory1 = ctx.testRootDirPath
Restler.Workflow.generateRestlerGrammar None
@ -42,7 +43,7 @@ module CodeGenerator =
[<Fact>]
member __.``python grammar request sanity test`` () =
let ``python grammar request sanity test`` () =
let q1 = LeafNode { LeafProperty.name = "page"; payload = Constant (Int, "1") ;isRequired = true ; isReadOnly = false }
let q2 = LeafNode { LeafProperty.name = "page"; payload = Constant (Int, "1") ;isRequired = true ; isReadOnly = false }
let b1 = LeafNode { LeafProperty.name = "payload"; payload = Constant (PrimitiveType.String, "hello") ;
@ -88,7 +89,7 @@ module CodeGenerator =
Assert.True(elements |> Seq.length > 0)
[<Fact>]
member __.``python grammar parameter sanity test`` () =
let ``python grammar parameter sanity test`` () =
let p1 = { LeafProperty.name = "region"; payload = Constant (PrimitiveType.String, "WestUS");
isRequired = true ; isReadOnly = false}
let p2 = { LeafProperty.name = "maxResources"; payload = Constant (Int, "10") ;
@ -112,5 +113,34 @@ module CodeGenerator =
let hasResources = result |> Seq.tryFind (fun s -> s = (Restler_static_string_constant "\"maxResources\":"))
Assert.True(hasResources.IsSome, "resources not found")
// Test that the generated python and json grammars for tracked parameters are correct
[<Fact>]
let ``tracked parameters tests`` () =
let grammarDirectoryPath = ctx.testRootDirPath
let config = { Restler.Config.SampleConfig with
IncludeOptionalParameters = true
GrammarOutputDirectoryPath = Some grammarDirectoryPath
ResolveBodyDependencies = true
UseBodyExamples = Some false
SwaggerSpecFilePath = Some [(Path.Combine(Environment.CurrentDirectory, @"swagger\example_demo1.json"))]
CustomDictionaryFilePath = Some (Path.Combine(Environment.CurrentDirectory, @"swagger\example_demo_dictionary.json"))
}
Restler.Workflow.generateRestlerGrammar None config
let grammarFilePath = Path.Combine(grammarDirectoryPath, "grammar.py")
let grammar = File.ReadAllText(grammarFilePath)
// The grammar should not contain gracked parameters by default
Assert.False(grammar.Contains("param_name="))
// Now turn on parameter tracking
let config = { config with TrackFuzzedParameterNames = true }
Restler.Workflow.generateRestlerGrammar None config
let grammarFilePath = Path.Combine(grammarDirectoryPath, "grammar.py")
let grammar = File.ReadAllText(grammarFilePath)
Assert.True(grammar.Contains("param_name=\"storeId\""))
Assert.True(grammar.Contains("param_name=\"bannedBrands\""))
Assert.True(grammar.Contains("param_name=\"groceryItemTags\""))
interface IClassFixture<Fixtures.TestSetupAndCleanup>

53
src/compiler/Restler.Compiler/CodeGenerator.fs
Просмотреть файл

@ -21,6 +21,7 @@ module Types =
defaultValue : string
isQuoted : bool
exampleValue : string option
trackedParameterName: string option
}
/// RESTler grammar built-in types
@ -62,18 +63,18 @@ let rec getRestlerPythonPayload (payload:FuzzingPayload) (isQuoted:bool) : Reque
match p with
| Constant (t,v) ->
Restler_static_string_constant v
| Fuzzable (t,v,exv) ->
| Fuzzable (t,v,exv,parameterName) ->
match t with
| Bool -> Restler_fuzzable_bool { defaultValue = v ; isQuoted = false ; exampleValue = exv }
| Bool -> Restler_fuzzable_bool { defaultValue = v ; isQuoted = false ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.DateTime ->
Restler_fuzzable_datetime { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv }
Restler_fuzzable_datetime { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.String ->
Restler_fuzzable_string { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv }
| PrimitiveType.Object -> Restler_fuzzable_object { defaultValue = v ; isQuoted = false ; exampleValue = exv }
| Int -> Restler_fuzzable_int { defaultValue = v ; isQuoted = false; exampleValue = exv }
| Number -> Restler_fuzzable_number { defaultValue = v ; isQuoted = false ; exampleValue = exv }
Restler_fuzzable_string { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.Object -> Restler_fuzzable_object { defaultValue = v ; isQuoted = false ; exampleValue = exv ; trackedParameterName = parameterName }
| Int -> Restler_fuzzable_int { defaultValue = v ; isQuoted = false; exampleValue = exv ; trackedParameterName = parameterName }
| Number -> Restler_fuzzable_number { defaultValue = v ; isQuoted = false ; exampleValue = exv ; trackedParameterName = parameterName }
| Uuid ->
Restler_fuzzable_uuid4 { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv }
Restler_fuzzable_uuid4 { defaultValue = v ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| PrimitiveType.Enum (enumPropertyName, _, enumeration, defaultValue) ->
let defaultStr =
match defaultValue with
@ -86,11 +87,11 @@ let rec getRestlerPythonPayload (payload:FuzzingPayload) (isQuoted:bool) : Reque
defaultStr
)
Restler_fuzzable_group
{ defaultValue = groupValue ; isQuoted = isQuoted ; exampleValue = exv }
{ defaultValue = groupValue ; isQuoted = isQuoted ; exampleValue = exv ; trackedParameterName = parameterName }
| Custom c ->
match c.payloadType with
| CustomPayloadType.String ->
Restler_custom_payload { defaultValue = c.payloadValue ; isQuoted = isQuoted ; exampleValue = None }
Restler_custom_payload { defaultValue = c.payloadValue ; isQuoted = isQuoted ; exampleValue = None ; trackedParameterName = None }
| CustomPayloadType.UuidSuffix ->
Restler_custom_payload_uuid4_suffix c.payloadValue
| CustomPayloadType.Header ->
@ -335,7 +336,7 @@ let generatePythonParameter includeOptionalParameters parameterKind (requestPara
| FuzzingPayload.Constant (primitiveType, v) ->
isPrimitiveTypeQuoted primitiveType (isNull v),
false
| FuzzingPayload.Fuzzable (primitiveType, _, _) ->
| FuzzingPayload.Fuzzable (primitiveType, _, _,_) ->
// Note: this is a current RESTler limitation -
// fuzzable values may not be set to null without changing the grammar.
isPrimitiveTypeQuoted primitiveType false,
@ -826,6 +827,14 @@ let getRequests(requests:Request list) includeOptionalParameters =
let quotedStr = sprintf "%s%s%s" exDelim exStr exDelim
sprintf ", examples=[%s]" quotedStr
let getTrackedParamPrimitiveParameter paramName =
match paramName with
| None -> ""
| Some str ->
let exStr, exDelim = quoteStringForPythonGrammar str
let quotedStr = sprintf "%s%s%s" exDelim exStr exDelim
sprintf ", param_name=%s" quotedStr
let str =
match p with
| Restler_static_string_jtoken_delim s ->
@ -856,32 +865,38 @@ let getRequests(requests:Request list) includeOptionalParameters =
let quotedDefaultString =
sprintf "%s%s%s" delim str delim
let exampleParameter = getExamplePrimitiveParameter s.exampleValue
sprintf "primitives.restler_fuzzable_string(%s, quoted=%s%s)"
let trackedParamName = getTrackedParamPrimitiveParameter s.trackedParameterName
sprintf "primitives.restler_fuzzable_string(%s, quoted=%s%s%s)"
quotedDefaultString
(if s.isQuoted then "True" else "False")
exampleParameter
trackedParamName
| Restler_fuzzable_group s ->
sprintf "primitives.restler_fuzzable_group(%s,quoted=%s%s)"
s.defaultValue
(if s.isQuoted then "True" else "False")
(getExamplePrimitiveParameter s.exampleValue)
| Restler_fuzzable_int s ->
sprintf "primitives.restler_fuzzable_int(\"%s\"%s)"
sprintf "primitives.restler_fuzzable_int(\"%s\"%s%s)"
s.defaultValue
(getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_number s ->
sprintf "primitives.restler_fuzzable_number(\"%s\"%s)"
sprintf "primitives.restler_fuzzable_number(\"%s\"%s%s)"
s.defaultValue
(getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_bool s ->
sprintf "primitives.restler_fuzzable_bool(\"%s\"%s)"
sprintf "primitives.restler_fuzzable_bool(\"%s\"%s%s)"
s.defaultValue
(getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_datetime s ->
sprintf "primitives.restler_fuzzable_datetime(\"%s\", quoted=%s%s)"
sprintf "primitives.restler_fuzzable_datetime(\"%s\", quoted=%s%s%s)"
s.defaultValue
(if s.isQuoted then "True" else "False")
(getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_object s ->
if String.IsNullOrEmpty s.defaultValue then
printfn "ERROR: fuzzable objects should not be empty. Skipping."
@ -891,14 +906,16 @@ let getRequests(requests:Request list) includeOptionalParameters =
let quotedDefaultString =
sprintf "%s%s%s" delim str delim
let exampleParameter = getExamplePrimitiveParameter s.exampleValue
sprintf "primitives.restler_fuzzable_object(%s%s)"
sprintf "primitives.restler_fuzzable_object(%s%s%s)"
quotedDefaultString
exampleParameter
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_fuzzable_uuid4 s ->
sprintf "primitives.restler_fuzzable_uuid4(\"%s\", quoted=%s%s)"
sprintf "primitives.restler_fuzzable_uuid4(\"%s\", quoted=%s%s%s)"
s.defaultValue
(if s.isQuoted then "True" else "False")
(getExamplePrimitiveParameter s.exampleValue)
(getTrackedParamPrimitiveParameter s.trackedParameterName)
| Restler_custom_payload p ->
sprintf "primitives.restler_custom_payload(\"%s\", quoted=%s)"
p.defaultValue

41
src/compiler/Restler.Compiler/Compiler.fs
Просмотреть файл

@ -156,7 +156,8 @@ module private Parameters =
| _ -> raise (UnsupportedType "Complex path parameters are not supported")
let private getParametersFromExample (examplePayload:ExampleRequestPayload)
(parameterList:seq<OpenApiParameter>) =
(parameterList:seq<OpenApiParameter>)
(trackParameters:bool) =
parameterList
|> Seq.choose (fun declaredParameter ->
// If the declared parameter isn't in the example, skip it. Here, the example is used to
@ -169,7 +170,7 @@ module private Parameters =
| PayloadFormat.JToken payloadValue ->
let parameterGrammarElement =
generateGrammarElementForSchema declaredParameter.ActualSchema
(Some payloadValue, false) [] id
(Some payloadValue, false) trackParameters [] id
Some { name = declaredParameter.Name
payload = parameterGrammarElement
serialization = getParameterSerialization declaredParameter }
@ -205,7 +206,8 @@ module private Parameters =
None
let pathParameters (swaggerMethodDefinition:OpenApiOperation) (endpoint:string)
(exampleConfig: ExampleRequestPayload list option) =
(exampleConfig: ExampleRequestPayload list option)
(trackParameters:bool) =
let declaredPathParameters = swaggerMethodDefinition.ActualParameters
|> Seq.filter (fun p -> p.Kind = NSwag.OpenApiParameterKind.Path)
@ -250,19 +252,22 @@ module private Parameters =
(tryGetEnumeration schema)
(tryGetDefault schema)
specExampleValue
trackParameters
Some { name = parameterName
payload = LeafNode leafProperty
serialization = serialization }
| Some (firstExample::remainingExamples) ->
// Use the first example specified to determine the parameter value.
getParametersFromExample firstExample (parameter |> stn)
getParametersFromExample firstExample (parameter |> stn) trackParameters
|> Seq.head
|> Some
)
ParameterList parameterList
let private getParameters (parameterList:seq<OpenApiParameter>)
(exampleConfig:ExampleRequestPayload list option) (dataFuzzing:bool) =
(exampleConfig:ExampleRequestPayload list option)
(dataFuzzing:bool)
(trackParameters:bool) =
// When data fuzzing is specified, both the full schema and examples should be available for analysis.
// Otherwise, use the first example if it exists, or the schema, and return a single schema.
@ -272,9 +277,9 @@ module private Parameters =
| Some [] -> None
| Some (firstExample::remainingExamples) ->
// Use the first example specified to determine the parameter schema and values.
let firstPayload = getParametersFromExample firstExample parameterList
let firstPayload = getParametersFromExample firstExample parameterList trackParameters
let restOfPayloads =
remainingExamples |> List.map (fun e -> getParametersFromExample e parameterList)
remainingExamples |> List.map (fun e -> getParametersFromExample e parameterList trackParameters)
Some (firstPayload::restOfPayloads)
let schemaPayload =
@ -289,15 +294,19 @@ module private Parameters =
let parameterPayload = generateGrammarElementForSchema
p.ActualSchema
(specExampleValue, true) [] id
(specExampleValue, true)
trackParameters
[] id
// Add the name to the parameter payload
let parameterPayload =
match parameterPayload with
| LeafNode leafProperty ->
let leafNodePayload =
match leafProperty.payload with
| Fuzzable (Enum(propertyName, propertyType, values, defaultValue), x, y) ->
Fuzzable (Enum(p.Name, propertyType, values, defaultValue), x, y)
| Fuzzable (Enum(propertyName, propertyType, values, defaultValue), x, y, z) ->
Fuzzable (Enum(p.Name, propertyType, values, defaultValue), x, y, z)
| Fuzzable (a, b, c, _) ->
Fuzzable (a, b, c, if trackParameters then Some p.Name else None)
| _ -> leafProperty.payload
LeafNode { leafProperty with payload = leafNodePayload }
| InternalNode (internalNode, children) ->
@ -333,7 +342,8 @@ module private Parameters =
let getAllParameters (swaggerMethodDefinition:OpenApiOperation)
(parameterKind:NSwag.OpenApiParameterKind)
exampleConfig dataFuzzing =
exampleConfig dataFuzzing
trackParameters =
let localParameters = swaggerMethodDefinition.ActualParameters
|> Seq.filter (fun p -> p.Kind = parameterKind)
// add shared parameters for the endpoint, if any
@ -342,7 +352,7 @@ module private Parameters =
let allParameters =
[localParameters ; declaredSharedParameters ] |> Seq.concat
getParameters allParameters exampleConfig dataFuzzing
getParameters allParameters exampleConfig dataFuzzing trackParameters
let generateRequestPrimitives (requestId:RequestId)
(responseParser:ResponseParser option)
@ -615,6 +625,7 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
Parameters.pathParameters
m.Value ep
(if usePathExamples then exampleConfig else None)
config.TrackFuzzedParameterNames
RequestParameters.header =
let useHeaderExamples =
config.UseHeaderExamples |> Option.defaultValue false
@ -623,7 +634,7 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
OpenApiParameterKind.Header
(if useHeaderExamples then exampleConfig else None)
config.DataFuzzing
config.TrackFuzzedParameterNames
RequestParameters.query =
let useQueryExamples =
config.UseQueryExamples |> Option.defaultValue false
@ -632,6 +643,7 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
OpenApiParameterKind.Query
(if useQueryExamples then exampleConfig else None)
config.DataFuzzing
config.TrackFuzzedParameterNames
RequestParameters.body =
let useBodyExamples =
config.UseBodyExamples |> Option.defaultValue false
@ -640,12 +652,13 @@ let generateRequestGrammar (swaggerDocs:Types.ApiSpecFuzzingConfig list)
OpenApiParameterKind.Body
(if useBodyExamples then exampleConfig else None)
config.DataFuzzing
config.TrackFuzzedParameterNames
}
let allResponseProperties = seq {
for r in m.Value.Responses do
if validResponseCodes |> List.contains r.Key && not (isNull r.Value.ActualResponse.Schema) then
yield generateGrammarElementForSchema r.Value.ActualResponse.Schema (None, false) [] id
yield generateGrammarElementForSchema r.Value.ActualResponse.Schema (None, false) false [] id
}
// 'allResponseProperties' contains the schemas of all possible responses

7
src/compiler/Restler.Compiler/Config.fs
Просмотреть файл

@ -102,6 +102,11 @@ type Config =
// When specified, use only this naming convention to infer
// producer-consumer dependencies.
ApiNamingConvention : NamingConvention option
// When this switch is on, the generated grammar will contain
// parameter names for all fuzzable values. For example:
// restler_fuzzable_string("1", param_name="num_items")
TrackFuzzedParameterNames : bool
}
let convertToAbsPath (currentDirPath:string) (filePath:string) =
@ -209,6 +214,7 @@ let SampleConfig =
EngineSettingsFilePath = None
DataFuzzing = true
ApiNamingConvention = None
TrackFuzzedParameterNames = false
}
/// The default config used for unit tests. Most of these should also be the defaults for
@ -238,4 +244,5 @@ let DefaultConfig =
EngineSettingsFilePath = None
DataFuzzing = false
ApiNamingConvention = None
TrackFuzzedParameterNames = false
}

6
src/compiler/Restler.Compiler/Dependencies.fs
Просмотреть файл

@ -734,7 +734,7 @@ let getParameterDependencies parameterKind globalAnnotations
let resourceAccessPath = PropertyAccessPaths.getLeafAccessPath parentAccessPath p
let primitiveType =
match p.payload with
| FuzzingPayload.Fuzzable (pt, _, _) -> Some pt
| FuzzingPayload.Fuzzable (pt, _, _,_) -> Some pt
| FuzzingPayload.Constant (pt, _) -> Some pt
| FuzzingPayload.Custom c -> Some c.primitiveType
| _ -> None
@ -1218,7 +1218,7 @@ module DependencyLookup =
else
let defaultPayload =
match p.payload with
| None -> Fuzzable (PrimitiveType.String, "", None)
| None -> Fuzzable (PrimitiveType.String, "", None, None)
| Some p -> p
let propertyAccessPath =
{ path = PropertyAccessPaths.getInnerAccessPath resourceAccessPath p
@ -1235,7 +1235,7 @@ module DependencyLookup =
// First, check if the parameter itself has a dependency
let (parameterName, properties) = requestParameter.name, requestParameter.payload
let defaultPayload = (Fuzzable (PrimitiveType.String, "", None))
let defaultPayload = (Fuzzable (PrimitiveType.String, "", None, None))
let dependencyPayload = getConsumerPayload dependencies pathPayload requestId parameterName EmptyAccessPath defaultPayload
let payloadWithDependencies =

4
src/compiler/Restler.Compiler/Grammar.fs
Просмотреть файл

@ -135,9 +135,9 @@ type FuzzingPayload =
/// Example: (Int "1")
| Constant of PrimitiveType * string
/// (data type, default value, example value)
/// (data type, default value, example value, parameter name)
/// Example: (Int "1", "2")
| Fuzzable of PrimitiveType * string * string option
| Fuzzable of PrimitiveType * string * string option * string option
/// The custom payload, as specified in the fuzzing dictionary
| Custom of CustomPayload

85
src/compiler/Restler.Compiler/SwaggerVisitors.fs
Просмотреть файл

@ -65,8 +65,10 @@ module SchemaUtilities =
let getGrammarPrimitiveTypeWithDefaultValue (objectType:NJsonSchema.JsonObjectType)
(format:string)
(exampleValue:string option) :
(PrimitiveType * string * string option) =
(exampleValue:string option)
(propertyName : string option)
(trackParameters:bool) :
(PrimitiveType * string * string option * string option) =
let defaultTypeWithValue =
match objectType with
| NJsonSchema.JsonObjectType.String ->
@ -106,10 +108,13 @@ module SchemaUtilities =
raise (UnsupportedType (sprintf "%A is not a fuzzable primitive type. Please make sure your Swagger file is valid." objectType))
let (primitiveType, defaultValue) = defaultTypeWithValue
primitiveType, defaultValue, exampleValue
let propertyName =
if trackParameters then propertyName else None
primitiveType, defaultValue, exampleValue, propertyName
let getFuzzableValueForObjectType (objectType:NJsonSchema.JsonObjectType) (format:string) (exampleValue: string option) =
Fuzzable (getGrammarPrimitiveTypeWithDefaultValue objectType format exampleValue)
let getFuzzableValueForObjectType (objectType:NJsonSchema.JsonObjectType) (format:string) (exampleValue: string option) (propertyName: string option)
(trackParameters:bool) =
Fuzzable (getGrammarPrimitiveTypeWithDefaultValue objectType format exampleValue propertyName trackParameters)
/// Get a boolean property from 'ExtensionData', if it exists.
let getExtensionDataBooleanPropertyValue (extensionData:System.Collections.Generic.IDictionary<string, obj>) (extensionDataKeyName:string) =
@ -129,7 +134,8 @@ open SchemaUtilities
module SwaggerVisitors =
let getFuzzableValueForProperty propertyName (propertySchema:NJsonSchema.JsonSchema) isRequired isReadOnly enumeration defaultValue (exampleValue:string option) =
let getFuzzableValueForProperty propertyName (propertySchema:NJsonSchema.JsonSchema) isRequired isReadOnly enumeration defaultValue (exampleValue:string option)
(trackParameters:bool) =
let payload =
match propertySchema.Type with
| NJsonSchema.JsonObjectType.String
@ -137,23 +143,23 @@ module SwaggerVisitors =
| NJsonSchema.JsonObjectType.Integer
| NJsonSchema.JsonObjectType.Boolean ->
match enumeration with
| None -> getFuzzableValueForObjectType propertySchema.Type propertySchema.Format exampleValue
| None -> getFuzzableValueForObjectType propertySchema.Type propertySchema.Format exampleValue (Some propertyName) trackParameters
| Some ev ->
let enumValues = ev |> Seq.map (fun e -> string e) |> Seq.toList
let grammarPrimitiveType,_,exv = getGrammarPrimitiveTypeWithDefaultValue propertySchema.Type propertySchema.Format exampleValue
let grammarPrimitiveType,_,exv,_ = getGrammarPrimitiveTypeWithDefaultValue propertySchema.Type propertySchema.Format exampleValue (Some propertyName) trackParameters
let defaultFuzzableEnumValue =
match enumValues with
| [] -> "null"
| h::rest -> h
Fuzzable (PrimitiveType.Enum (propertyName, grammarPrimitiveType, enumValues, defaultValue), defaultFuzzableEnumValue, exv)
Fuzzable (PrimitiveType.Enum (propertyName, grammarPrimitiveType, enumValues, defaultValue), defaultFuzzableEnumValue, exv, None)
| NJsonSchema.JsonObjectType.Object
| NJsonSchema.JsonObjectType.None ->
// Example of JsonObjectType.None: "content": {} without a type specified in Swagger.
// Let's treat these the same as Object.
getFuzzableValueForObjectType NJsonSchema.JsonObjectType.Object propertySchema.Format exampleValue
getFuzzableValueForObjectType NJsonSchema.JsonObjectType.Object propertySchema.Format exampleValue (Some propertyName) trackParameters
| NJsonSchema.JsonObjectType.File ->
// Fuzz it as a string.
Fuzzable (PrimitiveType.String, "file object", None)
Fuzzable (PrimitiveType.String, "file object", None, if trackParameters then Some propertyName else None)
| nst ->
raise (UnsupportedType (sprintf "Unsupported type formatting: %A" nst))
{ LeafProperty.name = propertyName; payload = payload ;isRequired = isRequired ; isReadOnly = isReadOnly }
@ -168,7 +174,23 @@ module SwaggerVisitors =
then None
else Some (property.Default.ToString())
/// Add property name to the fuzzable payload
/// This is used to track fuzzed parameter values
let addTrackedParameterName (tree:Tree<LeafProperty, InnerProperty>) paramName trackParameters =
if trackParameters then
match tree with
| LeafNode leafProperty ->
let payload = leafProperty.payload
match payload with
| Fuzzable(a, b, c, _) ->
let payload = Fuzzable (a, b, c, Some paramName)
LeafNode { leafProperty with LeafProperty.payload = payload }
| x -> tree
| InternalNode (_,_) -> tree
else tree
module GenerateGrammarElements =
/// Returns the specified property when the object contains it.
/// Note: if the example object does not contain the property,
let extractPropertyFromObject propertyName (objectType:NJsonSchema.JsonObjectType)
@ -258,6 +280,7 @@ module SwaggerVisitors =
let rec processProperty (propertyName, property:NJsonSchema.JsonSchemaProperty)
(propertyPayloadExampleValue: JToken option, generateFuzzablePayload:bool)
(trackParameters:bool)
(parents:NJsonSchema.JsonSchema list)
(cont: Tree<LeafProperty, InnerProperty> -> Tree<LeafProperty, InnerProperty>) =
@ -285,6 +308,7 @@ module SwaggerVisitors =
(tryGetEnumeration propertySchema)
(tryGetDefault propertySchema)
schemaExampleValue
trackParameters
let propertyPayload =
match propertyPayloadExampleValue with
| None ->
@ -293,13 +317,13 @@ module SwaggerVisitors =
| Some v ->
let examplePropertyPayload =
match fuzzablePropertyPayload.payload with
| Fuzzable (primitiveType, defaultValue, _) ->
| Fuzzable (primitiveType, defaultValue, _, propertyName) ->
let payloadValue = GenerateGrammarElements.formatJTokenProperty primitiveType v
// Replace the default payload with the example payload, preserving type information.
// 'generateFuzzablePayload' is specified a schema example is found for the parent
// object (e.g. an array).
if generateFuzzablePayload then
Fuzzable (primitiveType, defaultValue, Some payloadValue)
Fuzzable (primitiveType, defaultValue, Some payloadValue, propertyName)
else
Constant (primitiveType, payloadValue)
| _ -> raise (invalidOp(sprintf "invalid payload %A, expected fuzzable" fuzzablePropertyPayload))
@ -312,7 +336,8 @@ module SwaggerVisitors =
// The example property value needs to be examined according to the 'ActualSchema', which
// is passed through below.
generateGrammarElementForSchema property.ActualSchema (propertyPayloadExampleValue, generateFuzzablePayload) parents (fun tree ->
generateGrammarElementForSchema property.ActualSchema (propertyPayloadExampleValue, generateFuzzablePayload)
trackParameters parents (fun tree ->
let innerProperty = { InnerProperty.name = propertyName
payload = None
propertyType = Property
@ -330,17 +355,23 @@ module SwaggerVisitors =
let arrayWithElements =
generateGrammarElementForSchema property (propertyPayloadExampleValue, generateFuzzablePayload)
parents (fun tree -> tree)
trackParameters parents (fun tree -> tree)
|> cont
// For tracked parameters, it is required to add the name to the fuzzable primitives
// This name will not be added if the array elements are leaf nodes that do not have inner properties.
match arrayWithElements with
| InternalNode (n, tree) ->
let tree =
tree
|> Seq.map (fun elem -> addTrackedParameterName elem propertyName trackParameters)
InternalNode (innerArrayProperty, tree)
| LeafNode _ ->
raise (invalidOp("An array should be an internal node."))
| NJsonSchema.JsonObjectType.Object ->
let objTree =
// This object may or may not have nested properties.
// Similar to type 'None', just pass through the object and it will be taken care of downstream.
generateGrammarElementForSchema property.ActualSchema (None, false) parents (fun tree ->
generateGrammarElementForSchema property.ActualSchema (None, false) trackParameters parents (fun tree ->
// If the object has no properties, it should be set to its primitive type.
match tree with
| LeafNode l ->
@ -355,11 +386,16 @@ module SwaggerVisitors =
InternalNode (innerProperty, stn tree)
|> cont
)
// For tracked parameters, it is required to add the name to the fuzzable primitives
// This name will not be added if the object elements are leaf nodes that do not have inner properties.
let objTree = addTrackedParameterName objTree propertyName trackParameters
objTree
| nst ->
raise (UnsupportedType (sprintf "Found unsupported type in body parameters: %A" nst))
and generateGrammarElementForSchema (schema:NJsonSchema.JsonSchema)
(exampleValue:JToken option, generateFuzzablePayloadsForExamples:bool)
(trackParameters:bool)
(parents:NJsonSchema.JsonSchema list)
(cont: Tree<LeafProperty, InnerProperty> -> Tree<LeafProperty, InnerProperty>) =
@ -380,7 +416,7 @@ module SwaggerVisitors =
// TODO: need test case for this example. Raise an exception to flag these cases.
// Most likely, the current example value should simply be used in the leaf property
raise (UnsupportedRecursiveExample (sprintf "%A" exampleValue))
let leafProperty = { LeafProperty.name = ""; payload = Fuzzable (PrimitiveType.String, "", None); isRequired = true ; isReadOnly = false }
let leafProperty = { LeafProperty.name = ""; payload = Fuzzable (PrimitiveType.String, "", None, None); isRequired = true ; isReadOnly = false }
LeafNode leafProperty
|> cont
else
@ -395,6 +431,7 @@ module SwaggerVisitors =
else
Some (processProperty (name, item.Value)
(exValue, generateFuzzablePayloadsForExamples)
trackParameters
(schema::parents) id))
let arrayProperties =
if schema.IsArray then
@ -414,12 +451,13 @@ module SwaggerVisitors =
let schemaArrayExamples = ExampleHelpers.tryGetArraySchemaExample schema
match schemaArrayExamples with
| None ->
generateGrammarElementForSchema schema.Item.ActualSchema (None, false) (schema::parents) id
generateGrammarElementForSchema schema.Item.ActualSchema (None, false) trackParameters (schema::parents) id
|> stn
| Some sae ->
sae |> Seq.map (fun (schemaExampleValue, generateFuzzablePayload) ->
generateGrammarElementForSchema schema.Item.ActualSchema
(schemaExampleValue, generateFuzzablePayload)
trackParameters
(schema::parents)
id
)
@ -432,6 +470,7 @@ module SwaggerVisitors =
generateGrammarElementForSchema
schema.Item.ActualSchema
(Some example, false)
trackParameters
(schema::parents)
id |> stn)
|> Seq.concat
@ -440,7 +479,7 @@ module SwaggerVisitors =
let allOfParameterSchemas =
schema.AllOf
|> Seq.map (fun ao -> ao, generateGrammarElementForSchema ao.ActualSchema (exampleValue, false) (schema::parents) id)
|> Seq.map (fun ao -> ao, generateGrammarElementForSchema ao.ActualSchema (exampleValue, false) trackParameters (schema::parents) id)
let allOfProperties =
allOfParameterSchemas
@ -496,6 +535,7 @@ module SwaggerVisitors =
(tryGetEnumeration schema) (*enumeration*)
(tryGetDefault schema) (*defaultValue*)
specExampleValue
trackParameters
LeafNode leafProperty
|> cont
| Some v ->
@ -509,13 +549,14 @@ module SwaggerVisitors =
let schemaType =
if schema.Type = JsonObjectType.None then JsonObjectType.Object
else schema.Type
let primitiveType, defaultValue, _ = getGrammarPrimitiveTypeWithDefaultValue schemaType schema.Format None
// Note: since the paramName is not available here, set it to 'None'.
// This will be fixed up from the parent level.
let primitiveType, defaultValue, _ , _ = getGrammarPrimitiveTypeWithDefaultValue schemaType schema.Format None None trackParameters
let leafPayload =
let exampleValue = GenerateGrammarElements.formatJTokenProperty primitiveType v
if generateFuzzablePayloadsForExamples then
FuzzingPayload.Fuzzable (primitiveType, defaultValue, Some exampleValue)
FuzzingPayload.Fuzzable (primitiveType, defaultValue, Some exampleValue, None)
else
FuzzingPayload.Constant (primitiveType, exampleValue)