* Omitting authentication tokens from the request text while logging to the trace databases when no_tokens_in_logs setting is set to true
* Fixing the code to address unit test failures realted to the changes
* Have to pass the attribute no_tokens_in_logs, without which the setting will not be honored
---------
Co-authored-by: Swamy Nallamalli <swamyn@ntdev.microsoft.com>
* Fix Content-Type custom payload bugs
When a custom payload body is used to work around unsupported content types, the Content-Type
header must also be modified via the dictionary. This functionality had several bugs, which
are fixed in this change. After this update, using either `restler_custom_payload_header` or
`restler_custom_payload` with the request-specific syntax for dictionary payloads now works
as expected, for example:
"restler_custom_payload": {
"/stores/{storeId}/order/post/Content-Type": [ "xml" ]
}
This change also fixes the same bugs with per-request syntax with restler_custom_payload_query.
Testing:
- manual testing with demo_server
- added new compiler tests
* Fix prefix computation per PR feedback
Per discussion in #823, updating the README and updating the default mode to be `bfs-cheap`.
Originally, the reasoning behind the default value of `bfs-fast` was that it provides different coverage from `fuzz-lean`
(which is similar to `bfs-cheap` in that it does not exercise all of the renderings).
However, many users only want to run one fuzzing mode (e.g. for evaluation purposes, having just one job in CI/CD), and
that is typically the `fuzz` mode (since we document it to provide the best coverage).
Updating the documentation to reflect this.
Currently, the fuzzing timeout is 1 hour by default. This causes incorrect/misleading results
for more complex services that require a longer fuzzing time,
if timeout is accidentally omitted due to a configuration issue.
Providing a very long timeout in an attempt to mitigate this issue. (Note: this was already the default
timeout in the RESTler engine - this change is only for the driver).
To restore the default timeout, add the following to the engine settings:
'time_budget': 1
For use cases when a trace database is saved and re-played,
this change allows using only replay blocks without the rendered requests.
This is being added for scenarios where only replay blocks are saved during
custom serialization.
The rendered requests continue to be logged by default, since they are
useful for debugging.
Testing:
- manual testing
- new unit test that replays from ndjson with vs. without the request text
This reverts commit 86581a2c21.
This change needs to be re-submitted after refactoring the compiler config to
match the optional fields that are currently documented.
This change enables a new 'replay' mode option that replays requests from the trace database.
Changes include:
1. Refactor driver to consume sequences from different sources (with trace database another supported source of specific
sequences, similar to the existing smoke test mode)
2. Generate 'replay blocks' as part of payload rendering. Replay blocks contain dynamic objects, but do not support
custom payloads (this is future work).
3. Enables replay to be based on replay blocks
With this change, RESTler generates sequences from replay blocks if available,
or from raw request/responses (similar to the existing replay mode) when they are not available.
The current version of replay is based on the rendered data sent at the time of
original sequence rendering. This has several limitations, such as not being able to
garbage-collect resources or plugging in custom payloads.
In particular, when a bug is reproduced in RESTler today, the same replay mechanism is used, which
means that GC does not collect resources created while reproducing the bug. Moreover, re-running
the same sequence without GC will not work if the resource has not been deleted if a resource-generating
request has a unique ID parameter (e.g. PUT /resource/{resourceId}), causing false negative non-reproducible
bugs.
This change addresses this issue by implementing replay blocks-based replay, which is invoked when reproducing bugs.
Note: the existing replay based on .replay.txt files will also be able to use this new mechanism if a grammar is provided,
but this is not implemented as part of this change.
4. Added a setting to filter the origin during DB replay.
To only replay specific origins, add the following to the settings file:
"replay": {
"include_origins": ["main_driver", "InvalidValueChecker"]
}
Testing:
- manual testing of demo_server replay as follows:
1) Run 'test' task and generate trace database
>restler.exe test --grammar_file .\Compile\grammar.py --dictionary_file .\Compile\dict.json --host localhost --target_port 8888 --settings .\compile\engine_settings.json --no_ssl
Engine settings:
{
"use_trace_database": true,
"trace_database": {
"root_dir": "d:\\demo_server\\trace_databases",
},
}
2) Run 'replay' task from above trace database
>restler.exe replay --replay_log ./trace_data.ndjson --grammar_file .\Compile\grammar.py --dictionary_file .\Compile\dict.json --host localhost --target_port 8888 --settings .\compile\engine_settings.json --no_ssl
- updated unit tests
- Also bump sqlmodel version
- A few app issues needed to be fixed as part of this upgrade:
1) Use union type to be able to return None and trigger one of the planted bugs
2) Work around OverflowError: Python int too large to convert to SQLite INTEGER
3) adjust baseline since 500 is no longer getting returned after pydantic upgrade
The payload {"body":0} now generates a well-formed error response before reaching the demo_server implementation
4) Update baseline for use after free checker
This checker now returns a 500
* driver: set origin when performing replay
* test_basic_functionality_end_to_end: validate that origin is set
* test_basic_functionality_end_to_end: add replay file trace db test
* unit_tests: rename test_replay.txt to replay_sanity_test_for_tracedb.replay.txt
* driver: clear origin after sequence replay
* unit tests: add additional logging
* restler.py: wait for trace_db thread when replaying file
---------
Co-authored-by: William Baker <William.Baker@microsoft.com>
Add a setting that, when true, adds a unique ID for every request sent.
This setting is enabled by default.
Testing:
- manual testing
Note: the baseline diffs ignore headers, which is why baselines did not need to be updated.
This change adds structured logging for all of the network traffic that is currently
logged to the network.*.txt log. The data is written to a 'trace database'.
The default format is newline-delimited json, but custom logging formats
are supported via a module specified in the engine settings (similar to checkers).
Below is an example for demo_server:
```
{"sent_timestamp": "2023-11-02T10:10:59.544824+00:00", "received_timestamp": null, "request": "GET /api/blog/posts?page=1&per_page=1 HTTP/1.1\r\nAccept: application/json\r\nHost: localhost\r\nContent-Length: 0\r\nr\n\r\n", "response": null, "request_json": "null", "response_json": "null", "tags": {"request_id": "27f9653431313fdc3fecc4a890b72b80b4ce1e59", "sequence_id": "fe2172ab-151c-419f-b918-f3e7483b3230", "combination_id": "1950cbddab7726489624c3d346d3426561c921ad_1", "hex_definition": "6daf8d22c7a6b3472fc83c9f08f290b3507c3ff3", "origin": "main_driver"}}
{"sent_timestamp": null, "received_timestamp": "2023-11-02T10:10:59.597613+00:00", "request": null, "response": "HTTP/1.1 400 Bad Request\r\ndate: Thu, 02 Nov 2023 10:10:58 GMT\r\nserver: uvicorn\r\ncontent-length: 41\r\ncontent-type: application/json\r\n\r\n{\"detail\":\"per_page must be at least 2.\"}", "request_json": "null", "response_json": "null", "tags": {"request_id": "27f9653431313fdc3fecc4a890b72b80b4ce1e59", "sequence_id": "fe2172ab-151c-419f-b918-f3e7483b3230", "combination_id": "1950cbddab7726489624c3d346d3426561c921ad_1", "hex_definition": "6daf8d22c7a6b3472fc83c9f08f290b3507c3ff3", "origin": "main_driver"}}
```
Testing:
- demo_server
- added new unit test
- manual testing for multiple db files
This bug was found when specifying a `producer_timing_delay` for a resource that was
also specified as `create_once`. Here, the length of the sequence was 1, and RESTler
proceeded fuzzing with the `create_once` resource in a not-ready state.
By default, change the behavior to always wait after the resource creation.
This change adds two new settings, as specified in #780.
- A specific `random_seed` may now be set in the engine settings, which will be used everywhere random values are generated, except where a different random_seed is specified to checkers through the checker settings.
- An option to generate a new random seed `generate_random_seed`, which is helpful for CI/CD cases that run in `random-walk` mode
and would like to get different sequences exercised on every run
Testing:
- added new test
When one of the requests (e.g. DELETE) does not have an example payload, and testing all example payloads is configured in the engine settings, RESTler crashes [1].
The fix is to only mark combinations tested after at least one combination was found.
Exception in thread Garbage Collector:
Traceback (most recent call last):
File "C:\Users\marinapo\AppData\Local\Programs\Python\Python39\lib\threading.py", line 980, in _bootstrap_inner
self.run()
File "D:\restlerdrop\main\engine\engine\dependencies.py", line 627, in run
self._garbage_collector.run()
File "D:\restlerdrop\main\engine\engine\dependencies.py", line 339, in run
self.do_garbage_collection()
File "D:\restlerdrop\main\engine\engine\dependencies.py", line 429, in do_garbage_collection
self.apply_destructors(destructors)
File "D:\restlerdrop\main\engine\engine\dependencies.py", line 562, in apply_destructors
deleted_list = process_overflowing()
File "D:\restlerdrop\main\engine\engine\dependencies.py", line 514, in process_overflowing
rendered_data, _ , _, _ = destructor.\
File "D:\restlerdrop\main\engine\engine\core\requests.py", line 1276, in render_current
return next(self.render_iter(candidate_values_pool,
Closes#787 restler_custom_payload is not correctly plugged into examples
Also fix incorrectly passed 'quoted' parameters - this was caught by manually inspecting the logs and seeing incorrect quoting for several primitives when using the generated schema only.
Updated test baselines for invalidvalue after the bug fix.
Testing:
- confirmed the manual repro from #787 now works correctly
The checker was logging tokens to its checker log. This change removes
the statement that logged the payload separately in this checker log,
which is not necessary. The payloads are already logged to the network log,
which can be used for debugging the sent requests.
Testing:
- manual testing
Log the values (excluding tokens) to help pinpoint unexpected types of values in the request block.
Testing:
- manual testing with invalid dictionary ('restler_fuzzable_int' containing integer instead of string)
Changes in this version:
- support path parameters embedded in
- support equality constraint for uuid4_suffixes in the same request payload
- allow specifying a custom User-Agent
- minor improvements / bug fixes
Previously, ```Content-Length``` could only be specified as a ```restler_custom_payload_header```, which injects the header for
every request. This change allows 'Content-Length' to also be specified as a custom payload, which only includes it in requests that declare it in the specification.
Testing:
- manual testing
The specified annotation dependency is being validated after the previous change. However, GET producers should
be allowed with a user-specified annotation without having to specify 'allowGetProducers' globally.
Testing:
- manual testing
Some request types have payloads with a unique value generated via uuid4_suffix (e.g. a PUT request that must create a new ID every time), and require the same value to be passed in multiple places in the payload. RESTler did not previously have a way to specify the equality constraints on such payload parameters or properties.
This change enables another property to be assigned the same value as the uuid4 suffix via an annotation, for example:
```
{
"producer_endpoint": "/archive/{archiveId}",
"producer_method": "PUT",
"consumer_endpoint": "/archive/{archiveId}",
"consumer_method": "PUT",
"producer_resource_name": "archiveId",
"consumer_param": "/id"
}
```
Testing:
- added new unit test
Partially implements #89
Previously, RESTler only supported path parameters where the parameter fills the entire path part.
However, OpenAPI specs allow parameters to only take up part of the path, for example, "/vm({id})/deployment{id}{operation}cancel
This change supports such patterns in the compiler.
Closes#705
Also partially fixes#141
Testing:
- added new compiler test
Per customer feedback, these two options are not easily discoverable.
This change adds them to the default generated dictionary when compiling an OpenAPI spec directly.
Closes#742
Testing:
- Manual testing:
> restler compile --specs ./swagger.json
Swamy Nallamalli
marina-p
Cokun01
Dominik Stadler
Denis Porfiryev
Eddy Ashton
William Baker
Tim Thacker
Leon
Nicolas Oliver
Alessandro