2019-04-26 18:27:27 +03:00
Logging (C++)
-------------
Overview
Json schema: Take 3 (#113)
* Add a getApi method, listing all installed RPC method names
* Sketch RecordParams struct
* WIP
* Broken WIP
* Partial macro
* Basic examples working
* Partial file separation
* Move, rename, and fix FOR macro
* Use json get
* Build to_json from RequiredJsonFields too
* Remove unneeded pair specialisation
* Add comments, collide required and optional
* REformat
* Use new macros everywhere
* Remove unused template
* Rename getApi to listMethods
* Move frontend-specific calltypes to /rpc
* Specify GetTxHist return type
* Pretty-print client responses by default
* Add a GetSchema RPC
* Other tools demand ugly formatting by default
* mins and maxes for numerics, map of schemas
* Support _FOR_JSON_0
* Fix support for std::optional optional fields
* Test std optionals
* Define schemas for GetCommit
* More definitions for existing RPCs
* Tidy schema generation, including for vectors
* Add proper unit test
* Initial test of schema generation
* Fix failing tests
* Formatting
* Add (currently failing) test of nested structs
* Add misleadingly passing test
* Set correct expected pointers, test currently fails
* Oops - deexpand
* Correctly build pointer path for erroneous array elements
* Demonstrate invalid, not just missing, valeus
* Skeleton of json_bench
* Fix typo
* WIP
* Compare manual json parsers vs macro-defined
* mumble mumble
* Add valijson, +basic test
* Add benchmark of valijson validation
* Benchmark simple and complex structs
* Additional broken schema test
* Include pointer to parse errors
* Restore old basic translator macro
* Restore simpler macro for translators that don't need schema
* Add auto schema for private logging methods
* Add manual schema + validation for PUBLIC logging RPCs
* Match RPC format
* More RPC format fixes
* Correct scenario test target
* Add documentation entry on API schema
* Initial schema retrieval test
* Correct URLs in generated schema
* Send schema to a flat folder
* Remove unnecessary size_t max restriction
* Report non-matching schema
* Add current schemas
* Tidying
* clang-format
* Remove schema generation e2e test
* fmtlib, remove $id from schema
* Fix pointer paths
2019-06-05 12:36:50 +03:00
`` ` ` ` ` ``
2019-04-26 18:27:27 +03:00
A C++ transaction engine exposes itself to CCF by implementing:
.. literalinclude :: ../../src/enclave/appinterface.h
:language: cpp
:start-after: SNIPPET_START: rpc_handler
:end-before: SNIPPET_END: rpc_handler
:dedent: 2
The Logging application simply has:
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
:start-after: SNIPPET_START: rpc_handler
:end-before: SNIPPET_END: rpc_handler
:dedent: 2
.. note ::
:cpp:class: `kv::Store` tables are essentially the only interface between CCF
and the application, and the sole mechanism for it to have state.
2019-07-21 18:15:28 +03:00
The Logging application keeps its state in a pair of tables, one containing private encrypted logs and the other containing public unencrypted logs. Their type is defined as:
2019-04-26 18:27:27 +03:00
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
:start-after: SNIPPET: table_definition
:lines: 1
:dedent: 2
2019-07-21 18:15:28 +03:00
Table creation happens in the app's constructor:
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
:start-after: SNIPPET_START: constructor
:end-before: SNIPPET_END: constructor
:dedent: 4
2019-04-26 18:27:27 +03:00
RPC Handler
`` ` ` ` ` ` ` ` ``
The handler returned by :cpp:func: `ccfapp::getRpcHandler()` needs to subclass :cpp:class: `ccf::UserRpcFrontend` :
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
2019-07-21 18:15:28 +03:00
:start-after: SNIPPET: inherit_frontend
:lines: 1
:dedent: 2
2019-04-26 18:27:27 +03:00
2019-07-21 18:15:28 +03:00
The constructor then needs to create a handler function or lambda for each transaction type. This takes a transaction object and the request's `` params `` , interacts with the KV tables, and returns a result:
2019-04-26 18:27:27 +03:00
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
2019-07-21 18:15:28 +03:00
:start-after: SNIPPET_START: get
:end-before: SNIPPET_END: get
2019-04-26 18:27:27 +03:00
:dedent: 6
2019-07-21 18:15:28 +03:00
Each function is installed as the handler for a specific RPC `` method `` , optionally with schema included:
2019-04-26 18:27:27 +03:00
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
2019-07-21 18:15:28 +03:00
:start-after: SNIPPET: install_get
2019-04-26 18:27:27 +03:00
:lines: 1
:dedent: 6
2019-07-21 18:15:28 +03:00
A handler can either be installed as:
2019-04-26 18:27:27 +03:00
2019-08-15 19:52:43 +03:00
- `` Write `` : this handler can only be executed on the primary of the consensus network.
2019-04-26 18:27:27 +03:00
- `` Read `` : this handler can be executed on any node of the network.
- `` MayWrite `` : the execution of this handler on a specific node depends on the value of the `` "readonly" `` paramater in the JSON-RPC command.
2019-07-21 18:15:28 +03:00
App-defined errors
..................
Applications can define their own error codes. These should be between `` -32050 `` and `` -32099 `` to avoid conflicting with CCF's error codes. The Logging application returns errors if the user tries to get an ID which has not been logged, or tries to log an empty message. These error codes should be given their own `` enum class `` , and a `` get_error_prefix `` function should be defined in the same namespace to help users distinguish error messages:
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
:start-after: SNIPPET_START: errors
:end-before: SNIPPET_END: errors
:dedent: 2
Json schema: Take 3 (#113)
* Add a getApi method, listing all installed RPC method names
* Sketch RecordParams struct
* WIP
* Broken WIP
* Partial macro
* Basic examples working
* Partial file separation
* Move, rename, and fix FOR macro
* Use json get
* Build to_json from RequiredJsonFields too
* Remove unneeded pair specialisation
* Add comments, collide required and optional
* REformat
* Use new macros everywhere
* Remove unused template
* Rename getApi to listMethods
* Move frontend-specific calltypes to /rpc
* Specify GetTxHist return type
* Pretty-print client responses by default
* Add a GetSchema RPC
* Other tools demand ugly formatting by default
* mins and maxes for numerics, map of schemas
* Support _FOR_JSON_0
* Fix support for std::optional optional fields
* Test std optionals
* Define schemas for GetCommit
* More definitions for existing RPCs
* Tidy schema generation, including for vectors
* Add proper unit test
* Initial test of schema generation
* Fix failing tests
* Formatting
* Add (currently failing) test of nested structs
* Add misleadingly passing test
* Set correct expected pointers, test currently fails
* Oops - deexpand
* Correctly build pointer path for erroneous array elements
* Demonstrate invalid, not just missing, valeus
* Skeleton of json_bench
* Fix typo
* WIP
* Compare manual json parsers vs macro-defined
* mumble mumble
* Add valijson, +basic test
* Add benchmark of valijson validation
* Benchmark simple and complex structs
* Additional broken schema test
* Include pointer to parse errors
* Restore old basic translator macro
* Restore simpler macro for translators that don't need schema
* Add auto schema for private logging methods
* Add manual schema + validation for PUBLIC logging RPCs
* Match RPC format
* More RPC format fixes
* Correct scenario test target
* Add documentation entry on API schema
* Initial schema retrieval test
* Correct URLs in generated schema
* Send schema to a flat folder
* Remove unnecessary size_t max restriction
* Report non-matching schema
* Add current schemas
* Tidying
* clang-format
* Remove schema generation e2e test
* fmtlib, remove $id from schema
* Fix pointer paths
2019-06-05 12:36:50 +03:00
API Schema
..........
These handlers also demonstrate two different ways of defining schema for RPCs, and validating incoming requests against them. The record/get methods operating on public tables have manually defined schema and use [#valijson]_ for validation, returning an error if the input is not compliant with the schema:
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
:start-after: SNIPPET_START: valijson_record_public
:end-before: SNIPPET_END: valijson_record_public
:dedent: 6
This provides robust, extensible validation using the full JSON schema spec.
The methods operating on private tables use an alternative approach, with a macro-generated schema and parser converting compliant requests into a PoD C++ object:
.. literalinclude :: ../../src/apps/logging/logging_schema.h
:language: cpp
:start-after: SNIPPET_START: macro_validation_macros
:end-before: SNIPPET_END: macro_validation_macros
:dedent: 2
.. literalinclude :: ../../src/apps/logging/logging.cpp
:language: cpp
:start-after: SNIPPET_START: macro_validation_record
:end-before: SNIPPET_END: macro_validation_record
:dedent: 6
This produces validation error messages with a lower performance overhead, and ensures the schema and parsing logic stay in sync, but is only suitable for simple schema with required and optional fields of supported types.
Both approaches register their RPC's params and result schema, allowing them to be retrieved at runtime with calls to the getSchema RPC.
2019-04-26 18:27:27 +03:00
Build
`` ` ``
Once an application is complete, it needs be built into a shared object, and signed:
.. literalinclude :: ../../CMakeLists.txt
:language: cmake
:start-after: SNIPPET: Logging application
:lines: 1
For signing to work, a configuration is necessary. The configuration should be called `oe_sign.conf` , and
be placed under the same directory as the source files for the application.
.. literalinclude :: ../../src/apps/logging/oe_sign.conf
Running
`` ` ` ` ``
This produces the enclave library `` libloggingenc.so.signed `` which can be loaded by the cchost application:
.. code-block :: bash
./cchost --enclave-file libloggingenc.so.signed [args]
Json schema: Take 3 (#113)
* Add a getApi method, listing all installed RPC method names
* Sketch RecordParams struct
* WIP
* Broken WIP
* Partial macro
* Basic examples working
* Partial file separation
* Move, rename, and fix FOR macro
* Use json get
* Build to_json from RequiredJsonFields too
* Remove unneeded pair specialisation
* Add comments, collide required and optional
* REformat
* Use new macros everywhere
* Remove unused template
* Rename getApi to listMethods
* Move frontend-specific calltypes to /rpc
* Specify GetTxHist return type
* Pretty-print client responses by default
* Add a GetSchema RPC
* Other tools demand ugly formatting by default
* mins and maxes for numerics, map of schemas
* Support _FOR_JSON_0
* Fix support for std::optional optional fields
* Test std optionals
* Define schemas for GetCommit
* More definitions for existing RPCs
* Tidy schema generation, including for vectors
* Add proper unit test
* Initial test of schema generation
* Fix failing tests
* Formatting
* Add (currently failing) test of nested structs
* Add misleadingly passing test
* Set correct expected pointers, test currently fails
* Oops - deexpand
* Correctly build pointer path for erroneous array elements
* Demonstrate invalid, not just missing, valeus
* Skeleton of json_bench
* Fix typo
* WIP
* Compare manual json parsers vs macro-defined
* mumble mumble
* Add valijson, +basic test
* Add benchmark of valijson validation
* Benchmark simple and complex structs
* Additional broken schema test
* Include pointer to parse errors
* Restore old basic translator macro
* Restore simpler macro for translators that don't need schema
* Add auto schema for private logging methods
* Add manual schema + validation for PUBLIC logging RPCs
* Match RPC format
* More RPC format fixes
* Correct scenario test target
* Add documentation entry on API schema
* Initial schema retrieval test
* Correct URLs in generated schema
* Send schema to a flat folder
* Remove unnecessary size_t max restriction
* Report non-matching schema
* Add current schemas
* Tidying
* clang-format
* Remove schema generation e2e test
* fmtlib, remove $id from schema
* Fix pointer paths
2019-06-05 12:36:50 +03:00
.. rubric :: Footnotes
.. [#valijson] `Valijson is a header-only JSON Schema Validation library for C++11 <https://github.com/tristanpenman/valijson> `_ .