Document proposal_generator.py (#1440)

doc/developers/kv/kv_how_to.rst

@@ -10,7 +10,7 @@ The Key-Value :cpp:class:`kv::Store` is a collection of :cpp:class:`kv::Maps` th
 Creating a Map
 --------------
 
-A :cpp:class:`kv::Map` (often referred to as a ``Table``) is created in the constructor of an application. It maps a unique ``key`` to a ``value``.
+A :cpp:type:`kv::Map` (often referred to as a ``Table``) is created in the constructor of an application. It maps a unique ``key`` to a ``value``.
 
 When a ``Map`` is created, its name and the types of the key and value mapping should be specified.
 

doc/developers/kv/kv_serialisation.rst

@@ -3,7 +3,7 @@ KV Serialisation
 
 Every transaction executed by the primary on its key-value store is serialised before being replicated to all backups of the CCF network and written to the ledger.
 
-Writes to private :cpp:class:`kv::Map`\s are encrypted before being written to the ledger, and can only be decrypted by other nodes within the service. Writes to public :cpp:class:`kv::Map`\s are only integrity-protected; they are readable by anyone with access to the ledger.
+Writes to private :cpp:type:`kv::Map`\s are encrypted before being written to the ledger, and can only be decrypted by other nodes within the service. Writes to public :cpp:type:`kv::Map`\s are only integrity-protected; they are readable by anyone with access to the ledger.
 
 .. note:: Transactions are serialised to MessagePack_ with a prepended header for integrity protection.
 
@@ -27,8 +27,8 @@ The following table describes the structure of a serialised KV Store transaction
 +          +------------------------------------------+-------------------------------------------------------------------------+
 |          | **Repeating [0..n]**                     | With ``n`` the number of maps in the transaction                        |
 +          +-----+------------------------------------+-------------------------------------------------------------------------+
-|          |     | | ``KOT_MAP_START_INDICATOR``      | | Indicates the start of a new serialised :cpp:class:`kv::Map`          |
-|          |     | | char[]                           | | Name of the serialised :cpp:class:`kv::Map`                           |
+|          |     | | ``KOT_MAP_START_INDICATOR``      | | Indicates the start of a new serialised :cpp:type:`kv::Map`           |
+|          |     | | char[]                           | | Name of the serialised :cpp:type:`kv::Map`                            |
 |          +-----+------------------------------------+-------------------------------------------------------------------------+
 |          |     | | :cpp:type:`kv::Version`          | | Read version                                                          |
 |          +-----+------------------------------------+-------------------------------------------------------------------------+
@@ -63,7 +63,7 @@ The following table describes the structure of a serialised KV Store transaction
 Custom key and value types
 --------------------------
 
-User-defined types can be used for both the key and value types of a :cpp:class:`kv::Map`. It must be possible to use the key type as the key of an ``std::map`` (so it must be copyable, assignable, and less-comparable), and both types must be serialisable. By default, when using a :cpp:class:`kv::Map`, serialisation converts to `MessagePack`_ using `msgpack-c`_. To add support to your custom types, it should usually be possible to use the ``MSGPACK_DEFINE`` macro:
+User-defined types can be used for both the key and value types of a :cpp:type:`kv::Map`. It must be possible to use the key type as the key of an ``std::map`` (so it must be copyable, assignable, and less-comparable), and both types must be serialisable. By default, when using a :cpp:type:`kv::Map`, serialisation converts to `MessagePack`_ using `msgpack-c`_. To add support to your custom types, it should usually be possible to use the ``MSGPACK_DEFINE`` macro:
 
 .. literalinclude:: ../../../src/kv/test/kv_serialisation.cpp
     :language: cpp

doc/members/accept_recovery.rst

@@ -115,8 +115,3 @@ Summary Diagram
 
         Note over Node 2: Recovery procedure complete
         Note over Node 3: Recovery procedure complete
-
-
-.. rubric:: Footnotes
-
-.. [#recovery_share] Recovery shares are encrypted with the respective member public key and stored in CCF. As such, a recovery share can only be retrieved and used by the member who owns it.

doc/members/constitution.rst

@@ -31,7 +31,7 @@ It is possible for a set of operators to host a CCF network without being member
 In case of catastrophic failure, operators could also:
 
 - Start a network in recovery mode from the ledger
-- Hand it over to the members for them to Open (see :ref:`members/accepting_recovery_and_submitting_shares:Accepting Recovery`)
+- Hand it over to the members for them to Open (see :ref:`members/accept_recovery:Accepting Recovery`)
 
 Finally, operators could:
 

doc/members/proposals.rst

@@ -3,12 +3,70 @@ Proposing and Voting for a Proposal
 
 This page explains how members can submit and vote for proposals.
 
+Proposals and vote ballots are submitted as Lua scripts. These scripts are executed transactionally, able to read from the current KV state but not write directly to it. Proposals return a list of proposed actions which can make writes, but are only applied when the proposal is accepted. Each vote script is given this list of proposed calls, and also able to read from the KV, and returns a boolean indicating whether it supports or rejects the proposed actions.
+
 Any member can submit a new proposal. All members can then vote on this proposal using its unique proposal id. Each member may alter their vote (by submitting a new vote) any number of times while the proposal is open. The member who originally submitted the proposal (the `proposer`) votes for the proposal by default, but has the option to include a negative or conditional vote like any other member. Additionally, the proposer has the ability to `withdraw` a proposal while it is open.
 
 Each time a vote is submitted, all vote ballots for this proposal are re-executed on the current state to determine whether they are `for` or `against` the proposal. This vote tally is passed to the :term:`Constitution`, which determines whether the proposal is accepted or remains open. Once a proposal is accepted under the rules of the :term:`Constitution`, it is executed and its effects are recorded in the ledger.
 
 For transparency and auditability, all governance operations (including votes) are recorded in plaintext in the ledger and members are required to sign their requests.
 
+Creating a Proposal
+-------------------
+
+For custom proposals with multiple actions and precise conditional requirements you will need to write the proposal script by hand. For simple proposals there is a helper script in the CCF Python package - `proposal_generator.py`. This can be used to create proposals for common operations like adding members and users, without writing any Lua. It also produces sample vote scripts, which validate that the executed proposed actions exactly match what is expected. These sample proposals and votes can be used as a syntax and API reference for producing more complex custom proposals.
+
+Assuming the CCF Python package has been installed in the current Python environment, the proposal generator can be invoked directly as ``ccf.proposal_generator``. With no further argument it will print help text, including the list of possible actions as subcommands:
+
+.. code-block:: bash
+
+    $ python -m ccf.proposal_generator
+    usage: proposal_generator.py [-h] [-po PROPOSAL_OUTPUT_FILE]
+                             [-vo VOTE_OUTPUT_FILE] [-pp] [-i]
+                             [--vote-against] [-v]
+                             {accept_recovery,new_member,new_node_code,new_user,new_user_code,open_network,rekey_ledger,remove_user,retire_member,retire_node,set_js_app,set_lua_app,set_recovery_threshold,set_user_data,trust_node,update_recovery_shares}
+
+Additional detail is available from the ``--help`` option. You can also find the script in a checkout of CCF:
+
+.. code-block:: bash
+
+    $ python CCF/python/ccf/proposal_generator.py
+
+Some of these subcommands require additional arguments, such as the node ID or user certificate to add to the service. Additional options allow the generated votes and proposals to be redirected to other files or pretty-printed:
+
+.. code-block:: bash
+
+    $ python -m ccf.proposal_generator trust_node 5
+    SUCCESS | Writing proposal to ./trust_node_proposal.json
+    SUCCESS | Wrote vote to ./trust_node_vote_for.json
+
+    $ cat trust_node_proposal.json 
+    {"script": {"text": "tables, args = ...; return Calls:call(\"trust_node\", args)"}, "parameter": "5"}
+
+    $ cat trust_node_vote_for.json 
+    {"ballot": {"text": "tables, calls = ...; if not #calls == 1 then return false end; call = calls[1]; if not call.func == \"trust_node\" then return false end; args = call.args; if args == nil then return false end; if not args == [====[5]====] then return false end; return true"}}
+
+    $python -m ccf.proposal_generator --pretty-print --proposal-output-file add_pedro.json --vote-output-file vote_for_pedro.json new_user pedro_cert.pem 
+    SUCCESS | Writing proposal to ./add_pedro.json
+    SUCCESS | Wrote vote to ./vote_for_pedro.json
+    
+    $ cat add_pedro.json 
+    {
+      "script": {
+        "text": "tables, args = ...; return Calls:call(\"new_user\", args)"
+      },
+      "parameter": "-----BEGIN CERTIFICATE-----\nMIIBrzCCATSgAwIBAgIUJY+H0OzuFQWz/udd+WCD7Cv+cgwwCgYIKoZIzj0EAwMw\nDjEMMAoGA1UEAwwDYm9iMB4XDTIwMDcyNDE1MzYyOFoXDTIxMDcyNDE1MzYyOFow\nDjEMMAoGA1UEAwwDYm9iMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE7h75Xd1+0QDD\nWF2edGphgryHcDoBXdRowq6ciYH2++ilXXagi5Rybai7ewgV0YuvrDm+WfGyJ9CC\n5HbT6C/z5GCJQnLH2t3LaZrw9MQDF3bH6XOHGmaJh6m7rfpZZljpo1MwUTAdBgNV\nHQ4EFgQUN/LhCyVExERjt5f1RZx7820934wwHwYDVR0jBBgwFoAUN/LhCyVExERj\nt5f1RZx7820934wwDwYDVR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNpADBmAjEA\n5MsDNvjEMSgYXy+bPbE2nxOlmH6OhP375IVZxNQALJGzTfgHu+IbpyvDF0/VrMrW\nAjEA723VxgMgpuxB5SszN6eZuz8EW51DsgRIVWMSbBZYYBYyQmu5x3T+Hx/Cs7TD\nu4Ee\n-----END CERTIFICATE-----\n"
+    }
+    
+    $ cat vote_for_pedro.json 
+    {
+      "ballot": {
+        "text": "tables, calls = ...; if not #calls == 1 then return false end; call = calls[1]; if not call.func == \"new_user\" then return false end; args = call.args; if args == nil then return false end; if not args == [====[-----BEGIN CERTIFICATE-----\nMIIBrzCCATSgAwIBAgIUJY+H0OzuFQWz/udd+WCD7Cv+cgwwCgYIKoZIzj0EAwMw\nDjEMMAoGA1UEAwwDYm9iMB4XDTIwMDcyNDE1MzYyOFoXDTIxMDcyNDE1MzYyOFow\nDjEMMAoGA1UEAwwDYm9iMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE7h75Xd1+0QDD\nWF2edGphgryHcDoBXdRowq6ciYH2++ilXXagi5Rybai7ewgV0YuvrDm+WfGyJ9CC\n5HbT6C/z5GCJQnLH2t3LaZrw9MQDF3bH6XOHGmaJh6m7rfpZZljpo1MwUTAdBgNV\nHQ4EFgQUN/LhCyVExERjt5f1RZx7820934wwHwYDVR0jBBgwFoAUN/LhCyVExERj\nt5f1RZx7820934wwDwYDVR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNpADBmAjEA\n5MsDNvjEMSgYXy+bPbE2nxOlmH6OhP375IVZxNQALJGzTfgHu+IbpyvDF0/VrMrW\nAjEA723VxgMgpuxB5SszN6eZuz8EW51DsgRIVWMSbBZYYBYyQmu5x3T+Hx/Cs7TD\nu4Ee\n-----END CERTIFICATE-----\n]====] then return false end; return true"
+      }
+    }
+
+These proposals and votes should be sent as the body of HTTP requests as described below.
+
 Submitting a New Proposal
 -------------------------
 

doc/operators/container.rst

@@ -5,7 +5,7 @@ With every release, a sample runtime base container is provided. It contains the
 but no particular enclave file, and may be helpful when deploying CCF nodes via docker, k8s etc.
 
 That image is optimised for size above all. If you need an image that comes with peripheral utilities,
-you probably want the :doc:`Build Container <developers/build_app>` instead.
+you probably want the :ref:`Build Container <developers/build_app:Build Container>` instead.
 
 Dockerfile
 ----------

python/ccf/proposal_generator.py

@@ -345,9 +345,7 @@ class ProposalGenerator:
 
 
 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
-    )
+    parser = argparse.ArgumentParser()
 
     parser.add_argument(
         "-po",