Add note on experimental pbft in docs (#861)

sphinx/source/developers/consensus.rst

@@ -0,0 +1,31 @@
+Consensus Protocols
+===================
+
+CCF supports multiple consensus protocols.
+
+The default consensus protocol for CCF is Raft.
+
+There is an option of enabling PBFT for the consensus protocol.
+
+.. warning:: Currently CCF with PBFT is in development and should not be used in a production environment.
+
+Raft Consensus Protocol
+-----------------------
+
+The Raft implementation in CCF provides Crash Fault Tolerance.
+
+For more information on the Raft protocol please see the orignial `Raft paper <https://www.usenix.org/system/files/conference/atc14/atc14-paper-ongaro.pdf>`_.
+
+
+PBFT Consensus Protocol
+-----------------------
+
+There is an option of enabling CCF with PBFT as a consensus protocol providing Byzantine Fault Tolerance.
+
+For more information on the PBFT protocol please see the original `PBFT paper <http://pmg.csail.mit.edu/papers/osdi99.pdf>`_.
+
+PBFT is still under development and should not be enabled in a production environment. Features to be completed and bugs are tracked under the `Complete ePBFT support in CCF <https://github.com/microsoft/CCF/milestone/4>`_ milestone.
+
+There is an open research question of `node identity with Byzantine nodes <https://github.com/microsoft/CCF/issues/893>`_.
+
+By default CCF runs with Raft. To enable PBFT CCF must be built with the the build switch **PBFT** set to ON. The ``--consensus pbft`` CLI argument must also be provided when starting up the nodes (see :ref:`here <operators/start_network:Starting a New Network>` for starting up a newtork and nodes).

sphinx/source/developers/index.rst

@@ -18,6 +18,7 @@ These endpoints mutate the state of a unique :ref:`developers/kv/index:Key-Value
   demo
   kv/index.rst
   ledger
+  consensus
   cryptography
   threading
   performance

sphinx/source/glossary.rst

@@ -7,7 +7,7 @@ Glossary
     `Azure Confidential Compute <https://azure.microsoft.com/en-us/solutions/confidential-compute/>`_ protects the confidentiality and integrity of your data and code while it’s processed in the public cloud.
 
   Consensus Protocols
-    The term Consensus protocol refers to either Raft or PBFT. Generic Consensus terminology will use `primary` node and `backup` node to indicate node responsibility in carrying out the protocol(s). These correspond in Raft to `leader` and `follower`.
+    The term Consensus protocol refers to either :ref:`Raft <developers/consensus:Raft Consensus Protocol>` or :ref:`PBFT <developers/consensus:PBFT Consensus Protocol>`. Generic Consensus terminology will use `primary` node and `backup` node to indicate node responsibility in carrying out the protocol(s). These correspond in Raft to `leader` and `follower`. More information about consensus protocols can be found `here <https://en.wikipedia.org/wiki/Consensus_(computer_science)>`_.
 
   Constitution
     Set of rules written as a Lua script that define how members' proposals are accepted.

sphinx/source/operators/start_network.rst

@@ -43,6 +43,7 @@ The :term:`constitution`, as defined by the initial members, should be passed vi
 The network is now in its opening state and any new nodes can join the network without being trusted by members.
 
 .. note:: Once a CCF network is started, :ref:`members can add other members and users via governance <members/open_network:Opening a Network>`.
+.. note:: If starting up the network with PBFT enabled as the consensus protocol, be sure to add the ``--consensus pbft`` CLI argument  when starting up the node. For more info on the provided consensus protocols please see :ref:`here <developers/consensus:Consensus Protocols>`
 
 Adding a New Node to the Network
 --------------------------------
@@ -70,6 +71,7 @@ If the network has not yet been opened by members (see :ref:`members/open_networ
 If the network has already been opened to users, members need to trust the joining node before it can become part of the network (see :ref:`members/common_member_operations:Trusting a New Node`).
 
 .. note:: When starting up the network or when joining an existing network, the secrets required to decrypt the ledger are sealed and written to a file so that the network can later be recovered. See :ref:`operators/recovery:Catastrophic Recovery` for more details on how to recover a crashed network.
+.. note:: If starting up the network with PBFT enabled as the consensus protocol, be sure to add the ``--consensus pbft`` CLI argument when starting up the node. For more info on the provided consensus protocols please see :ref:`here <developers/consensus:Consensus Protocols>`
 
 Opening a Network to Users
 --------------------------

sphinx/source/quickstart/build.rst

@@ -50,6 +50,7 @@ The full list of build switches can be obtained by running:
 * **SAN**: Boolean. Build unit tests with Address and Undefined behaviour sanitizers enabled. Default to OFF.
 * **TARGET**: String, one of ``sgx``, ``virtual``, or ``sgx;virtual``. Defaults to ``sgx;virtual``, which builds both "virtual" enclaves and actual SGX enclaves.
 * **VERBOSE_LOGGING**: Boolean. Enable all logging levels. Default to OFF.
+* **PBFT**: Boolean. Enables PBFT as the consensus protocol. Defaults to OFF where the Raft consensus protocol will be used.
 
 Running Tests
 -------------