This commit is contained in:
olgavrou 2019-10-11 09:45:54 +01:00 коммит произвёл Amaury Chamayou
Родитель c82ccb7dbf
Коммит aa86312e3e
12 изменённых файлов: 125 добавлений и 563 удалений

Просмотреть файл

@ -1,24 +0,0 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = ePBFT
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
livehtml:
sphinx-autobuild -b html "$(SOURCEDIR)" "$(BUILDDIR)/html" -H 0.0.0.0

Просмотреть файл

@ -1,8 +0,0 @@
breathe
sphinx
sphinx_rtd_theme
pygments-style-solarized
bottle
sphinx-autobuild
sphinxcontrib-mermaid
sphinx-jsonschema

Просмотреть файл

Просмотреть файл

Просмотреть файл

@ -1,179 +0,0 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = u"ePBFT"
copyright = u""
author = u"Microsoft Research"
# The short X.Y version
version = u""
# The full version, including alpha/beta/rc tags
release = u""
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"sphinx.ext.todo",
"sphinx.ext.mathjax",
"sphinx.ext.ifconfig",
"sphinx.ext.viewcode",
"breathe",
"sphinxcontrib.mermaid",
"sphinx.ext.autosectionlabel",
"sphinx.ext.githubpages",
"sphinx-jsonschema",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = ".rst"
# The master toctree document.
master_doc = "index"
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "solarizeddark"
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "sphinx_rtd_theme"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ["_static"]
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = "ePBFTdoc"
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, "ePBFT.tex", u"ePBFT Documentation", u"Microsoft Research", "manual")
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [(master_doc, "epbft", u"ePBFT Documentation", [author], 1)]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(
master_doc,
"ePBFT",
u"ePBFT Documentation",
author,
"ePBFT",
"One line description of project.",
"Miscellaneous",
)
]
# -- Extension configuration -------------------------------------------------
# -- Options for todo extension ----------------------------------------------
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = True
# -- Breathe configuration
# Setup the breathe extension
# breathe_projects = {"ePBFT": "../../doxygen/xml"}
# breathe_default_project = "ePBFT"

Просмотреть файл

@ -1,15 +0,0 @@
Welcome to ePBFT's documentation
================================
.. toctree::
:maxdepth: 2
:caption: Contents:
start_network
proposed_integration_layout
integration_progress
Index
=====
* :ref:`genindex`

Просмотреть файл

@ -1,331 +0,0 @@
Work In progress
================
note "28/06"
~~~~~~~~~~~~
Basic integration design
.. mermaid::
sequenceDiagram
participant Client
participant Primary Frontend
participant Primary KV
participant Primary History
participant Primary PBFT
participant Backup PBFT
participant Backup Frontend
participant Backup KV
participant Backup History
Note over Primary PBFT: + PBFT Client Proxy
Client->>Primary Frontend: JSON-RPC Request
Primary Frontend->>Primary History: add_request(RequestID, JSON-RPC Request)
Primary History->>Primary PBFT: PBFT ON_REQUEST callback
Primary PBFT->>Backup PBFT: Send JSON-RPC Request
Backup PBFT->>Backup PBFT: Record JSON-RPC Request
Primary PBFT->>Primary PBFT: execute tentative: Noop
Primary PBFT->>Backup PBFT: Send Pre-prepare
Primary PBFT-->>Primary History: _
Primary History-->>Primary Frontend: _
Primary Frontend->>Primary KV: execution
Primary KV->>Primary History: add_result(RequestID, version, tree root)
Primary History->>Primary History: add hash of write set to tree
Note right of Primary History: TODO: Pass version and tree root to PBFT
Primary History-->>Primary KV: _
Primary KV-->>Primary Frontend: _
Primary Frontend->>Primary History: add_response(Request ID, JSON-RPC Response)
Note right of Primary History: TODO: ?
Primary History-->>Primary Frontend: _
Primary Frontend-->>Client: JSON-RPC Response
Backup PBFT->>Backup PBFT: execute tentative
Note right of Backup PBFT: TODO: Dispatch to correct frontend
Backup PBFT->>Backup Frontend: JSON-RPC Request execution
Note right of Backup KV: TODO: Create new KV for PBFT?
Backup Frontend->>Backup KV: execution
Backup KV->>Backup History: add hash of write set to tree
Backup KV-->>Backup Frontend: _
Backup Frontend-->>Backup PBFT: _
Note over Backup PBFT, Backup History: TODO: Verify Merkle root
note "05/07"
~~~~~~~~~~~~
.. mermaid::
sequenceDiagram
participant Client
participant Primary Frontend
participant Primary KV
participant Primary History
participant Primary Client Proxy
participant Primary PBFT
participant Backup PBFT
participant Backup Frontend
participant Backup KV
participant Backup History
Note over Client: TODO: Handle first request
Client->>Primary Frontend: JSON-RPC Request
Primary Frontend->>Primary History: add_request(RequestID, JSON-RPC Request)
Primary History->>Primary Client Proxy: ON_REQUEST callback
Primary Client Proxy->>Primary PBFT: Handle request
Primary PBFT->>Backup PBFT: send JSON-RPC Request
Backup PBFT->>Backup PBFT: record JSON-RPC Request
Primary PBFT->>Primary PBFT: execute tentative
Primary PBFT->>Primary Frontend: process(JSON-RPC Request)
Note over Primary Frontend: Business Logic
Primary Frontend->>Primary KV: Commit TX
Primary KV->>Primary History: add_result(RequestID, version, tree root)
Primary History->>Primary History: add hash of write set to tree
Note over Primary History: TODO: Record Merkle root and version
Primary History-->>Primary KV: _
Primary KV-->>Primary Frontend: _
Note over Primary Frontend: JSON-RPC Response generated
Primary Frontend->>Primary History: add_response(Request ID, JSON-RPC Response)
Note right of Primary History: TODO: ?
Primary History-->>Primary Frontend: _
Primary Frontend-->>Primary PBFT: _
Note over Client, Primary PBFT: TODO: Only send the request once a round of PBFT has completed, hack for now
Primary PBFT->>Primary Client Proxy: Send JSON-RPC Response
Primary Client Proxy->>Client: JSON-RPC Response (via rpc_sessions)
Note over Primary PBFT: Collect a batch of requests
Primary PBFT->>Backup PBFT: send Pre-prepare (Merkle root + version)
loop Ordered requests
Backup PBFT->>Backup PBFT: execute tentative
Backup PBFT->>Backup Frontend: process(JSON-RPC Request)
Backup Frontend->>Backup KV: Commit TX
Backup KV->>Backup History: add hash of write set to tree
Backup KV-->>Backup Frontend: _
Backup Frontend-->>Backup PBFT: _
end
Backup PBFT->>Backup PBFT: Verify Merkle root
Note over Backup PBFT, Backup History: TODO: How do we signal to PBFT that the Merkle roots don't match up?
Done (some on master, some on the `pbft_integration` branch):
* Dispatching message to appropriate frontend
* CCF frontend + KV execution via PBFT's exec_command
* Integrate Client Proxy
To Do next:
* Make first transaction (genesis) work with PBFT
* Fix Client Proxy reply to client
* Pass Request ID from CCF to PBFT
* Support full round of PBFT before replying to client
* Pass the Merkle root and version in Pre-prepare message
note "19/07"
~~~~~~~~~~~~
.. mermaid::
sequenceDiagram
participant Client
participant Primary Frontend
participant Primary KV
participant Primary History
participant Primary Client Proxy
participant Primary PBFT
Client->>Primary Frontend: JSON-RPC Request
Primary Frontend->>Primary History: add_request(RequestID, JSON-RPC Request)
Primary History->>Primary Client Proxy: ON_REQUEST callback
Primary Client Proxy->>Primary PBFT: Handle request
Primary PBFT->>Backup PBFT: send JSON-RPC Request
Backup PBFT->>Backup PBFT: record JSON-RPC Request
Primary PBFT->>Primary PBFT: execute tentative
Primary PBFT->>Primary Frontend: process(JSON-RPC Request)
Note over Primary Frontend: Business Logic
Primary Frontend->>Primary KV: Commit TX
Primary KV->>Primary History: add_result(RequestID, version, tree root)
Primary History->>Primary History: add hash of write set to tree
Note over Primary History: TODO: Record Merkle root and version
Primary History-->>Primary KV: _
Primary KV-->>Primary Frontend: _
Note over Primary Frontend: JSON-RPC Response generated
Primary Frontend->>Primary History: add_response(Request ID, JSON-RPC Response)
Note right of Primary History: TODO: ?
Primary History-->>Primary Frontend: _
Primary Frontend-->>Primary PBFT: _
Primary PBFT->>Primary Client Proxy: Send JSON-RPC Response
Primary Client Proxy->>Client: JSON-RPC Response (via rpc_sessions)
Note over Primary PBFT: Collect a batch of requests
Done (all on `master`):
* Integration for f = 0:
* Dispatching message to appropriate frontend
* CCF frontend + KV execution via PBFT's exec_command
* Integrate Client Proxy
* Pass Request ID from CCF to PBFT
* Support full round of PBFT before replying to client
To Do next:
* Unify consensus interface
* Pass the Merkle root and version in Pre-prepare message
* Support for f = 1:
* Startup on all nodes
* Dynamic node configuration
09/08
~~~~~
`Primary`
.. mermaid::
sequenceDiagram
participant Client
participant LedgerEnclave(RB)
participant Frontend
participant KV
participant History
participant Client Proxy
participant Replica
participant Backup Replica
Client->>Frontend: JSON-RPC Request
Frontend->>History: add_request(RequestID, actor, caller_id, JSON-RPC Request)
History->>Client Proxy: ON_REQUEST callback(RequestID, actor, caller_id, JSON-RPC Request)
Client Proxy->>Client Proxy: Wrap JSON-RPC Request in PBFT command
Client Proxy->>Replica: send(PBFT command, All_replicas)
Replica->>Backup Replica: send(PBFT command)
Replica-->>Client Proxy:_
Client Proxy->>Replica: handle(Request [PBFT command])
Replica->>Replica: execute_tentative: starts
Replica->>Replica: exec_command: starts
Replica->>Frontend: process_pbft(PBFT command [Wrapped JSON-RPC Request])
Frontend->>History: register ON_RESULT callback
Frontend->>Frontend: process_json()
Frontend->>KV: COMMIT TX
KV->>History: add_result(RequestID, version, merkle_root)
History->>History: ON_RESULT callback: populated merkle_root reference that was passed by process_pbft when callback was registered
History-->>KV:_
KV-->>Frontend:_
Frontend->>Replica: return from process_pbft(): ProcessPbftResult{result, merkle_root}
Replica->>Replica: cp merkle_root into exec_command's Byz_info
Replica->>Replica: exec_command: returns
Replica->>Replica: execute_tentative: returns
Replica->>Replica: put merkle_root in pre prepare msg
Replica->>Replica: write pre prepare and requests to ledger
Replica->>LedgerEnclave(RB): ON_APPEND_LEDGER_ENTRY Callback: put_entry()
LedgerEnclave(RB)-->>Replica:_
Replica->>Replica Backup: send(pre prepare, All_replicas)
Replica-->>Client Proxy: return from handle()
Client Proxy-->>History: return from ON_REQUEST callback()
History-->>Frontend: return from add_request()
Frontend-->>Client: return from process()
`Backup/Replica`
[handling a prepare is the same if the Replica is Primary]
.. mermaid::
sequenceDiagram
participant Nodestate
participant LedgerEnclave(RB)
participant Frontend
participant Client Proxy
participant Replica
participant All Other Replicas
Nodestate->>Replica: receive_message()
Replica-->>Replica: receive_process_one_msg()
Replica->>Replica: handle(Request)
Replica->>Replica: store request
Replica->>Replica: forward request to primary
Nodestate->>Replica: receive_message()
Replica-->>Replica: receive_process_one_msg()
Replica->>Replica: handle(Pre_prepare)
Replica->>Replica: write pre prepare to ledger [as shown for primary]
Replica-->>LedgerEnclave(RB): [as shown above]
Replica->>Replica: execute_tentative [as shown for primary]
Replica->>Replica: exec_command [as shown for primary]
Replica->>Replica: check that merkle_root matches
Note over Replica: check that merkle_root returned from exec_command (populated by history after exec_command calls out to frontend) matches the one from the pre_prepare msg
Replica->>All Other Replicas: [if not ok just return] if ok send(Prepare with pp's digest, All_replicas)
Nodestate->>Replica: receive_message()
Replica-->>Replica: receive_process_one_msg()
Replica->>Replica: handle(Prepare)
Replica->>Replica: [if prepare cert is complete] write prepare cert info to ledger
Note over Replica: writing prepare includes writing a header [seqno, num of pp proofs] and writing the proofs [all pp proofs for each prepare that I have in this certifcate]
Replica-->>LedgerEnclave(RB): write header
Replica-->>LedgerEnclave(RB): write proofs
Replica->> All Other Replicas: [if prepare cert complete] send(commit, All_replicas)
Done (all on `master`):
* Integration for f = 0:
* Dispatching message to appropriate frontend
* CCF frontend + KV execution via PBFT's exec_command
* Integrate Client Proxy
* Pass Request ID from CCF to PBFT
* Support full round of PBFT before replying to client
* Pass the Merkle root and version in Pre-prepare message
To Do next:
* Unify consensus interface
* Support for f = 1:
* Startup on all nodes
* Dynamic node configuration

Просмотреть файл

@ -0,0 +1,11 @@
Design
================
Design diagrams that show the work in progress on the ePBFT/CCF integration.
.. note:: Operators, users, members, and application developers should not be affected by this section and can safely ignore it.
.. toctree::
start_network
proposed_integration_layout
integration_progress

Просмотреть файл

@ -0,0 +1,111 @@
Integration progress diagram
============================
`Primary`
.. mermaid::
sequenceDiagram
participant Client
participant LedgerEnclave(RB)
participant Frontend
participant KV
participant History
participant Client Proxy
participant Replica
participant Backup Replica
Client->>Frontend: JSON-RPC Request
Frontend->>History: add_request(RequestID, actor, caller_id, JSON-RPC Request)
History->>Client Proxy: ON_REQUEST callback(RequestID, actor, caller_id, JSON-RPC Request)
Client Proxy->>Client Proxy: Wrap JSON-RPC Request in PBFT command
Client Proxy->>Replica: send(PBFT command, All_replicas)
Replica->>Backup Replica: send(PBFT command)
Replica-->>Client Proxy:_
Client Proxy->>Replica: handle(Request [PBFT command])
Replica->>Replica: execute_tentative: starts
Replica->>Replica: exec_command: starts
Replica->>Frontend: process_pbft(PBFT command [Wrapped JSON-RPC Request])
Frontend->>History: register ON_RESULT callback
Frontend->>Frontend: process_json()
Frontend->>KV: COMMIT TX
KV->>History: add_result(RequestID, version, merkle_root)
History->>History: ON_RESULT callback: populated merkle_root reference that was passed by process_pbft when callback was registered
History-->>KV:_
KV-->>Frontend:_
Frontend->>Replica: return from process_pbft(): ProcessPbftResult{result, merkle_root}
Replica->>Replica: cp merkle_root into exec_command's Byz_info
Replica->>Replica: exec_command: returns
Replica->>Replica: execute_tentative: returns
Replica->>Replica: put merkle_root in pre prepare msg
Replica->>Replica: write pre prepare and requests to ledger
Replica->>LedgerEnclave(RB): ON_APPEND_LEDGER_ENTRY Callback: put_entry()
LedgerEnclave(RB)-->>Replica:_
Replica->>Replica Backup: send(pre prepare, All_replicas)
Replica-->>Client Proxy: return from handle()
Client Proxy-->>History: return from ON_REQUEST callback()
History-->>Frontend: return from add_request()
Frontend-->>Client: return from process()
`Backup/Replica`
[handling a prepare is the same if the Replica is Primary]
.. mermaid::
sequenceDiagram
participant Nodestate
participant LedgerEnclave(RB)
participant Frontend
participant Client Proxy
participant Replica
participant All Other Replicas
Nodestate->>Replica: receive_message()
Replica-->>Replica: receive_process_one_msg()
Replica->>Replica: handle(Request)
Replica->>Replica: store request
Replica->>Replica: forward request to primary
Nodestate->>Replica: receive_message()
Replica-->>Replica: receive_process_one_msg()
Replica->>Replica: handle(Pre_prepare)
Replica->>Replica: write pre prepare to ledger [as shown for primary]
Replica-->>LedgerEnclave(RB): [as shown above]
Replica->>Replica: execute_tentative [as shown for primary]
Replica->>Replica: exec_command [as shown for primary]
Replica->>Replica: check that merkle_root matches
Note over Replica: check that merkle_root returned from exec_command (populated by history after exec_command calls out to frontend) matches the one from the pre_prepare msg
Replica->>All Other Replicas: [if not ok just return] if ok send(Prepare with pp's digest, All_replicas)
Nodestate->>Replica: receive_message()
Replica-->>Replica: receive_process_one_msg()
Replica->>Replica: handle(Prepare)
Replica->>Replica: [if prepare cert is complete] write prepare cert info to ledger
Note over Replica: writing prepare includes writing a header [seqno, num of pp proofs] and writing the proofs [all pp proofs for each prepare that I have in this certifcate]
Replica-->>LedgerEnclave(RB): write header
Replica-->>LedgerEnclave(RB): write proofs
Replica->> All Other Replicas: [if prepare cert complete] send(commit, All_replicas)

Просмотреть файл

@ -1,9 +1,7 @@
Proposed integration layout
===========================
Primary
~~~~~~~
`Primary`
.. mermaid::
@ -45,8 +43,7 @@ Primary
Replica
~~~~~~~
`Replica`
.. mermaid::

Просмотреть файл

@ -2,7 +2,6 @@ Starting up a network
=====================
Proposed diagram for creating a new network when running with CCF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. mermaid::

Просмотреть файл

@ -18,6 +18,7 @@ Welcome to CCF's documentation
rpc_api
api
glossary
design/index.rst
Index
=====