0xDeCA10B/simulation

Sharing Updatable Models (SUM) on Blockchain Simulation

(formerly Decentralized & Collaborative AI on Blockchain Simulation)

Tools to run simulations for AI models in smart contracts.

Examples

Even when a bad actor submits incorrect data, an honest contributor profits while the model's accuracy remains stable.

In the above example, a Perceptron was trained on the IMDB reviews dataset for sentiment classification.

Here's a more detailed example:

For this simulation, again a Perceptron was trained on the IMDB reviews dataset for sentiment classification. The model was initially trained on 2000 of the 25000 training data samples. The model has 1000 binary features which are the presence of the 1000 most frequent words in the dataset. The graph below shows the results of a simulation where for simplicity, we show just one honest contributor and one malicious contributor but these contributors effectively represent many contributors submitting the remaining 92% of the training data over time. In this simulation, we use the Deposit, Refund, and Take (DRT) incentive mechanism where contributors have 1 day to claim a refund. Any contributor can take the remaining deposit from a contribution after 9 days. "Bad Agent" is willing to spend about twice as much on deposits than an honest contributor, "Good Agent". The adversary is only submitting data about one sixth as often. Despite the malicious efforts, the accuracy can still be maintained and the honest contributors profit.

Setup

This section explains how to set up locally, alternatively, you can skip ahead and use a Docker image. Run:

conda create --channel conda-forge --name decai-simulation 'python=3.8' bokeh mkl mkl-service numpy pandas phantomjs scikit-learn scipy tensorflow
conda activate decai-simulation
pip install -e .

Docker Setup

1. Clone this repo.

2. Navigate into the folder containing this README.md file:

cd 0xDeCA10B/simulation

3. Using a Docker Image

docker build -t decai-simulation .

docker run --rm -it -p 5006:5006 -v ${PWD}:/root/workspace/0xDeCA10B/simulation --name decai-simulation decai-simulation bash

4. You have completed the setup steps. Next, refer to the Running Simulations steps lower down on this page.

Running Simulations

Run:

bokeh serve decai/simulation/simulate_imdb_perceptron.py

Then open the browser to the address the above command tells you. It should be something like: http://localhost:5006/simulate_imdb_perceptron.

Customizing Simulations

To try out your own models or incentive mechanisms, you'll need to implement the interfaces. You can proceed by just copying the examples. Here are the details if you need them:

Suppose you want to use a neural network for the classifier:

1. Implement the Classifier interface in a class NeuralNetworkClassifier. The easiest way is to copy an existing classifier like the Perceptron.
2. Create a Module called NeuralNetworkModule which binds Classifier to your new class just like in PerceptronModule.

Setting up a custom incentive mechanism is similar:

1. Implement IncentiveMechanism. You can use Stakeable as an example.
2. Bind your implementation in a module.

Now set up the main entry point to run the simulation: copy (decai/simulation/simulate_imdb_perceptron.py) to a new file, e.g. decai/simulation/simulate_imdb_neural_network.py.

In simulate_imdb_neural_network.py, you can set up the agents that will act in the simulation. Then set the modules you created. So instead of PerceptronModule put NeuralNetworkModule.

Run bokeh serve decai/simulation/simulate_imdb_neural_network.py and open your browse to the displayed URL to try it out.

Testing

Setup the testing environment:

pip install -e .[test]

Run tests: (it might take a few minutes the first time because some spaCy models might need to be downloaded)

pytest