Widgets dev harness (#104)

* Separate dev harnesses for CompatibilityAnalsysis and ModelComparison

* Fix some issues and bump Axios version

* Update README instructions and remove commented code

* Remove unused files

* Remove unnecessary arguments from ModelComparison widget instantiation

* Remove unused arguments from ModelComparison widget instantiation within the dev harness code

* Pin numpy to 1.19.3

* Fix typo in README

* Add docstrings for the ModelComparison widget and ComparisonManagement class

* Add documentation for Model Comparison widget

.flaskenv

@@ -1,3 +0,0 @@
-FLASK_APP=development/app.py
-FLASK_ENV=development
-FLASK_RUN_PORT=5000

.gitignore

@@ -105,6 +105,9 @@ venv.bak/
 # mypy
 .mypy_cache/
 
+# Pytorch models
+*.state
+
 development/sweeps-mnist/*.json
 development/sweeps-mnist/*state
 

README.md

@@ -48,18 +48,27 @@ To run tests, make sure that you are in the project root folder and do:
 3. `npm install`
 4. `npm run test`
 
-# Development environment
+# Development Environment
 
-To run the widget and APIs in a development environment:
-- Open a new terminal, then from within the root folder do `npm start`. This will host the widget with `webpack-dev-server`.
-  - To customize the host IP, run `webpack-dev-server --hot --mode development --host <ip>` instead of `npm start`.
-  - `npm start` uses `0.0.0.0` which makes the server accessible externally.
-- Open a new terminal, then from within the root folder do `flask run`. This will start the Flask server for the APIs used by the widget.
+This is provided as a convenience tool to developers, in order to allow development of the widget proceed outside of a Jupyter notebook environment.
 
-The widget can be loaded in the web browser at `localhost:3000`. It will be loaded independently from a Jupyter notebook. The APIs will be hosted at `localhost:5000`. If you are running the server within a VM, you might need to use your machine's local IP instead of `localhost`.
+The widget can be loaded in the web browser at `localhost:3000` or `<your-ip>:3000`. It will be loaded independently from a Jupyter notebook. The APIs will be hosted at `localhost:5000` or `<your-ip>:5000`.
 
 Changes to the CSS or TypeScript code will be hot loaded automatically in the browser. Flask will run in debug mode and automatically restart whenever the Python code is changed.
 
+## Compatibility Analysis Widget
+
+- Open a new terminal and within the project root folder do `FLASK_ENV=development FLASK_APP=development/compatibility-analysis/app.py flask run --host 0.0.0.0 --port 5000` on Linux or `set FLASK_ENV=development && set FLASK_APP=development\compatibility-analysis\app.py && flask run --host 0.0.0.0 --port 5000` on Windows. This will start the Flask server for the APIs used by the widget.
+- Open a new terminal, then from within the project root folder do `npm run start-compatibility-analysis`
+- Open your browser and point it to `http://<your-ip-address>:3000`
+
+## Model Comparison Widget
+
+- Open a new terminal and within the project root folder do `FLASK_ENV=development FLASK_APP=development/model-comparison/app.py flask run --host 0.0.0.0 --port 5000` on Linux or `set FLASK_ENV=development && set FLASK_APP=development\model-comparison\app.py && flask run --host 0.0.0.0 --port 5000` on Windows. This will start the Flask server for the APIs used by the widget.
+- Open a new terminal, then from within the project root folder do `npm run start-model-comparison`.
+- Open your browser and point it to `http://<your-ip-address>:3000`
+
+
 # Contributing
 
 Check [CONTRIBUTING](CONTRIBUTING.md) page.

backwardcompatibilityml/comparison_management.py

@@ -1,45 +1,31 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 
-import os
-import json
-import threading
 import io
 import numpy as np
-import mlflow
 from flask import send_file
 from PIL import Image
-from queue import Queue
-from backwardcompatibilityml.helpers import training
-from backwardcompatibilityml.metrics import model_accuracy
 
 
 class ComparisonManager(object):
     """
-    The ComparisonManager class is used to manage an experiment that performs
-    model comparison.
+    The ComparisonManager class is used to field any REST requests by the ModelComparison
+    widget UI components from within the Jupyter notebook.
+
+    Args:
+        training_set: The list of training samples as (batch_ids, input, target).
+        dataset: The list of dataset samples as (batch_ids, input, target).
+        get_instance_image_by_id: A function that returns an image representation of the data corresponding to the instance id, in PNG format. It should be a function of the form:
+                get_instance_image_by_id(instance_id)
+                    instance_id: An integer instance id
+
+            And should return a PNG image.
     """
 
-    def __init__(self, folder_name, h1, h2, dataset,
-                 performance_metric=model_accuracy,
-                 get_instance_image_by_id=None,
-                 get_instance_metadata=None,
-                 device="cpu",
-                 use_ml_flow=False,
-                 ml_flow_run_name="model_comparison"):
-        self.folder_name = folder_name
-        self.h1 = h1
-        self.h2 = h2
+    def __init__(self, dataset,
+                 get_instance_image_by_id=None):
         self.dataset = dataset
-        self.performance_metric = performance_metric
         self.get_instance_image_by_id = get_instance_image_by_id
-        self.get_instance_metadata = get_instance_metadata
-        self.device = device
-        self.use_ml_flow = use_ml_flow
-        self.ml_flow_run_name = ml_flow_run_name
-        self.last_sweep_status = 0.0
-        self.percent_complete_queue = Queue()
-        self.sweep_thread = None
 
     def get_instance_image(self, instance_id):
         get_instance_image_by_id = self.get_instance_image_by_id

backwardcompatibilityml/widgets/model_comparison/model_comparison.py

@@ -112,32 +112,64 @@ def init_app_routes(app, comparison_manager):
 class ModelComparison(object):
     """
     Model Comparison widget
-    ...
+
+    The ModelComparison class is an interactive widget intended for use
+    within a Jupyter Notebook. It provides an interactive UI for the user
+    that allows the user to:
+
+        1. Compare two models h1 and h2 on a dataset with regard to compatibility.
+        2. The comparison is run by comparing the set of classification errors that h1 and h2
+           make on the dataset.
+        3. The Venn Diagram plot within the widget provides a breakdown of the overlap between
+           the sets of classification errors made by h1 and h2.
+        4. The bar chart indicates the number of errors made by h2 that are not made by h1
+           on a per class basis.
+        5. The error instances table, provides an exploratory view to allow the user to
+           explore the instances which h1 and h2 have misclassified. This table is linked
+           to the Venn Diagram and Bar Charts, so that the user may filter the error
+           instances displayed in the table by clicking on regions of those components.
+
+    Args:
+        h1: The reference model being used.
+        h2: The model that we want to compare against model h1.
+        dataset: The list of dataset samples as (batch_ids, input, target). This data needs to be batched.
+        performance_metric: A function to evaluate model performance. The function is expected to have the following signature:
+                metric(model, dataset, device)
+                    model: The model being evaluated
+                    dataset: The dataset as a list of (input, target) pairs
+                    device: The device Pytorch is using for training - "cpu" or "cuda"
+
+            If unspecified, then accuracy is used.
+        port: An integer value to indicate the port to which the Flask service
+            should bind.
+        get_instance_image_by_id: A function that returns an image representation of the data corresponding to the instance id, in PNG format. It should be a function of the form:
+                get_instance_image_by_id(instance_id)
+                    instance_id: An integer instance id
+            And should return a PNG image.
+        get_instance_metadata: A function that returns a text string representation of some metadata corresponding to the instance id. It should be a function of the form:
+                get_instance_metadata(instance_id)
+                    instance_id: An integer instance id
+            And should return a string.
+        device: A string with values either "cpu" or "cuda" to indicate the
+            device that Pytorch is performing training on. By default this
+            value is "cpu". But in case your models reside on the GPU, make sure
+            to set this to "cuda". This makes sure that the input and target
+            tensors are transferred to the GPU during training.
     """
 
-    def __init__(self, folder_name, number_of_epochs, h1, h2, dataset,
+    def __init__(self, h1, h2, dataset,
                  performance_metric=model_accuracy,
                  port=None,
                  get_instance_image_by_id=None,
                  get_instance_metadata=None,
-                 device="cpu",
-                 use_ml_flow=False,
-                 ml_flow_run_name="model_comparison"):
+                 device="cpu"):
 
         if get_instance_metadata is None:
             get_instance_metadata = default_get_instance_metadata
 
         self.comparison_manager = ComparisonManager(
-            folder_name,
-            h1,
-            h2,
             dataset,
-            performance_metric=performance_metric,
-            get_instance_image_by_id=get_instance_image_by_id,
-            get_instance_metadata=get_instance_metadata,
-            device=device,
-            use_ml_flow=use_ml_flow,
-            ml_flow_run_name=ml_flow_run_name)
+            get_instance_image_by_id=get_instance_image_by_id)
 
         self.flask_service = FlaskHelper(ip="0.0.0.0", port=port)
         app_has_routes = False

compatibilityanalysis.webpack.config.js

@@ -23,7 +23,7 @@ module.exports = {
       filename: 'widget.css'
     }),
     isDevelopment && new HtmlWebpackPlugin({
-      template: './development/index.html'
+      template: './development/compatibility-analysis/index.html'
     }),
     isDevelopment && new ReactRefreshPlugin()
   ].filter(Boolean),

development/compatibility-analysis/app.py

@@ -103,7 +103,7 @@ def breast_cancer_sweep():
 
 
 def mnist_sweep():
-    sweeps_folder = "development/sweeps-mnist"
+    sweeps_folder = "development/compatibility-analysis/sweeps-mnist"
     n_epochs = 3
     n_samples = 5000
     batch_size_train = 64

development/model-comparison/app.py

@@ -0,0 +1,197 @@
+# Copyright (c) Microsoft Corporation
+# Licensed under the MIT License.
+
+import os
+import copy
+import io
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.optim as optim
+import torch.nn.functional as F
+import torchvision
+import random
+from backwardcompatibilityml import loss as bcloss
+from backwardcompatibilityml.helpers import training
+from backwardcompatibilityml.helpers.models import LogisticRegression, MLPClassifier
+from backwardcompatibilityml.widgets import ModelComparison
+from flask import send_file
+from PIL import Image
+from rai_core_flask.flask_helper import FlaskHelper
+
+use_ml_flow = True
+ml_flow_run_name = "dev_app_sweep"
+
+
+def breast_cancer_sweep():
+    folder_name = "tests/sweeps"
+    number_of_epochs = 10
+    batch_size_train = 70
+    batch_size_test = 139
+    learning_rate = 0.01
+    momentum = 0.5
+
+    dataset_file = open("tests/datasets/breast-cancer-wisconsin.data", "r")
+    raw_data = dataset_file.read()
+    dataset_file.close()
+    data = list(map(lambda l: l.split(",")[1:], filter(lambda l: len(l) > 0, map(lambda l: l.strip(), raw_data.split("\n")))))
+    for i in range(len(data)):
+        row = data[i]
+        row = list(map(lambda e: 0 if (e == "?") else int(e), row))
+        if row[9] == 2:
+            row[9] = 0
+        elif row[9] == 4:
+            row[9] = 1
+        data[i] = row
+
+    data = list(map(lambda r: (torch.tensor(r[:9], dtype=torch.float32), torch.tensor(r[9])), data))
+
+    random.shuffle(data)
+
+    training_set = data[:560]
+    testing_set = data[560:]
+
+    training_set_torch = []
+    prev = 0
+    for i in range((batch_size_train - 1), len(training_set), batch_size_train):
+        training_data = list(map(lambda r: r[0], training_set[prev:(i + 1)]))
+        training_labels = list(map(lambda r: r[1], training_set[prev:(i + 1)]))
+        training_ids = list(range(prev, i+1))
+        prev = i
+        training_set_torch.append([training_ids, torch.stack(training_data, dim=0), torch.stack(training_labels, dim=0)])
+
+    testing_set_torch = []
+    prev = 0
+    for i in range((batch_size_test - 1), len(testing_set), batch_size_test):
+        testing_data = list(map(lambda r: r[0], testing_set[prev:(i + 1)]))
+        testing_labels = list(map(lambda r: r[1], testing_set[prev:(i + 1)]))
+        testing_ids = list(range(prev, i+1))
+        prev = i
+        testing_set_torch.append([testing_ids, torch.stack(testing_data, dim=0), torch.stack(testing_labels, dim=0)])
+
+    training_set = training_set_torch
+    partial_training_set = training_set[:int(0.5 * (len(training_set)))]
+    testing_set = testing_set_torch
+
+    h1 = LogisticRegression(9, 2)
+    optimizer = optim.SGD(h1.parameters(), lr=learning_rate, momentum=momentum)
+    train_counter, test_counter, train_losses, test_losses = training.train(
+        number_of_epochs, h1, optimizer, F.nll_loss, partial_training_set, testing_set,
+        batch_size_train, batch_size_test)
+    h1.eval()
+
+    with torch.no_grad():
+        _, _, output = h1(testing_set[0][1])
+
+    h2 = MLPClassifier(9, 2)
+
+    model_comparison = ModelComparison(h1, h2, training_set,
+                                       use_ml_flow=True, device="cuda")
+
+
+def mnist_sweep():
+    sweeps_folder = "development/model-comparison/sweeps-mnist"
+    n_epochs = 3
+    n_samples = 5000
+    batch_size_train = 64
+    batch_size_test = 1000
+    learning_rate = 0.01
+    momentum = 0.5
+
+    random_seed = 1
+    torch.backends.cudnn.enabled = False
+    torch.manual_seed(random_seed)
+    transform = torchvision.transforms.Compose([
+        torchvision.transforms.ToTensor(),
+        torchvision.transforms.Normalize((0.1307,), (0.3081,))])
+    download = not os.path.exists("examples/datasets/MNIST")
+    data_set = torchvision.datasets.MNIST('examples/datasets/', train=True, download=download,
+                                          transform=transform)
+    data_loader = list(torch.utils.data.DataLoader(data_set, shuffle=True))
+
+    instance_ids = list(range(len(data_loader)))
+    dataset = []
+    for (instance_id, (data_instance, data_label)) in zip(instance_ids, data_loader):
+        dataset.append([instance_id, data_instance.view(1, 28, 28), data_label.view(1)])
+
+    training_set = dataset[:int(0.8*n_samples)]
+    testing_set = dataset[int(0.8*n_samples):]
+
+    train_loader = []
+    prev = 0
+    for i in range((batch_size_train - 1), len(training_set), batch_size_train):
+        batch_ids = list(map(lambda r: r[0], training_set[prev:i]))
+        training_data = list(map(lambda r: r[1], training_set[prev:i]))
+        training_labels = list(map(lambda r: r[2], training_set[prev:i]))
+        prev = i
+        train_loader.append([batch_ids, torch.stack(training_data, dim=0), torch.stack(training_labels, dim=0).view(len(training_labels))])
+
+    test_loader = []
+    prev = 0
+    for i in range((batch_size_test - 1), len(testing_set), batch_size_test):
+        batch_ids = list(map(lambda r: r[0], testing_set[prev:i]))
+        testing_data = list(map(lambda r: r[1], testing_set[prev:i]))
+        testing_labels = list(map(lambda r: r[2], testing_set[prev:i]))
+        prev = i
+        test_loader.append([batch_ids, torch.stack(testing_data, dim=0), torch.stack(testing_labels, dim=0).view(len(testing_labels))])
+
+    class Net(nn.Module):
+        def __init__(self):
+            super(Net, self).__init__()
+            self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
+            self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
+            self.conv2_drop = nn.Dropout2d()
+            self.fc1 = nn.Linear(320, 50)
+            self.fc2 = nn.Linear(50, 10)
+
+        def forward(self, x):
+            x = F.relu(F.max_pool2d(self.conv1(x), 2))
+            x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
+            x = x.view(-1, 320)
+            x = F.relu(self.fc1(x))
+            x = F.dropout(x, training=self.training)
+            x = self.fc2(x)
+            return x, F.softmax(x, dim=1), F.log_softmax(x, dim=1)
+
+    h1 = Net().cuda()
+    state_file = f"{sweeps_folder}/h1-model.state"
+    if (os.path.exists(state_file)):
+        h1.load_state_dict(torch.load(state_file))
+        print("Loaded state dict")
+    else:
+        optimizer = optim.SGD(h1.parameters(), lr=learning_rate, momentum=momentum)
+        print("Training model")
+        train_counter, test_counter, train_losses, test_losses = training.train(
+            n_epochs, h1, optimizer, F.nll_loss, train_loader, test_loader,
+            batch_size_train, batch_size_test, device="cuda")
+        torch.save(h1.state_dict(), state_file)
+        print("Saved state dict")
+
+    h2 = Net().cuda()
+
+    def unnormalize(img):
+        img = img / 2 + 0.5
+        return img
+
+    def get_instance_image(instance_id):
+        img_bytes = io.BytesIO()
+        data = np.reshape(
+            np.uint8(np.transpose((unnormalize(dataset[instance_id][1])), (1, 2, 0)).numpy() * 255),
+            (28, 28))
+        img = Image.fromarray(data)
+        img.save(img_bytes, format="PNG")
+        img_bytes.seek(0)
+        return send_file(img_bytes, mimetype='image/png')
+
+    def get_instance_label(instance_id):
+        label = dataset[instance_id][2].item()
+        return f'{label}'
+
+    model_comparison = ModelComparison(h1, h2, train_loader,
+                                       get_instance_image_by_id=get_instance_image,
+                                       use_ml_flow=True, device="cuda")
+
+
+mnist_sweep()
+app = FlaskHelper.app
+app.logger.info('initialization complete')

development/model-comparison/css/bootstrap-tour.min.css

@@ -0,0 +1,19 @@
+/* ===========================================================
+# bootstrap-tour - v0.9.0
+# http://bootstraptour.com
+# ==============================================================
+# Copyright 2012-2013 Ulrich Sossou
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+*/
+.tour-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;z-index:1100;background-color:#000;opacity:.8}.tour-step-backdrop{position:relative;z-index:1101;background:inherit}.tour-step-background{position:absolute;z-index:1100;background:inherit;border-radius:6px}.popover[class*=tour-]{z-index:1100}.popover[class*=tour-] .popover-navigation{padding:9px 14px}.popover[class*=tour-] .popover-navigation [data-role=end]{float:right}.popover[class*=tour-] .popover-navigation [data-role=prev],.popover[class*=tour-] .popover-navigation [data-role=next],.popover[class*=tour-] .popover-navigation [data-role=end]{cursor:pointer}.popover[class*=tour-] .popover-navigation [data-role=prev].disabled,.popover[class*=tour-] .popover-navigation [data-role=next].disabled,.popover[class*=tour-] .popover-navigation [data-role=end].disabled{cursor:default}.popover[class*=tour-].orphan{position:fixed;margin-top:0}.popover[class*=tour-].orphan .arrow{display:none}

development/model-comparison/css/codemirror.css

@@ -0,0 +1,349 @@
+/* BASICS */
+
+.CodeMirror {
+  /* Set height, width, borders, and global font properties here */
+  font-family: monospace;
+  height: 300px;
+  color: black;
+  direction: ltr;
+}
+
+/* PADDING */
+
+.CodeMirror-lines {
+  padding: 4px 0; /* Vertical padding around content */
+}
+.CodeMirror pre.CodeMirror-line,
+.CodeMirror pre.CodeMirror-line-like {
+  padding: 0 4px; /* Horizontal padding of content */
+}
+
+.CodeMirror-scrollbar-filler, .CodeMirror-gutter-filler {
+  background-color: white; /* The little square between H and V scrollbars */
+}
+
+/* GUTTER */
+
+.CodeMirror-gutters {
+  border-right: 1px solid #ddd;
+  background-color: #f7f7f7;
+  white-space: nowrap;
+}
+.CodeMirror-linenumbers {}
+.CodeMirror-linenumber {
+  padding: 0 3px 0 5px;
+  min-width: 20px;
+  text-align: right;
+  color: #999;
+  white-space: nowrap;
+}
+
+.CodeMirror-guttermarker { color: black; }
+.CodeMirror-guttermarker-subtle { color: #999; }
+
+/* CURSOR */
+
+.CodeMirror-cursor {
+  border-left: 1px solid black;
+  border-right: none;
+  width: 0;
+}
+/* Shown when moving in bi-directional text */
+.CodeMirror div.CodeMirror-secondarycursor {
+  border-left: 1px solid silver;
+}
+.cm-fat-cursor .CodeMirror-cursor {
+  width: auto;
+  border: 0 !important;
+  background: #7e7;
+}
+.cm-fat-cursor div.CodeMirror-cursors {
+  z-index: 1;
+}
+.cm-fat-cursor-mark {
+  background-color: rgba(20, 255, 20, 0.5);
+  -webkit-animation: blink 1.06s steps(1) infinite;
+  -moz-animation: blink 1.06s steps(1) infinite;
+  animation: blink 1.06s steps(1) infinite;
+}
+.cm-animate-fat-cursor {
+  width: auto;
+  border: 0;
+  -webkit-animation: blink 1.06s steps(1) infinite;
+  -moz-animation: blink 1.06s steps(1) infinite;
+  animation: blink 1.06s steps(1) infinite;
+  background-color: #7e7;
+}
+@-moz-keyframes blink {
+  0% {}
+  50% { background-color: transparent; }
+  100% {}
+}
+@-webkit-keyframes blink {
+  0% {}
+  50% { background-color: transparent; }
+  100% {}
+}
+@keyframes blink {
+  0% {}
+  50% { background-color: transparent; }
+  100% {}
+}
+
+/* Can style cursor different in overwrite (non-insert) mode */
+.CodeMirror-overwrite .CodeMirror-cursor {}
+
+.cm-tab { display: inline-block; text-decoration: inherit; }
+
+.CodeMirror-rulers {
+  position: absolute;
+  left: 0; right: 0; top: -50px; bottom: 0;
+  overflow: hidden;
+}
+.CodeMirror-ruler {
+  border-left: 1px solid #ccc;
+  top: 0; bottom: 0;
+  position: absolute;
+}
+
+/* DEFAULT THEME */
+
+.cm-s-default .cm-header {color: blue;}
+.cm-s-default .cm-quote {color: #090;}
+.cm-negative {color: #d44;}
+.cm-positive {color: #292;}
+.cm-header, .cm-strong {font-weight: bold;}
+.cm-em {font-style: italic;}
+.cm-link {text-decoration: underline;}
+.cm-strikethrough {text-decoration: line-through;}
+
+.cm-s-default .cm-keyword {color: #708;}
+.cm-s-default .cm-atom {color: #219;}
+.cm-s-default .cm-number {color: #164;}
+.cm-s-default .cm-def {color: #00f;}
+.cm-s-default .cm-variable,
+.cm-s-default .cm-punctuation,
+.cm-s-default .cm-property,
+.cm-s-default .cm-operator {}
+.cm-s-default .cm-variable-2 {color: #05a;}
+.cm-s-default .cm-variable-3, .cm-s-default .cm-type {color: #085;}
+.cm-s-default .cm-comment {color: #a50;}
+.cm-s-default .cm-string {color: #a11;}
+.cm-s-default .cm-string-2 {color: #f50;}
+.cm-s-default .cm-meta {color: #555;}
+.cm-s-default .cm-qualifier {color: #555;}
+.cm-s-default .cm-builtin {color: #30a;}
+.cm-s-default .cm-bracket {color: #997;}
+.cm-s-default .cm-tag {color: #170;}
+.cm-s-default .cm-attribute {color: #00c;}
+.cm-s-default .cm-hr {color: #999;}
+.cm-s-default .cm-link {color: #00c;}
+
+.cm-s-default .cm-error {color: #f00;}
+.cm-invalidchar {color: #f00;}
+
+.CodeMirror-composing { border-bottom: 2px solid; }
+
+/* Default styles for common addons */
+
+div.CodeMirror span.CodeMirror-matchingbracket {color: #0b0;}
+div.CodeMirror span.CodeMirror-nonmatchingbracket {color: #a22;}
+.CodeMirror-matchingtag { background: rgba(255, 150, 0, .3); }
+.CodeMirror-activeline-background {background: #e8f2ff;}
+
+/* STOP */
+
+/* The rest of this file contains styles related to the mechanics of
+   the editor. You probably shouldn't touch them. */
+
+.CodeMirror {
+  position: relative;
+  overflow: hidden;
+  background: white;
+}
+
+.CodeMirror-scroll {
+  overflow: scroll !important; /* Things will break if this is overridden */
+  /* 50px is the magic margin used to hide the element's real scrollbars */
+  /* See overflow: hidden in .CodeMirror */
+  margin-bottom: -50px; margin-right: -50px;
+  padding-bottom: 50px;
+  height: 100%;
+  outline: none; /* Prevent dragging from highlighting the element */
+  position: relative;
+}
+.CodeMirror-sizer {
+  position: relative;
+  border-right: 50px solid transparent;
+}
+
+/* The fake, visible scrollbars. Used to force redraw during scrolling
+   before actual scrolling happens, thus preventing shaking and
+   flickering artifacts. */
+.CodeMirror-vscrollbar, .CodeMirror-hscrollbar, .CodeMirror-scrollbar-filler, .CodeMirror-gutter-filler {
+  position: absolute;
+  z-index: 6;
+  display: none;
+}
+.CodeMirror-vscrollbar {
+  right: 0; top: 0;
+  overflow-x: hidden;
+  overflow-y: scroll;
+}
+.CodeMirror-hscrollbar {
+  bottom: 0; left: 0;
+  overflow-y: hidden;
+  overflow-x: scroll;
+}
+.CodeMirror-scrollbar-filler {
+  right: 0; bottom: 0;
+}
+.CodeMirror-gutter-filler {
+  left: 0; bottom: 0;
+}
+
+.CodeMirror-gutters {
+  position: absolute; left: 0; top: 0;
+  min-height: 100%;
+  z-index: 3;
+}
+.CodeMirror-gutter {
+  white-space: normal;
+  height: 100%;
+  display: inline-block;
+  vertical-align: top;
+  margin-bottom: -50px;
+}
+.CodeMirror-gutter-wrapper {
+  position: absolute;
+  z-index: 4;
+  background: none !important;
+  border: none !important;
+}
+.CodeMirror-gutter-background {
+  position: absolute;
+  top: 0; bottom: 0;
+  z-index: 4;
+}
+.CodeMirror-gutter-elt {
+  position: absolute;
+  cursor: default;
+  z-index: 4;
+}
+.CodeMirror-gutter-wrapper ::selection { background-color: transparent }
+.CodeMirror-gutter-wrapper ::-moz-selection { background-color: transparent }
+
+.CodeMirror-lines {
+  cursor: text;
+  min-height: 1px; /* prevents collapsing before first draw */
+}
+.CodeMirror pre.CodeMirror-line,
+.CodeMirror pre.CodeMirror-line-like {
+  /* Reset some styles that the rest of the page might have set */
+  -moz-border-radius: 0; -webkit-border-radius: 0; border-radius: 0;
+  border-width: 0;
+  background: transparent;
+  font-family: inherit;
+  font-size: inherit;
+  margin: 0;
+  white-space: pre;
+  word-wrap: normal;
+  line-height: inherit;
+  color: inherit;
+  z-index: 2;
+  position: relative;
+  overflow: visible;
+  -webkit-tap-highlight-color: transparent;
+  -webkit-font-variant-ligatures: contextual;
+  font-variant-ligatures: contextual;
+}
+.CodeMirror-wrap pre.CodeMirror-line,
+.CodeMirror-wrap pre.CodeMirror-line-like {
+  word-wrap: break-word;
+  white-space: pre-wrap;
+  word-break: normal;
+}
+
+.CodeMirror-linebackground {
+  position: absolute;
+  left: 0; right: 0; top: 0; bottom: 0;
+  z-index: 0;
+}
+
+.CodeMirror-linewidget {
+  position: relative;
+  z-index: 2;
+  padding: 0.1px; /* Force widget margins to stay inside of the container */
+}
+
+.CodeMirror-widget {}
+
+.CodeMirror-rtl pre { direction: rtl; }
+
+.CodeMirror-code {
+  outline: none;
+}
+
+/* Force content-box sizing for the elements where we expect it */
+.CodeMirror-scroll,
+.CodeMirror-sizer,
+.CodeMirror-gutter,
+.CodeMirror-gutters,
+.CodeMirror-linenumber {
+  -moz-box-sizing: content-box;
+  box-sizing: content-box;
+}
+
+.CodeMirror-measure {
+  position: absolute;
+  width: 100%;
+  height: 0;
+  overflow: hidden;
+  visibility: hidden;
+}
+
+.CodeMirror-cursor {
+  position: absolute;
+  pointer-events: none;
+}
+.CodeMirror-measure pre { position: static; }
+
+div.CodeMirror-cursors {
+  visibility: hidden;
+  position: relative;
+  z-index: 3;
+}
+div.CodeMirror-dragcursors {
+  visibility: visible;
+}
+
+.CodeMirror-focused div.CodeMirror-cursors {
+  visibility: visible;
+}
+
+.CodeMirror-selected { background: #d9d9d9; }
+.CodeMirror-focused .CodeMirror-selected { background: #d7d4f0; }
+.CodeMirror-crosshair { cursor: crosshair; }
+.CodeMirror-line::selection, .CodeMirror-line > span::selection, .CodeMirror-line > span > span::selection { background: #d7d4f0; }
+.CodeMirror-line::-moz-selection, .CodeMirror-line > span::-moz-selection, .CodeMirror-line > span > span::-moz-selection { background: #d7d4f0; }
+
+.cm-searching {
+  background-color: #ffa;
+  background-color: rgba(255, 255, 0, .4);
+}
+
+/* Used to force a border model for a node */
+.cm-force-border { padding-right: .1px; }
+
+@media print {
+  /* Hide the cursor when printing */
+  .CodeMirror div.CodeMirror-cursors {
+    visibility: hidden;
+  }
+}
+
+/* See issue #2901 */
+.cm-tab-wrap-hack:after { content: ''; }
+
+/* Help users use markselection to safely style text background */
+span.CodeMirror-selectedtext { background: none; }

development/model-comparison/css/custom.css

@@ -0,0 +1,8 @@
+/*
+Placeholder for custom user CSS
+
+mainly to be overridden in profile/static/custom/custom.css
+
+This will always be an empty file
+*/
+

development/model-comparison/css/override.css

@@ -0,0 +1,7 @@
+/*This file contains any manual css for this page that needs to override the global styles.
+This is only required when different pages style the same element differently. This is just
+a hack to deal with our current css styles and no new styling should be added in this file.*/
+
+#ipython-main-app {
+    position: relative;
+}

docs/backwardcompatibilityml/topics.rst

@@ -10,3 +10,4 @@ Help Topics
     topics/getting_started
     topics/integrating_loss_functions
     topics/using_the_compatibility_analysis_widget
+    topics/using_and_integrating_the_model_comparison_widget

docs/backwardcompatibilityml/topics/using_and_integrating_the_model_comparison_widget.rst

@@ -0,0 +1,83 @@
+.. _using_and_integrating_the_model_comparison_widget:
+
+Using the Model Comparison Widget
+=================================================================
+
+The model comparison widget uses the notion of compatibility to compare two
+models ``h1`` and ``h2``. The widget itself uses two graphs to display this
+comparison:
+
+1. A Venn diagram that displays the overlaps between the set of misclassified
+instances produced by ``h1`` and the set of misclassified instances produced
+by model ``h2``.
+
+2. A histogram that shows the number of incompatible instances, i.e. instances
+which have been misclassified by ``h2`` but not by ``h1``, on a per-class
+basis.
+
+A tabular view is also provided, that allows the user to explore the instances
+which have been misclassified. This tabular view is connected to both the
+Venn diagram and the histogram, and gets filtered based on how the user
+interacts with both those graphs.
+
+The following is an image of the model comparison widget.
+
+.. image:: ../media/model_comparison_widget.png
+    :width: 700
+
+
+How to Use the ModelComparison API
+----------------------------------------
+
+The model comparison widget accepts two models which are classifiers, ``h1`` and ``h2``.
+It also accepts a batched dataset consisting of a list of triples of the form:
+
+``[batch_of_instance_ids, batch_of_instance_data, batch_of_ground_truth_labels]``
+
+An optional parameter is a function passed in as a keyword parameter called
+``get_instance_image_by_id`` which is a function that returns a PNG image
+for a given instance id. This is what allows the error instances table to
+display an image representation of each instance. If this parameter is not
+specified, then the image representation of the instance defaults to a blank
+PNG image.
+
+An additional optional parameter is ``device``, which tells it whether it needs to run
+the comparison on the GPU. This depends on whether your models are on the GPU or not.
+
+With all the parameters as specified above, the widget may be invoked::
+
+    model_comparison = ModelComparison(h1, h2, train_loader,
+                                       get_instance_image_by_id=get_instance_image,
+                                       device="cuda")
+
+Within a Jupyter Notebook the above will render the component.
+
+An example notebook which walks you through a working example may be found at
+``./examples/model-comparison-MNIST`` from the BackwardCompatibilityML project root.
+
+
+Integrating the Model Comparison Widget
+---------------------------------------
+
+The data used to summarize the comparison of the models ``h1`` and ``h2``
+and display the results int he widget, are all pre-computed at the time of
+instantiation of the widget. This data is then passed directly to the
+widget UI at the time of rendering. And as such, any interactions tht the
+user performs with the widget, simply re-render the widget using this
+pre-computed data.
+
+Integration should just involve pre-computing the comparison data and making
+sure that it is passed to the javascript component at render time.
+
+The relevant places to see where this happens are within the following files.
+
+- ``backwardcompatibilityml/widgets/model_comparison/resources/widget.html``
+- ``backwardcompatibilityml/widgets/model_comparison/model_comparison.py``
+
+The Flask service is only used to field the requests from the widget UI that
+are needed to render the image representation of the widgets. This is currently
+done within the file ``backwardcompatibilityml/comparison_management.py``.
+
+It should be possible to do away with the Flask service in theory, if we simply
+pre-render each image as a base64 encoded data URL and include that in the UI.
+However this risks making the UI a bit slow to load.

modelcomparison.webpack.config.js

@@ -23,7 +23,7 @@ module.exports = {
       filename: 'widget.css'
     }),
     isDevelopment && new HtmlWebpackPlugin({
-      template: './development/index.html'
+      template: './development/model-comparison/index.html'
     }),
     isDevelopment && new ReactRefreshPlugin()
   ].filter(Boolean),

package.json

@@ -1,6 +1,6 @@
 {
   "dependencies": {
-    "axios": "^0.19.2",
+    "axios": "^0.21.1",
     "d3": "^5.16.0",
     "office-ui-fabric-react": "^7.117.0",
     "jquery": "^3.5.1",
@@ -53,7 +53,8 @@
     "webpack-dev-server": "^3.11.0"
   },
   "scripts": {
-    "start": "webpack-dev-server --hot --mode development --host 0.0.0.0",
+    "start-compatibility-analysis": "webpack-dev-server --config ./compatibilityanalysis.webpack.config.js --hot --mode development --host 0.0.0.0",
+    "start-model-comparison": "webpack-dev-server --config ./modelcomparison.webpack.config.js --hot --mode development --host 0.0.0.0",
     "build": "cross-env NODE_ENV=production webpack --config ./compatibilityanalysis.webpack.config.js && cross-env NODE_ENV=production webpack --config ./modelcomparison.webpack.config.js",
     "test": "jest"
   }

requirements.txt

@@ -1,6 +1,6 @@
 torch==1.5.1
 Jinja2==2.11.2
-numpy==1.19.4
+numpy==1.19.3
 scikit-learn==0.23.1
 rai_core_flask==0.0.2
 tensorboard==2.4.0

webpack.config.js

@@ -1,73 +0,0 @@
-const webpack = require('webpack');
-const path = require('path');
-const MiniCssExtractPlugin = require('mini-css-extract-plugin');
-const HtmlWebpackPlugin = require('html-webpack-plugin');
-const ReactRefreshPlugin = require('@pmmmwh/react-refresh-webpack-plugin');
-
-const isDevelopment = process.env.NODE_ENV !== 'production';
-
-module.exports = {
-  mode: isDevelopment ? 'development' : 'production',
-  entry: ['./widgets/compatibilityanalysis/index.tsx'],
-  devtool: 'eval-source-map',
-  performance: {
-    hints: false
-  },
-  watch: false,
-  plugins: [  
-    new webpack.ProvidePlugin({
-      $: "jquery",
-      jQuery: "jquery"
-    }),
-    new MiniCssExtractPlugin({
-      filename: 'widget.css'
-    }),
-    isDevelopment && new HtmlWebpackPlugin({
-      template: './development/index.html'
-    }),
-    isDevelopment && new ReactRefreshPlugin()
-  ].filter(Boolean),
-  module: {
-    rules: [
-      {
-        test: /\.ts(x?)$/,
-        exclude: /node_modules/,
-        use: [
-          isDevelopment && {
-            loader: 'babel-loader',
-            options: { plugins: ['react-refresh/babel'] }
-          },
-          {
-              loader: "ts-loader"
-          }
-        ].filter(Boolean)
-      },
-      // All output '.js' files will have any sourcemaps re-processed by 'source-map-loader'.
-      {
-        enforce: "pre",
-        test: /\.js$/,
-        loader: "source-map-loader"
-      },
-      {
-        test: /\.css$/,
-        use: [
-          isDevelopment && 'style-loader',
-          !isDevelopment && MiniCssExtractPlugin.loader,
-          'css-loader'
-        ].filter(Boolean)
-      }
-    ]
-  },
-  resolve: {
-    extensions: ['*', '.js', '.jsx', '*.ts', '*.tsx']
-  },
-  output: {
-    path: path.resolve(__dirname, 'backwardcompatibilityml/widget/resources'),
-    publicPath: '/',
-    filename: 'widget-build.js'
-  },
-  devServer: {
-    contentBase: path.resolve(__dirname, "development"),
-    port: 3000
-  }
-};

widgets/backwardcompatibilityml/widget/resources/widget.css

@@ -1,286 +0,0 @@
-/**
- * Copyright (c) Microsoft Corporation.
- * Licensed under the MIT License.
- */
-
-#viz {
-  font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
-  position: relative;
-  width: 960px;
-}
- 
-.axis text {
-  font: 10px sans-serif;
-}
- 
-.axis path,
-.axis line {
-  fill: none;
-  stroke: #000;
-  shape-rendering: crispEdges;
-}
- 
-.bar {
-  fill: rgba(170, 170, 198, 0.8);
-}
- 
-/*.bar:hover {
-  fill: rgba(226, 75, 158, 0.8);
-}*/
- 
-label {
-  position: absolute;
-  top: 10px;
-  right: 10px;
-}
-
-.plot {
-  position: relative;
-  display: inline-block;
-}
-
-.plot svg {
-  background-color: white
-}
-
-.plot-btc {
-  background-color: lightgrey;
-  padding-left: 20px;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 10px;
-}
-
-.plot-bec {
-  background-color: lightgrey;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 10px;
-}
-
-.plot-venn {
-  background-color: lightgrey;
-  padding-left: 20px;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 39px;
-  vertical-align: top;
-  line-height: 20px;
-}
-
-.plot-distribution {
-  background-color: lightgrey;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 10px;
-  line-height: 20px;
-}
-
-.plot-title-row {
-  display: flex;
-  background-color: white;
-  flex-direction: row;
-  justify-content: center;
-  align-items: center;
-  font-family: sans-serif;
-  font-size: 10px;
-  padding-left: 40px;
-  padding-top: 5px;
-  padding-bottom: 5px;
-}
-
-.page-button-row {
-  display: flex;
-  flex-direction: row;
-  justify-content: center;
-  margin-top: 5px;
-}
-
-.page-button-row span {
-  margin-left: 5px;
-  margin-right: 5px;
-  padding-top: 2px;
-}
-
-.page-button-row button {
-  background-color: white;
-  border: 1px solid black;
-}
-
-.venn-legend-row {
-  display: flex;
-  background-color: white;
-  flex-direction: row;
-  font-family: sans-serif;
-  justify-content: center;
-  align-items: center;
-  font-size: 10px;
-  padding-left: 37px;
-}
-
-.venn-legend-row-block {
-  display: flex;
-  flex-direction: row;
-  align-items: center;
-  padding: 5px;
-}
-
-.venn-legend-row-block-selected {
-  display: flex;
-  flex-direction: row;
-  align-items: center;
-  border: 2px solid rgb(200,200,200);
-  padding: 3px;
-}
-
-.venn-legend-row-block:hover {
-  border: 2px solid black;
-  padding: 3px;
-}
-
-.venn-legend-row-block-selected:hover {
-  border: 2px solid black;
-  padding: 3px;
-}
-
-.venn-legend-color-box {
-  height: 10px;
-  width: 10px;
-  margin-right: 5px;
-}
-
-.tooltip {
-  background-color: black;
-  color: white;
-  border-radius: 3px;
-  position: absolute;
-  padding-left: 2px;
-  padding-right: 2px;
-}
-
-.table {
-  width: 700px;
-  text-align: center;
-  background-color: lightblue;
-}
-
-.data-selector .control-group {
-  display: inline-block;
-  margin-right: 50px;
-}
-
-.data-selector {
-  width: 700px;
-  text-align: left;
-}
-
-.data-selector .control-subgroup {
-  display: inline-block;
-  margin-right: 20px;
-}
-
-.data-selector .control {
-  display: inline-block;
-  margin-right: 5px;
-}
-
-.dropdown-check-list {
-  font-size: 10px;
-  position: relative;
-  display: inline-block;
-}
-
-.dropdown-check-list .anchor {
-  font-weight: bold;
-  position: relative;
-  cursor: pointer;
-  display: inline-block;
-  padding: 5px 50px 5px 10px;
-  border: 1px solid #ccc;
-}
-
-.dropdown-check-list .anchor:after {
-  position: absolute;
-  content: "";
-  border-left: 2px solid black;
-  border-top: 2px solid black;
-  padding: 5px;
-  right: 10px;
-  top: 20%;
-  -moz-transform: rotate(-135deg);
-  -ms-transform: rotate(-135deg);
-  -o-transform: rotate(-135deg);
-  -webkit-transform: rotate(-135deg);
-  transform: rotate(-135deg);
-}
-
-.dropdown-check-list .anchor:active:after {
-  right: 8px;
-  top: 21%;
-}
-
-.dropdown-check-list ul {
-  position: absolute;
-}
-
-.dropdown-check-list ul.items {
-  padding: 2px;
-  display: none;
-  margin: 0;
-  border: 1px solid #ccc;
-  border-top: none;
-  background: white;
-  z-index: 100;
-}
-
-.dropdown-check-list ul.items li {
-  list-style: none;
-}
-
-.dropdown-check-list.visible .anchor {
-  color: #0094ff;
-}
-
-.dropdown-check-list.visible .items {
-  display: block;
-}
-
-#raw-values-table {
-  width: 700px;
-  margin-top: 0px;
-  border-style: solid;
-  border-width: 1px;
-}
-
-.model-details .model-details-row {
-  width: 100%;
-  display: inline-block;
-  text-align: center;
-  padding: 0.1em 0.1em;
-}
-
-.model-details-row td {
-  display: inline-block;
-  text-align: center;
-  width: 180px;
-}
-
-.model-details-info-data {
-  display: inline-block;
-  font-weight: bold;
-  font-size: 10px;
-  margin-right: 4px;
-}
-
-.model-details-info-type {
-  display: inline-block;
-  font-style: italic;
-  font-size: 10px;
-  font-weight: normal;
-}
-
-.highlighted-bar {
-  stroke-width: 2;
-  stroke: rgb(0,0,0);
-}
-

widgets/backwardcompatibilityml/widgets/compatibility_analysis/resources/widget.css

@@ -1,286 +0,0 @@
-/**
- * Copyright (c) Microsoft Corporation.
- * Licensed under the MIT License.
- */
-
-#viz {
-  font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
-  position: relative;
-  width: 960px;
-}
- 
-.axis text {
-  font: 10px sans-serif;
-}
- 
-.axis path,
-.axis line {
-  fill: none;
-  stroke: #000;
-  shape-rendering: crispEdges;
-}
- 
-.bar {
-  fill: rgba(170, 170, 198, 0.8);
-}
- 
-/*.bar:hover {
-  fill: rgba(226, 75, 158, 0.8);
-}*/
- 
-label {
-  position: absolute;
-  top: 10px;
-  right: 10px;
-}
-
-.plot {
-  position: relative;
-  display: inline-block;
-}
-
-.plot svg {
-  background-color: white
-}
-
-.plot-btc {
-  background-color: lightgrey;
-  padding-left: 20px;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 10px;
-}
-
-.plot-bec {
-  background-color: lightgrey;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 10px;
-}
-
-.plot-venn {
-  background-color: lightgrey;
-  padding-left: 20px;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 39px;
-  vertical-align: top;
-  line-height: 20px;
-}
-
-.plot-distribution {
-  background-color: lightgrey;
-  padding-right:20px;
-  padding-top: 20px;
-  padding-bottom: 10px;
-  line-height: 20px;
-}
-
-.plot-title-row {
-  display: flex;
-  background-color: white;
-  flex-direction: row;
-  justify-content: center;
-  align-items: center;
-  font-family: sans-serif;
-  font-size: 10px;
-  padding-left: 40px;
-  padding-top: 5px;
-  padding-bottom: 5px;
-}
-
-.page-button-row {
-  display: flex;
-  flex-direction: row;
-  justify-content: center;
-  margin-top: 5px;
-}
-
-.page-button-row span {
-  margin-left: 5px;
-  margin-right: 5px;
-  padding-top: 2px;
-}
-
-.page-button-row button {
-  background-color: white;
-  border: 1px solid black;
-}
-
-.venn-legend-row {
-  display: flex;
-  background-color: white;
-  flex-direction: row;
-  font-family: sans-serif;
-  justify-content: center;
-  align-items: center;
-  font-size: 10px;
-  padding-left: 37px;
-}
-
-.venn-legend-row-block {
-  display: flex;
-  flex-direction: row;
-  align-items: center;
-  padding: 5px;
-}
-
-.venn-legend-row-block-selected {
-  display: flex;
-  flex-direction: row;
-  align-items: center;
-  border: 2px solid rgb(200,200,200);
-  padding: 3px;
-}
-
-.venn-legend-row-block:hover {
-  border: 2px solid black;
-  padding: 3px;
-}
-
-.venn-legend-row-block-selected:hover {
-  border: 2px solid black;
-  padding: 3px;
-}
-
-.venn-legend-color-box {
-  height: 10px;
-  width: 10px;
-  margin-right: 5px;
-}
-
-.tooltip {
-  background-color: black;
-  color: white;
-  border-radius: 3px;
-  position: absolute;
-  padding-left: 2px;
-  padding-right: 2px;
-}
-
-.table {
-  width: 700px;
-  text-align: center;
-  background-color: lightblue;
-}
-
-.data-selector .control-group {
-  display: inline-block;
-  margin-right: 50px;
-}
-
-.data-selector {
-  width: 700px;
-  text-align: left;
-}
-
-.data-selector .control-subgroup {
-  display: inline-block;
-  margin-right: 20px;
-}
-
-.data-selector .control {
-  display: inline-block;
-  margin-right: 5px;
-}
-
-.dropdown-check-list {
-  font-size: 10px;
-  position: relative;
-  display: inline-block;
-}
-
-.dropdown-check-list .anchor {
-  font-weight: bold;
-  position: relative;
-  cursor: pointer;
-  display: inline-block;
-  padding: 5px 50px 5px 10px;
-  border: 1px solid #ccc;
-}
-
-.dropdown-check-list .anchor:after {
-  position: absolute;
-  content: "";
-  border-left: 2px solid black;
-  border-top: 2px solid black;
-  padding: 5px;
-  right: 10px;
-  top: 20%;
-  -moz-transform: rotate(-135deg);
-  -ms-transform: rotate(-135deg);
-  -o-transform: rotate(-135deg);
-  -webkit-transform: rotate(-135deg);
-  transform: rotate(-135deg);
-}
-
-.dropdown-check-list .anchor:active:after {
-  right: 8px;
-  top: 21%;
-}
-
-.dropdown-check-list ul {
-  position: absolute;
-}
-
-.dropdown-check-list ul.items {
-  padding: 2px;
-  display: none;
-  margin: 0;
-  border: 1px solid #ccc;
-  border-top: none;
-  background: white;
-  z-index: 100;
-}
-
-.dropdown-check-list ul.items li {
-  list-style: none;
-}
-
-.dropdown-check-list.visible .anchor {
-  color: #0094ff;
-}
-
-.dropdown-check-list.visible .items {
-  display: block;
-}
-
-#raw-values-table {
-  width: 700px;
-  margin-top: 0px;
-  border-style: solid;
-  border-width: 1px;
-}
-
-.model-details .model-details-row {
-  width: 100%;
-  display: inline-block;
-  text-align: center;
-  padding: 0.1em 0.1em;
-}
-
-.model-details-row td {
-  display: inline-block;
-  text-align: center;
-  width: 180px;
-}
-
-.model-details-info-data {
-  display: inline-block;
-  font-weight: bold;
-  font-size: 10px;
-  margin-right: 4px;
-}
-
-.model-details-info-type {
-  display: inline-block;
-  font-style: italic;
-  font-size: 10px;
-  font-weight: normal;
-}
-
-.highlighted-bar {
-  stroke-width: 2;
-  stroke: rgb(0,0,0);
-}
-