* Fix bug that didn't correctly calculate the error instances for the h2_new_error and h2_strict_imitation_models
* Bump tensorflow and PIL versions
* Add test
* Apply appropriate re-scaling factor to the dissonance to take into account the mean reduction being performed on the loss over the batch
* Fix dev harness CSS imports
* checkboxes and lambda range slider - render only
* Create LambdaSlider component
* Functional lambda slider
* Remove debug display for lambda range
* increase widget width to 1000px
* start sweep button rework
* Add shapes to DataSelector component
* Add padding to data selector and inner components
* New legend with updated shapes
* Triangles on plot, header above dataset selector
* Point selection from dropdowns
* Enlarge triangles on hover
* Sync point selection between scatter plot and the selection dropdowns
* Heading and zoom buttons for scatter plots
* remove some prototype code
* Zoom prototype using d3.zoom()
* Fix points disappearing, add slider ticks, filter dropdown values
* zoom with buttons, clip points outside plot area
* lambda slider styles, ticks; webpack file-loader
* resize zoom button, new reset zoom button, h1 accuracy text
* save zoom transform in state
* Remove unneccessary changes
* Ignore generated .png files in build output
* Add proper handling for rendering of components for certain blank states
Co-authored-by: Xavier Fernandes <fernandes.praphat@gmail.com>
* Add the accuracy by class grouped bar chart and bar chart selector
Update the training helper to keep track of the accuracy by class per model.
Add the accuracy by class grouped bar chart. Add the chart selector to allow
the user to switch between the accuracy by class chart and the incompatible
points chart.
* Add model accuracy by class legend
* Use Primary and Default buttons from FabricUI
* Fix ModelComparison widget UI alignment issue
* Model comparison dev harness fix
* Make sure the compatibility analysis examples describe realistic scenarios
Make sure that h1 is trained on a fraction of the daata.
And make sure that h2 is then trained on all of the data.
* Use Logistic Regression model in the compatibility analysis example over the Adult dataset for better results
* Fix bug that prevented the ModelComparison widget from being re-instantiated without re-starting the notebook / kernel
* Remove use_mlflow_flag from ModelComparison widget
* Remove use_mlflow flag
* Include h1 performance when calculating the Y-axis scale, and add a buffer region above and below on the scale
* Add some checks to verify that the error instances are calculated correctly from the appropriate dataset, i.e. testing vs training and that they don't get mixed up
* Fix the issue with training on the UCI Adult dataset, by dropping the "fnlwgt" column which was biasing the training
* Update one more example that uses the UCI Adult dataset to fix the training
* Add docstrings to Tensorflow models module, and add more Tensorflow documentation in topics section
* Add docstrings to backwardcompatibilityml.tensorflow.helpers.bc_fit
* Bump version to v1.4.1
* Documentation style changes
* More edits
Bumped version to 1.4.0
Update README.md to include new team members
Update README.md to include links to representative example notebooks
Update docs to note that widget only works with Pytorch
Update docs to include references to all new Tensorflow loss functions and new Models
* Fix the pagination bugs from filter instances
* Clear the selected regions of histogram and venn diagram when a new data point is selected
* Typo fix
* Reset to page 1 when the table is filtered
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Separate dev harnesses for CompatibilityAnalsysis and ModelComparison
* Fix some issues and bump Axios version
* Update README instructions and remove commented code
* Remove unused files
* Remove unnecessary arguments from ModelComparison widget instantiation
* Remove unused arguments from ModelComparison widget instantiation within the dev harness code
* Pin numpy to 1.19.3
* Fix typo in README
* Add docstrings for the ModelComparison widget and ComparisonManagement class
* Add documentation for Model Comparison widget
* Create widgets folder structure and build system for building multiple widgets with shared components
Create a widgets folder for multiple widgets written in Typescript.
Create widgets folder structure for separate Python infrastructure
for multiple widgets. Update the build system to allow the build
of multiple widgets. Separate out UI components and logic for the
Model Comparison widget. Update all examples and code to import
CmpatibilityAnalysis and ModelComparison widgets from the updated
import path.
* Remove old single widget folder paths and files and change references in TS tests
* Remove build artifacts and fix path mistake in .gitignore and update setup.py
* Remove old text from docstrings
* Re-add old webpack.config.js file
* Change entrypoint
* Add backward compatibility loss functions for new error and strict imitation under Tensorflow implmentation together with example notebooks
* Add parameters to clip values before taking log to prevent gradients from becoming NaNs
* Add docstrings and cleanup implementation
* Bump tensorflow version and related requirements
* MLFlow integration working, threading issue solved, API route conflict with multiple instantiation of widget fixed
* Add parameters use_ml_flow and ml_flow_run_name to control MLFlow
* Update readme with MLflow info
* Set use_ml_flow=False by default. Add MLflow to requirements.txt.
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Implement a Backward Compatibility New Error Model for training Tensorflow models using the BC New Error Loss
* Raise error if parameter h1 is not set
* Add generalized BCLoss implementation for Tensorflow to allow training arbitrary models
* Rename variable
* temporary experimental notebook for widget examples
* First draft of widget usage documentation
* remove test file from branch
* Typo fix
* Revisions from Kathleen
* Add some additional info about how the widget works
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Initial experimentation
* Created InfoTooltip component. Chart titles have tooltips.
* Make the i icon bigger and change color to black
* Full functionality implemented
* Fix venn diagram alignment
* Refactor venn diagram legend into its own component.
* Change VennLegend to a class component
* implementation basically done but actions not dispatched to reducers
* Add missing setSelectedClass and setSelectedRegion parameters to the Container component
* Revert "Change VennLegend to a class component"
This reverts commit 78e36e8cf7.
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Replace usage of size_average keyword argument with reduction keyword argument set to "sum" where appropriate
* Selectable legend in Venn diagram
* Remove commented code
* Make the clickable area larger
* Fix CSS to align venn diagram box properly. Structure dev harness HTML to resemble Jupyter notebok.
* Remove some unnecessary experimental code
* Add image loading to dev harness backend
* Rename get_instance_data_by_id to get_instance_image_by_id
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Add cases for Venn diagram to deal with disjoint h1 and h2 error sets and total intersection
* Fix the selectivity issue found in disjoint sets and do some variable renaming and fix inversion issue
* Added new loss functions to and examples to docs
* Updated docstrings to reflect binary cross entropy loss naming
* Updated auto-generated docs from docstrings
* Fixed sphinx doc build warnings
* Fixed spelling of "cross-entropy"
* Fix the random seed to make demos deterministic and rename the image generation function passed to the widget
* Set the torc.backends.cudnn.deterministic to True and torch.backends.cudnn.benchmark to False
* Add statefulness to selectivity of the regions of the Venn diagram, and calculate the percentage of errors out of the total number of instances
* Better differentiate hover vs select styles
* Change h2 errors by class to h1h2 incompatible points by class
Fix some variable names - feature is working end to end
Fix app crash if there are no incompatible points
Refactor finding incompatible instances to its own method
simplify get_incompatible_instances_by_class and add docstring
* Merged code in working state
* remove debug print
* fix bar click to use d.incompatibleInstanceIds instead of d.errorInstanceIds
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Add textual rendering option for error instances table
* Apply filters to the error instances table based on clicking on the venn diagram and the bar chart
* Remove console.log debug statements and add relevant docstrings
* Change SelectedModelDetails to use a table, rework styles, remove RawValues files that are not used.
* Revise styling based on design review
* Fix text centering
* Fix double negative sign issue
* Remove !important from styles
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Setup gitignore for development/sweeps folder
* Functional pagination on histogram. Styling is still needed.
* CSS styling for page buttons
* Fix null ref with selectedDataPoint, fix styling of widget in notebook
* Change typescript variables to camelCase
* Fix infinite loop in error instances table
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Basic table component with error instances
* Send index select indices to appropriate device
* Remove un-necessary move to device
* kRender instance image data within table component
* Add pagination to error instances table
* Update all the example Notebooks to use a dataset with associated ids
* Fix the dev environment and update the docstrings regarding the use of data instance ids
* Update docstrings for `get_error_instance_indices`
* Use user provied funtion to display images
* Fix pagination in error instances table
* Create performanceFunctionLabel prop
* Add performance_metric parameter to compatibility_analysis API, display the name in scatter plot.
* Rename performanceFunctionLabel prop -> performanceMetric for consistency
* Don't rename model_accuracy to Accuracy. Metric name centered.
* Documentation for performance metric function
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* First experiments in setting up test framework
* Basic React widget test passing
* Jest setup in package.json. Move tests to their own folder.
* Remove jest.config.ts
* Basic DataSelector test
* Remove sum.js used in testing test setup
* Configure ts-jest
* Rename main test file
* Added test for 'select all' checkboxes in DataSelector
* Remove unnecessary code
* Add UI test info to the Readme
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* 2 build targets for webpack - jupyter and dev
* First revision of Flask dev server hacked together
* Refactored code to maximize reuse. Changed webpack dev build to export a separate artifact to widget/resources directory instead of dist/
* Remove widget-build-dev from checkin, add to gitignore
* Setup webpack-dev-server and React hot loader. 'npm start' runs the web server. Debugger setup for VSCode. Integrated front end and backend.
* Code cleanup - remove unnecessary files, move .flaskenv to project root, cleanup linter errors
* widget.css has to be exported for Jupyter notebook - fix webpack
* Refactored compatibility_analysis.py and app.py to separate code more appropriately. Webpack now exports dev widget to dev-artifacts folder. Todo: Use FlaskHelper in dev mode, include Jupyter styles in dev widget build
* Merge webpack configs into 1 config that changes dynamically. Fix React hot loading. Use HtmlWebpackPlugin to generate index.html from template, removing dev build artifacts on file system. Linting & simplification of python code.
* Remove dev-artifacts folder from .gitignore
* Mode dev harness code to development/ folder outside of the widget code. Update readme with dev harness usage instructions. Update release.sh with new build command.
* Fix CompatibilityAnalysis import in app.py
* Include all CSS from Jupyter notebook
* Remove .dev-container CSS class
* Move development artifacts to development/ directory
* Change dev API port to 5000
* Revert a deletion in training.py, add info about non-localhost usage in Readme
Co-authored-by: Nicholas King <v-nicki@microsoft.com>
* Display error instance counts as well as percentages in the Venn diagram
* Fix imports in one of the example notebooks, and pass on the device correctly when calling modeL_accuracy
* Move batch size and epochs variable definitions to the top
* Add basic component for showing selected h2 model details, update color scheme, and make the point selection stateful and sticky
* Use dropdowns with checkboxes within the DataSelector component and remove the old plain check boxes
* Styling of the SelectedModel comonent
* Style dropdowns
* Display the h1 model performance information in the scatterplots
Re-structure the sweep summary data so that we can include information such as h1 model performance.
Update the UI to deal with this new data structure. And update the scatter plots to display the
h1 model performance as a horizontal dotted line.
* Correctly use the xScale and yScale to plot the h1 performance dotted line
* Changed Read the Docs master file to index.rst (#2)
* Added example-requirements.txt (#3)
Updated getting started with required pip find-links URL
* Adding new error and strict imitation compatibility loss functions for KL Divergence and Binary Cross Entropy losses (#4)
* Add new error and strict imitation compatibility loss functions for KL Divergence and add a few notebook examples using it
* Add New Error and Strict Imitation compatibility losses for Binary Cross Entropy Loss
* Finalize changes to strict imitation binary cross entropy loss and add a simple example for new error binary cross entropy loss
* update acknowledgements README (#5)
Co-authored-by: Besmira Nushi <benushi@microsoft.com>
* Add component to display colot legend for the scatter plots, and fix the Venn diagram component so that all regions are proportional (#6)
* Add tooltips to both the scatterplot and the Venn diagram, and highlight the correct area on hover (#7)
* PR to add dropdowns, and miscellaneous changes discussed previously (#8)
* Add basic component for showing selected h2 model details, update color scheme, and make the point selection stateful and sticky
* Use dropdowns with checkboxes within the DataSelector component and remove the old plain check boxes
* Styling of the SelectedModel comonent
* Style dropdowns
* Display the h1 model performance information in the scatterplots
Re-structure the sweep summary data so that we can include information such as h1 model performance.
Update the UI to deal with this new data structure. And update the scatter plots to display the
h1 model performance as a horizontal dotted line.
* Correctly use the xScale and yScale to plot the h1 performance dotted line
* Version bump to 1.1.0 (#22)
Co-authored-by: ilmarinen <fernandes.praphat@gmail.com>
Co-authored-by: nushib <besmiranushi@gmail.com>
Co-authored-by: Besmira Nushi <benushi@microsoft.com>
ilmarinen
Nicholas King
ilmarinen
John (Juan) Lema
Nicholas King
Nicholas King