microsoft
/
onnxruntime-web-benchmark
зеркало из
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Refactoring a benchmark tool

This commit is contained in:
Sunghoon 2021-09-08 22:17:21 -07:00
Родитель b3968107ed
Коммит 3ec7733b74
22 изменённых файлов: 796 добавлений и 901 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

35
README.md
Просмотреть файл

@ -1,45 +1,30 @@
# ORTWebBenchmark
# ONNX Runtime Web Benchmark
A benchmarking tool under development. Current it supports running wasm and webgl backends with profiling for tfjs and ort-web frameworks.
## Setup
Change the package version for onnxruntime-web in package.json to point to either an official release or a local build.
i.e
If using local build and onnxruntime's base folder is `~/onnxruntime`, set `onnxruntime-web` dependency in package.json to:
i.e. if using local build and onnxruntime's base folder is `~/onnxruntime`, set `onnxruntime-web` dependency in package.json to:
`"onnxruntime-web": "file:~/onnxruntime/js/web"`.
***Note: Make sure that all of the .wasm libraries including threaded and simd.threaded files exist in the dist folder(js/web/dist) of ort-web.
Note that all of the .wasm libraries including threaded and simd.threaded files must exist in the dist folder(js/web/dist) of ort-web.
If using official release, simply set it with a version number.
In the root directory, run `npm install` to install all required packages.
Set the environment variable `ORT_BASE` to the base directory of onnxruntime.
i.e
onnxruntime is located at `~/onnxruntime`, then run `export ORT_BASE=~/onnxruntime`.
To build the bundle, run `npm run build` in the root directory or `ORT_BASE=~/onnxruntime npm run build` if previous step is not done.
***Note: The build script triggers webpack to look for .wasm files in the onnxruntime directory, if files are not found, the build will continue without the capability to benchmark wasm backend.
To build the bundle, run `npm run build` in the root directory.
## Run
To start benchmarking, run `npm run benchmark`. Users need to provide a runtime configuration file that contains all parameters. By default, it looks for `run_config.json` in the root folder. Some sample configuration files are provided under `/sample-configs`. To pass a custom config to the benchmarking run, run `RUN_CONFIG=path_to_config npm run benchmark`.
i.e.
`RUN_CONFIG=./sample-configs/run_config_softmax_test.json npm run benchmark`
To start benchmarking, run `npm run benchmark`. Users need to provide a runtime configuration file that contains all parameters. By default, it looks for `run_config.json` in the root folder. Some sample configuration files are provided under `/sample-configs`. To pass a custom config to the benchmarking run, run `npm run benchmark --run_config=path_to_config`.
i.e. `npm run benchmark --run_config=./sample-configs/run_config_softmax_test.json`
Profiling data can be generated using `npm run profile` command. It will create a chrome-trace event file named trace.json in the root folder. This file can be loaded into `chrome://tracing` for visualization.
Profiling data can be generated using `npm run profile` command. It will create a chrome-trace event file named trace.json in the root folder. This file can be loaded into `chrome://tracing` for visualization. Note that model files must exist under the `data/` directory in the root. The browser doesn't have access to local file system so files need to be uploaded to the local server when benchmark starts up. All content under `data/` will be served.
***Note: Make sure to put the image and model files under the `data/` directory in the root. The browser doesn't have access to local file system so files need to be uploaded to the local server when benchmark starts up. All content under `data/` will be served.
### Input formats
Currently, this tool supports 3 types of inputs:
1. Synthetic data -> Randomly generated tensors during runtime. Set `inputConfig.useSyntheticData` to true to enable it. Note that this will override other input methods.
2. Image data -> tensors are created using given images of types jpg or bmp. Set urls in `inputConfig.customInputPaths` to instruct the tool of the location of the images to load. Urls can be a local path or web url.
3. Protobuf files -> tensors are created using given .pb files. Set urls in `inputConfig.customInputPaths` to instruct the tool of the location of the pb files to load.
This tool generates synthetic input data referring to a model input shape and a hint by 'config.ortweb.shape' or 'config.tfjs.shape' when a shape is dynamic.
### Input models
#### Ort-web
Only .onnx models are supported as of now. Set inputConfig.onnxModelConfig.path to the file path.
Only .onnx models are supported as of now. Set config.ortweb.path to the file path.
#### Tensorflow.js
Tensorflow.js models can be loaded in 2 ways:
1. Plain javascript file -> This method supports dynamically loading a javascript file that describes a tfjs model. The .js file will be inlined into the main page at runtime. Set `inputConfig.tfjsModelConfig.path` to the .js file path. An example is provided in `sample-configs/run_config_super_resolution.json`. The .js file should only contain 1 function with this signature `function createModel(tf)` and return a constructed tfjs model object. This function is then called in the main script tp retrieve the model. An example is provided in `data/model-tfjs/super-resolution/super-res.js`
2. model.json and model.bin -> This is the official way provided by TFJS project to load models. It will use `tf.LoadLayeredModel` or `tf.LoadGraphModel` to load the model structure and weights. Set `inputConfig.tfjsModelConfig.path` to the path of `model.json`. The bin file which contains the weights should be in the same directory as `model.json`.
Tensorflow.js models can be either a layers model or a graph model with model.json and model.bin -> This is the official way provided by TFJS project to load models. It will use `tf.LoadLayeredModel` or `tf.LoadGraphModel` to load the model structure and weights. Set `config.tfjs.path` to the path of `model.json`. The bin file which contains the weights should be in the same directory as `model.json`.

227
ThirdPartyNotices.txt Normal file
Просмотреть файл

@ -0,0 +1,227 @@
THIRD PARTY SOFTWARE NOTICES AND INFORMATION
Do Not Translate or Localize
This software incorporates material from third parties. Microsoft makes certain
open source code available at http://3rdpartysource.microsoft.com, or you may
send a check or money order for US $5.00, including the product name, the open
source component name, and version number, to:
Source Code Compliance Team
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
USA
Notwithstanding any other terms, you may reverse engineer this software to the
extent required to debug changes to any libraries licensed under the GNU Lesser
General Public License.
_______________________________________________________________________________
Mediapipe Facemesh
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
_______________________________________________________________________________

Двоичные данные
data/model-onnx/mediapipe-facemesh/model.onnx Normal file
Просмотреть файл

Двоичный файл не отображается.

Двоичные данные
data/model-onnx/super-resolution/super_resolution.onnx
Просмотреть файл

Двоичный файл не отображается.

1
data/model-onnx/test_softmax_axis_0/test_data_set_0/input_0.pb
Просмотреть файл

@ -1 +0,0 @@
BxJđxĚá?háĚ>“Žz?Ëj@$ ď?â.z?˙8s?bý>hdÓ=ř9Ň>(€>˘%ş?^ÓB?Ŕ0ů= Bă>]ת>ü=ż?R>iJ >¦Z?/d#@ŚS'?±K]?‡ţ=?©C@¨(ş?Hm;= ­?>2Ä?óĽ?ŠŞ>…žÁ>íEc?˝Šý?‹!˛>ň >*zť?•ç™?łOĆ>mÇš>ü6†?&õ?gÚ?łů?‘x?FKŕ>™[ ?ś G?4”Î?—ŘY>L=e?Ć> Ä?ő—?kŢć<QNŰ>.:<3A>=™Ýš>ďb"?6ąą>

Двоичные данные
data/model-onnx/test_softmax_axis_0/test_data_set_0/output_0.pb
Просмотреть файл

Двоичный файл не отображается.

Двоичные данные
data/model-tfjs/mediapipe-facemesh/group1-shard1of1.bin Normal file
Просмотреть файл

Двоичный файл не отображается.

1
data/model-tfjs/mediapipe-facemesh/model.json Normal file
Просмотреть файл

Различия файлов скрыты, потому что одна или несколько строк слишком длинны

48
data/model-tfjs/super-resolution/super-res.js
Просмотреть файл

@ -1,48 +0,0 @@
function createModel(tf) {
var tfModel = tf.sequential();
tfModel.add(tf.layers.conv2d({
inputShape: [224,224,1],
kernelSize: [5,5],
padding: 'same',
filters: 64,
strides: 1,
activation: 'relu',
useBias: true,
biasInitializer: 'ones',
kernelInitializer: 'varianceScaling'
}));
tfModel.add(tf.layers.conv2d({
kernelSize: 3,
padding: 'same',
filters: 64,
strides: 1,
activation: 'relu',
useBias: true,
biasInitializer: 'ones',
kernelInitializer: 'varianceScaling'
}));
tfModel.add(tf.layers.conv2d({
kernelSize: 3,
padding: 'same',
filters: 32,
strides: 1,
activation: 'relu',
useBias: true,
biasInitializer: 'ones',
kernelInitializer: 'varianceScaling'
}));
tfModel.add(tf.layers.conv2d({
kernelSize: 3,
padding: 'same',
filters: 9,
strides: 1,
activation: 'relu',
useBias: true,
biasInitializer: 'ones',
kernelInitializer: 'varianceScaling'
}));
tfModel.add(tf.layers.reshape({targetShape: [1,3,3,224,224]}))
tfModel.add(tf.layers.permute({dims: [1, 4, 2, 5, 3]}));
tfModel.add(tf.layers.reshape({targetShape: [1,672,672]}));
return tfModel;
}

54
karma.conf.js
Просмотреть файл

@ -1,33 +1,18 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
'use strict';
// Karma configuration
const path = require('path')
const fs = require('fs');
function getMachineIpAddress() {
var os = require('os');
var ifaces = os.networkInterfaces();
for (const ifname in ifaces) {
for (const iface of ifaces[ifname]) {
if ('IPv4' !== iface.family || iface.internal !== false) {
// skip over internal (i.e. 127.0.0.1) and non-ipv4 addresses
continue;
}
// returns the first available IP address
return iface.address;
}
}
// if no available IP address, fallback to "localhost".
return 'localhost';
}
module.exports = function(config) {
let runConfigFile = config.run_config ? config.run_config : 'run_config.json';
if (!fs.existsSync(runConfigFile) && !fs.existsSync(path.resolve(__dirname, runConfigFile))){
runConfigFile = 'run_config.json';
if(!fs.existsSync(path.resolve(__dirname, runConfigFile))) {
throw new Error("Couldn't find any configuration file. Please make sure that RUN_CONFIG is set in env variable or run_config.json exists in root.");
throw new Error("Couldn't find any configuration file. Set a configuration as '--run_config=<config file>'or put run_config.json in root.");
}
}
@ -35,10 +20,15 @@ module.exports = function(config) {
basePath: './',
frameworks: ['mocha'],
files: [
{ pattern: 'dist/tf.min.js' },
{ pattern: 'dist/tf-backend-wasm.min.js' },
{ pattern: 'dist/ort-common.min.js' },
{ pattern: 'dist/ort-web.min.js' },
{ pattern: 'dist/main.js' },
{ pattern: 'dist/ort-wasm.wasm', included: false},
{ pattern: 'dist/ort-wasm-simd.wasm', included: false},
{ pattern: 'dist/ort-wasm-threaded.wasm', included: false},
{ pattern: 'dist/ort-wasm-threaded.worker.js', included: false},
{ pattern: 'dist/ort-wasm-simd-threaded.wasm', included: false},
{ pattern: 'dist/tfjs-backend-wasm-simd.wasm', included: false, nocache: true},
{ pattern: 'dist/tfjs-backend-wasm-threaded-simd.wasm', included: false, nocache: true},
@ -51,10 +41,14 @@ module.exports = function(config) {
'/ort-wasm-simd.wasm': '/base/dist/ort-wasm-simd.wasm',
'/ort-wasm-threaded.wasm': '/base/dist/ort-wasm-threaded.wasm',
'/ort-wasm-simd-threaded.wasm': '/base/dist/ort-wasm-simd-threaded.wasm',
'/ort-wasm-threaded.worker.js': '/base/dist/ort-wasm-threaded.worker.js',
'/tfjs-backend-wasm-simd.wasm': '/base/dist/tfjs-backend-wasm-simd.wasm',
'/tfjs-backend-wasm-threaded-simd.wasm': '/base/dist/tfjs-backend-wasm-threaded-simd.wasm',
'/tfjs-backend-wasm.wasm': '/base/dist/tfjs-backend-wasm.wasm'
},
},
mime: {
"text/x-typescript": ["ts"],
},
exclude: [
],
// available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
@ -68,20 +62,16 @@ module.exports = function(config) {
browserDisconnectTolerance: 0,
browserSocketTimeout: 60000,
logLevel: config.LOG_VERBOSE,
hostname: getMachineIpAddress(),
customLaunchers: {
ChromeTest: {base: 'Chrome', flags: ['--window-size=1,1',
'--disable-renderer-backgrounding',
'--disable-web-security',
'--disable-site-isolation-trials']},
ChromeDebug: {debug: true, base: 'Chrome', flags: ['--remote-debugging-port=9333']}
ChromeTest: {base: 'Chrome', flags: ['--window-size=1,1', '--enable-features=SharedArrayBuffer']},
ChromeDebug: {debug: true, base: 'ChromeHeadless', flags: ['--remote-debugging-port=9333', '--enable-features=SharedArrayBuffer']}
},
client: {
captureConsole: true,
mocha: {expose: ['body'], timeout: 3000000},
browser: config.browsers,
printMatches: false,
// To use custom config, run 'RUN_CONFIG=config_file_name npm run benchmark'
// To use custom config, run 'npm run benchmark --run_config=config_file_name'
runConfig: runConfigFile,
profile: config.profile
},
@ -89,6 +79,6 @@ module.exports = function(config) {
browserConsoleLogOptions: {level: "debug", format: "%b %T: %m", terminal: true},
autoWatch: false,
concurrency: Infinity,
singleRun: true,
})
}
singleRun: false
});
}

25
package.json
Просмотреть файл

@ -4,13 +4,13 @@
"description": "",
"main": "./src/index.js",
"scripts": {
"build": "webpack --config ./webpack.conf.js --mode production",
"build-debug": "webpack --config ./webpack.conf.js --mode development",
"benchmark": "karma start --browsers ChromeTest --single-run --run_config=$RUN_CONFIG",
"profile": "karma start --browsers ChromeTest --single-run --profile --run_config=$RUN_CONFIG > output.txt && tsc && node utils/gen-chrome-trace < output.txt",
"test-debug": "karma start --browsers ChromeDebug --run_config=$RUN_CONFIG",
"test-edge": "karma start --browsers Edge --single-run --run_config=$RUN_CONFIG",
"test-safari": "karma start --browsers Safari --single-run --run_config=$RUN_CONFIG"
"build": "tsc && webpack --config ./webpack.conf.js --mode production",
"build-debug": "tsc && webpack --config ./webpack.conf.js --mode development",
"benchmark": "cross-var karma start --browsers ChromeTest --single-run --run_config=$npm_config_run_config",
"profile": "cross-var karma start --browsers ChromeTest --single-run --profile --run_config=$npm_config_run_config > output.txt && tsc && node utils/gen-chrome-trace < output.txt",
"test-debug": "cross-var karma start --browsers ChromeDebug --run_config=$npm_config_run_config",
"test-edge": "cross-var karma start --browsers Edge --single-run --run_config=$npm_config_run_config",
"test-safari": "cross-var karma start --browsers Safari --single-run --run_config=$npm_config_run_config"
},
"keywords": [
"benchmark"
@ -18,9 +18,10 @@
"author": "",
"license": "ISC",
"devDependencies": {
"@types/karma": "^6.3.1",
"blueimp-load-image": "^2.19.0",
"karma": "^6.3.2",
"karma-chrome-launcher": "^2.2.0",
"karma-chrome-launcher": "^3.1.0",
"karma-edge-launcher": "^0.4.2",
"karma-mocha": "^2.0.1",
"karma-mocha-reporter": "^2.2.5",
@ -28,18 +29,20 @@
"karma-safari-launcher": "^1.0.0",
"mocha": "^8.3.2",
"requirejs": "^2.3.6",
"webpack": "^4.24.0",
"webpack-cli": "^3.1.2"
"ts-loader": "^8.2.0",
"webpack": "^4.46.0",
"webpack-cli": "^3.3.12"
},
"dependencies": {
"@tensorflow/tfjs": "3.6.0",
"@tensorflow/tfjs-backend-wasm": "3.6.0",
"cross-var": "^1.1.0",
"fsevents": "2.3.2",
"lodash": "^4.17.11",
"ndarray": "^1.0.18",
"ndarray-ops": "^1.2.2",
"onnx-proto": "^4.0.4",
"onnxruntime-web": "file:../onnxruntime/js/web"
"onnxruntime-web": "^1.8.2-dev.20210831.0"
},
"optionalDependencies": {
"fsevents": "2.3.2"

33
sample-configs/run_config_mediapipe_facemesh.json Normal file
Просмотреть файл

@ -0,0 +1,33 @@
{
"benchmarkName": "mediapipe facemesh",
"frameworksToTest": ["tensorflow.js", "ort-web"],
"backendsToTest": ["wasm", "webgl"],
"warmupIteration": 5,
"runIteration": 25,
"logLevel": "error",
"debug": false,
"ortweb": {
"path" : "data/model-onnx/mediapipe-facemesh/model.onnx",
"webgl": {
"pack": true,
"textureCacheMode": "",
"matmulMaxBatchSize": "",
"contextId": ""
},
"wasm": {
"numThreads": 0,
"initTimeout": 0,
"simd": true
}
},
"tfjs": {
"path" : "data/model-tfjs/mediapipe-facemesh/model.json",
"webgl": {
"pack": true
},
"wasm": {
"threading": true,
"simd": true
}
}
}

38
sample-configs/run_config_softmax_test.json
Просмотреть файл

@ -1,30 +1,12 @@
{
"benchmarkName": "onnx-softmax",
"frameworksToTest": ["ort-web"],
"backendsToTest": ["webgl"],
"warmupIteration": 2,
"runIteration": 10,
"inputConfig": {
"runPreprocessor": false,
"customPreprocessorScript": "",
"useSyntheticData": false,
"inputWidth": 5,
"inputHeight": 4,
"inputDepth": 3,
"inputNames": ["x"],
"customInputPaths": [
{ "url": "data/model-onnx/test_softmax_axis_0/test_data_set_0/input_0.pb" }
]
},
"outputConfig": {
"name": "y",
"value": [],
"dims": [],
"customOutputPath": "data/model-onnx/test_softmax_axis_0/test_data_set_0/output_0.pb"
},
"backendsToTest": ["wasm", "webgl"],
"warmupIteration": 5,
"runIteration": 25,
"logLevel": "warning",
"debug": false,
"onnxModelConfig": {
"ortweb": {
"path" : "data/model-onnx/test_softmax_axis_0/model.onnx",
"webgl": {
"pack": true,
@ -33,15 +15,19 @@
"contextId": ""
},
"wasm": {
"numThreads": "",
"initTimeout": ""
"numThreads": 0,
"initTimeout": 0,
"simd": true
}
},
"tfjsModelConfig": {
"tfjs": {
"path" : "",
"webgl": {
"pack": true
},
"wasm": {}
"wasm": {
"threading": true,
"simd": true
}
}
}

41
sample-configs/run_config_super_resolution.json
Просмотреть файл

@ -1,41 +0,0 @@
{
"benchmarkName": "superResolution",
"frameworksToTest": ["ort-web", "tensorflow.js"],
"backendsToTest": ["webgl"],
"warmupIteration": 0,
"runIteration": 2,
"inputConfig": {
"runPreprocessor": true,
"customPreprocessorScript": "",
"useSyntheticData": true,
"inputWidth": 224,
"inputHeight": 224,
"inputDepth": 1,
"inputNames": ["input"],
"customInputPaths": [
{ "url": "https://i.imgur.com/CzXTtJV.jpg" }
]
},
"logLevel": "verbose",
"debug": false,
"onnxModelConfig": {
"path" : "data/model-onnx/super-resolution/super_resolution.onnx",
"webgl": {
"pack": true,
"textureCacheMode": "",
"matmulMaxBatchSize": "",
"contextId": ""
},
"wasm": {
"numThreads": 1,
"initTimeout": 0
}
},
"tfjsModelConfig": {
"path" : "data/model-tfjs/super-resolution/super-res.js",
"webgl": {
"pack": true
},
"wasm": {}
}
}

80
src/benchmark-utils.ts Normal file
Просмотреть файл

@ -0,0 +1,80 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
'use strict';
import {OrtWebBenchmark} from './ort-web-benchmark';
import {TensorFlowBenchmark} from './tfjs-benchmark';
export interface BenchmarkResult {
framework: string;
backend: string;
duration: number;
}
export const readTextFile = async (file: string): Promise<string> => {
var xhr = new XMLHttpRequest();
return new Promise(function(resolve) {
xhr.overrideMimeType("application/json");
xhr.open("GET", file, true);
xhr.onreadystatechange = function() {
if (xhr.readyState === 4 && xhr.status === 200) {
resolve(xhr.responseText);
}
}
xhr.send(null)
});
}
const createBenchmark = (framework: string): TensorFlowBenchmark|OrtWebBenchmark => {
switch (framework) {
case 'tensorflow.js':
return new TensorFlowBenchmark();
case 'ort-web':
return new OrtWebBenchmark();
default:
throw new Error("'tensorflow-js' and 'ort-web' are supported.");
}
}
export const runBenchmark = async (config: any, framework: string, backend: string, profile: boolean): Promise<BenchmarkResult> => {
console.log(`runBenchmark is being called on ${backend} of ${framework}`);
const benchmark = createBenchmark(framework);
await benchmark.init(config, backend, profile);
const durations = [];
console.log(`Running ${config.runIteration} iterations.`);
if (config.warmupIteration && config.warmupIteration > 0) {
console.log("Running warmup...");
for(let i = 0 ; i < config.warmupIteration; i++) {
await benchmark.run();
}
console.log("Warmup done");
}
for (let i = 0 ; i < config.runIteration; i++) {
const start = performance.now();
await benchmark.run();
const duration = performance.now() - start;
console.log(`Duration: ${duration}ms`);
durations.push(duration);
}
if (profile) {
benchmark.endProfiling();
}
durations.shift();
const sum = durations.reduce((a, b) => a + b);
const avg = sum / durations.length;
console.log(`avg duration: ${avg}`);
return {
framework: framework,
backend: backend,
duration: avg
};
}

14
src/benchmark.ts Normal file
Просмотреть файл

@ -0,0 +1,14 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
'use strict';
import * as ort from 'onnxruntime-web';
export const BenchmarkBasePath = '/base';
export interface Benchmark {
init(config: any, backend: string, profile: boolean): Promise<void>;
run(): Promise<any[]|Uint8Array|Float32Array|Int32Array|ort.InferenceSession.OnnxValueMapType>;
endProfiling(): void;
}

558
src/index.js
Просмотреть файл

@ -1,548 +1,42 @@
import * as tf from '@tensorflow/tfjs';
import '@tensorflow/tfjs-backend-wasm';
import loadImage from 'blueimp-load-image';
import ndarray from 'ndarray';
import ops from 'ndarray-ops';
import {onnx as onnxProto} from 'onnx-proto';
import * as onnx from 'onnxruntime-web';
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
import {tensorDataTypeFromProto, tensorDimsFromProto, sizeofProto, readProto, longToNumber} from './index_utils.js'
'use strict';
const SERVER_BASE_PATH = '/base';
const IMAGE_TYPES = ['jpg', 'bmp'];
const PROTO_BUFT_TYPES = ['pb', 'npy'];
const BackendMapping = {
'ort-web' : {
'webgl': 'GPU-webgl',
'wasm': 'wasm',
},
'tensorflow.js': {
'webgl': 'GPU-webgl',
'wasm': 'wasm',
},
}
import {BenchmarkBasePath} from './benchmark';
import {readTextFile, runBenchmark} from './benchmark-utils';
let run_config;
function readTextFile(file) {
var xhr = new XMLHttpRequest();
return new Promise(function(resolve) {
xhr.overrideMimeType("application/json");
xhr.open("GET", file, true);
xhr.onreadystatechange = function() {
if (xhr.readyState === 4 && xhr.status === 200) {
resolve(xhr.responseText);
}
}
xhr.send(null)
});
}
describe('Benchmark', ()=> {
let config;
const results = [];
const profile = __karma__.config.profile;
async function readTextFileAsync(file) {
const result = await readTextFile(file);
run_config = JSON.parse(result);
}
const results = [];
const browser = __karma__.config.browser[0];
const profile = __karma__.config.profile;
if(profile) {
console.log("****Starting profiling****");
}
let runIteration = 10;
class ImageLoader {
constructor(imageWidth, imageHeight) {
this.canvas = document.createElement('canvas');
this.canvas.width = imageWidth;
this.canvas.height = imageHeight;
this.ctx = this.canvas.getContext('2d');
}
async getImageData(url) {
await this.loadImageAsync(url);
const imageData = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
return imageData;
}
loadImageAsync(url) {
return new Promise((resolve, reject)=>{
this.loadImageCb(url, ()=>{
resolve();
});
});
}
loadImageCb(url, cb) {
loadImage(
url,
img => {
if (img.type === 'error') {
throw `Could not load image: ${url}`;
} else {
// load image data onto input canvas
this.ctx.drawImage(img, 0, 0)
window.setTimeout(() => { cb(); }, 0);
}
},
{
maxWidth: this.canvas.width,
maxHeight: this.canvas.height,
cover: true,
crop: true,
canvas: true,
crossOrigin: 'Anonymous'
}
);
}
}
function createBenchmark(name) {
switch (name) {
case 'tensorflow.js': return new TensorFlowBenchmark();
case 'ort-web': return new OrtWebBenchmark();
}
}
function isHttpUrl(string) {
let url;
try {
url = new URL(string);
} catch (_) {
return false;
}
return url.protocol === "http:" || url.protocol === "https:";
}
function fromProto(tensorProto) {
if (!tensorProto) {
throw new Error('cannot construct Value from an empty tensor');
}
const type = tensorDataTypeFromProto(tensorProto.dataType, onnxProto.TensorProto);
const dims = tensorDimsFromProto(tensorProto.dims);
let n = 1;
dims.forEach(element => {
n = n * element;
});
let value = Array(n).fill(0);
if (type === 'string') {
// When it's STRING type, the value should always be stored in field
// 'stringData'
tensorProto.stringData.forEach((str, i) => {
const buf = Buffer.from(str.buffer, str.byteOffset, str.byteLength);
value[i] = buf.toString();
});
} else if (
tensorProto.rawData && typeof tensorProto.rawData.byteLength === 'number' &&
tensorProto.rawData.byteLength > 0) {
// NOT considering segment for now (IMPORTANT)
// populate value from rawData
const dataSource =
new DataView(tensorProto.rawData.buffer, tensorProto.rawData.byteOffset, tensorProto.rawData.byteLength);
const elementSize = sizeofProto(tensorProto.dataType, onnxProto.TensorProto);
const length = tensorProto.rawData.byteLength / elementSize;
if (tensorProto.rawData.byteLength % elementSize !== 0) {
throw new Error('invalid buffer length');
}
if (value.length !== length) {
throw new Error('buffer length mismatch');
}
for (let i = 0; i < length; i++) {
const n = readProto(dataSource, tensorProto.dataType, i * elementSize, onnxProto.TensorProto);
value[i] = n;
}
} else {
// populate value from array
let array;
switch (tensorProto.dataType) {
case onnxProto.TensorProto.DataType.FLOAT:
array = tensorProto.floatData;
break;
case onnxProto.TensorProto.DataType.INT32:
case onnxProto.TensorProto.DataType.INT16:
case onnxProto.TensorProto.DataType.UINT16:
case onnxProto.TensorProto.DataType.INT8:
case onnxProto.TensorProto.DataType.UINT8:
case onnxProto.TensorProto.DataType.BOOL:
array = tensorProto.int32Data;
break;
case onnxProto.TensorProto.DataType.INT64:
array = tensorProto.int64Data;
break;
case onnxProto.TensorProto.DataType.DOUBLE:
array = tensorProto.doubleData;
break;
case onnxProto.TensorProto.DataType.UINT32:
case onnxProto.TensorProto.DataType.UINT64:
array = tensorProto.uint64Data;
break;
default:
// should never run here
throw new Error('unspecific error');
}
if (array === null || array === undefined) {
throw new Error('failed to populate data from a tensorproto value');
}
if (value.length !== array.length) {
throw new Error('array length mismatch');
}
for (let i = 0; i < array.length; i++) {
value[i] = array[i];
}
}
return new onnx.Tensor(type, value, dims);
}
async function runBenchmark(framework, backend) {
console.log(`runBenchmark is being called with ${framework} ${BackendMapping[framework][backend]},
${run_config.inputConfig.inputWidth} x ${run_config.inputConfig.inputHeight}`)
const impl = createBenchmark(framework);
await impl.init(backend);
const imageLoader = new ImageLoader(run_config.inputConfig.inputWidth, run_config.inputConfig.inputHeight);
const durations = [];
let benchmarkInputs = [];
run_config.inputConfig.inputNames.forEach((element, i) => {
if (!run_config.inputConfig.useSyntheticData) {
benchmarkInputs.push(
{
name : element,
url: run_config.inputConfig.customInputPaths[i].url
}
);
}
else {
const arraySize = run_config.inputConfig.inputDepth * run_config.inputConfig.inputHeight * run_config.inputConfig.inputWidth;
benchmarkInputs.push(
{
name : element,
data: Float32Array.from({length: arraySize}, () => Math.floor(Math.random() * 256))
}
);
}
});
for(const input of benchmarkInputs) {
console.log(`Running ${input.name} for ${runIteration} iterations.`)
let inputData = new Object();
inputData.name = input.name;
if(!run_config.inputConfig.useSyntheticData) {
var n = input.url.lastIndexOf('.');
var type = input.url.substring(n + 1);
if(!isHttpUrl(input.url)) {
input.url = `${SERVER_BASE_PATH}/${input.url}`;
}
if(IMAGE_TYPES.indexOf(type) >= 0) {
inputData.data = await imageLoader.getImageData(input.url).data;
}
else if (PROTO_BUFT_TYPES.indexOf(type) == 0) {
const response = await fetch(input.url);
const file = await response.arrayBuffer();
const tensorProto = onnxProto.TensorProto.decode(Buffer.from(file));
inputData.data = fromProto(tensorProto);
}
else {
throw new Error("Unsupprted input type")
}
}
else {
inputData.data = input.data;
}
let expectedOutput = undefined;
if(run_config.outputConfig) {
expectedOutput = new Object();
expectedOutput.name = run_config.outputConfig.name;
if(!run_config.inputConfig.useSyntheticData) {
if(run_config.outputConfig.value && run_config.outputConfig.value.length > 0) {
expectedOutput.data = new onnx.Tensor("float32", value, run_config.outputConfig.dims);
}
else if(run_config.outputConfig.customOutputPath) {
if(!isHttpUrl(run_config.outputConfig.customOutputPath)) {
run_config.outputConfig.customOutputPath = `${SERVER_BASE_PATH}/${run_config.outputConfig.customOutputPath}`;
}
var n = run_config.outputConfig.customOutputPath.lastIndexOf('.');
var type = run_config.outputConfig.customOutputPath.substring(n + 1);
if (PROTO_BUFT_TYPES.indexOf(type) == 0) {
const response = await fetch(run_config.outputConfig.customOutputPath);
const file = await response.arrayBuffer();
const tensorProto = onnxProto.TensorProto.decode(Buffer.from(file));
expectedOutput.data = fromProto(tensorProto);
}
else {
throw new Error(`Unsupprted output type ${type}`);
}
}
}
}
if(run_config.warmupIteration && run_config.warmupIteration > 0) {
console.log("Running warmup...");
for(let i = 0 ; i < run_config.warmupIteration; i++) {
await impl.runModel(inputData, expectedOutput, true/*isWarmup*/);
}
console.log("Warmup done");
}
for(let i = 0 ; i < runIteration; i++) {
const outputData = await impl.runModel(inputData, expectedOutput);
durations.push(impl.duration);
}
}
if(profile) {
impl.endProfiling();
}
durations.shift();
const sum = durations.reduce((a,b)=>a+b);
const avg = sum / durations.length;
console.log(`avg duration: ${avg}`);
return {
framework: framework,
backend: backend,
duration: avg
};
}
class TensorFlowBenchmark {
async init(backend) {
this.inputWidth = run_config.inputConfig.inputWidth;
this.inputHeight = run_config.inputConfig.inputHeight;
this.inputDepth = run_config.inputConfig.inputDepth;
this.isScriptFile = run_config.tfjsModelConfig.path.slice(-3) == '.js';
let model;
let loadingError;
if(this.isScriptFile) {
var loadScript = async function () {
return new Promise((resolve, reject) => {
// Create script element and set attributes
const script = document.createElement('script')
script.type = 'text/javascript'
script.async = false;
script.defer = false;
script.src = `${SERVER_BASE_PATH}/${run_config.tfjsModelConfig.path}`;
script.onload = function() {
model = createModel(tf);
};
// Resolve the promise once the script is loaded
// Catch any errors while loading the script
script.addEventListener('load', () => {
resolve(script)
});
script.addEventListener('error', () => {
reject(new Error(`${SERVER_BASE_PATH}/${run_config.tfjsModelConfig.path} failed to load.`));
});
// Append the script to the DOM
const el = document.getElementsByTagName('script')[0]
el.parentNode.insertBefore(script, el)
});
};
await loadScript();
}
else {
let realPath = isHttpUrl(run_config.tfjsModelConfig.path) ? run_config.tfjsModelConfig.path :
`${SERVER_BASE_PATH}/${run_config.tfjsModelConfig.path}`;
// first try to load it as layers model
try {
model = await tf.loadLayersModel(realPath);
}
catch (e) {
loadingError = e;
model = undefined;
}
// if empty, then try loading as graph model
if(model === undefined && loadingError !== undefined) {
try {
model = await tf.loadGraphModel(realPath);
}
catch (e) {
loadingError = e;
model = undefined;
}
}
}
if(model === undefined) {
if(loadingError !== undefined) {
throw loadingError;
}
throw new Error('Model failed to load, please check model path or script in configuration.');
}
this.pack_texture = run_config.tfjsModelConfig.webgl.pack;
tf.env().set('WEBGL_PACK', this.pack_texture);
console.log(`Tfjs pack mode enabled: ${tf.env().getBool('WEBGL_PACK')}`);
if(backend) {
console.log(`Setting the backend to ${backend}`);
await tf.setBackend(backend);
await tf.ready();
console.log(`Set the backend to ${tf.getBackend()}`);
}
this.model = model;
}
async runModel(data) {
let preprocessedData;
if(run_config.inputConfig.runPreprocessor) {
preprocessedData = this.preprocess(data.data, this.inputWidth, this.inputHeight, this.inputDepth);
}
else {
preprocessedData = data.data
}
const start = performance.now();
if(!this.model && this.isScriptFile) {
this.model = createModel(tf);
}
const output = await this.model.predict(preprocessedData);
const outputData = output.dataSync();
const stop = performance.now();
this.duration = stop - start;
console.log(`Duration:${this.duration}ms`);
return outputData;
}
preprocess(data, width, height, depth) {
// data processing
const dataTensor = ndarray(new Float32Array(data), [width, height, depth])
const dataProcessedTensor = ndarray(new Float32Array(width * height * depth), [1, width, height, depth])
ops.subseq(dataTensor.pick(null, null, 2), 103.939)
ops.subseq(dataTensor.pick(null, null, 1), 116.779)
ops.subseq(dataTensor.pick(null, null, 0), 123.68)
ops.assign(dataProcessedTensor.pick(0, null, null, 0), dataTensor.pick(null, null, 2))
ops.assign(dataProcessedTensor.pick(0, null, null, 1), dataTensor.pick(null, null, 1))
ops.assign(dataProcessedTensor.pick(0, null, null, 2), dataTensor.pick(null, null, 0))
return tf.tensor(dataProcessedTensor.data, dataProcessedTensor.shape);
}
endProfiling() {
}
}
class OrtWebBenchmark {
async init(backend) {
onnx.env.logLevel = profile ? 'verbose' : run_config.logLevel;
if(run_config.onnxModelConfig.webgl.pack) {
onnx.env.webgl.pack = run_config.onnxModelConfig.webgl.pack;
}
if(run_config.onnxModelConfig.webgl.contextId) {
onnx.env.webgl.contextId = run_config.onnxModelConfig.webgl.contextId;
}
if(run_config.onnxModelConfig.wasm.numThreads) {
onnx.env.wasm.numThreads = run_config.onnxModelConfig.wasm.numThreads
}
if(run_config.onnxModelConfig.wasm.initTimeout) {
onnx.env.wasm.initTimeout = run_config.onnxModelConfig.wasm.initTimeout
}
console.log(`ort-web Pack mode enabled: ${onnx.env.webgl.pack}`);
this.inputWidth = run_config.inputConfig.inputWidth;
this.inputHeight = run_config.inputConfig.inputHeight;
this.inputDepth = run_config.inputConfig.inputDepth;
this.modelPath = `${SERVER_BASE_PATH}/${run_config.onnxModelConfig.path}`;
this.session = await onnx.InferenceSession.create(this.modelPath,
{executionProviders: [backend]});
console.log(`Session initialized with: ${backend} backend(s).`);
before('Prepare benchmark', async ()=> {
const configFilePath = `${BenchmarkBasePath}/${__karma__.config.runConfig}`;
const result = await readTextFile(configFilePath);
config = JSON.parse(result);
console.log(`browser: ${__karma__.config.browser[0]}`)
if (profile) {
this.session.startProfiling();
console.log("Start profiling");
}
}
async runModel(data, output, isWarmup) {
let preprocessedData;
if(run_config.inputConfig.runPreprocessor) {
preprocessedData = this.preprocess(data.data, this.inputWidth, this.inputHeight, this.inputDepth);
}
else {
preprocessedData = data.data
}
const start = performance.now();
const outputName = output === undefined ? 'output' : output.name;
const inputName = data.name;
const outputMap = await this.session.run({[inputName]: preprocessedData}, [outputName]);
const outputData = outputMap[this.session.outputNames[0]];
const stop = performance.now();
this.duration = stop - start;
let prefix;
if(isWarmup) {
prefix = 'Warmup duration';
}
else {
prefix = 'Duration';
}
console.log(`${prefix}:${this.duration}ms`);
return outputData;
}
preprocess(data, width, height, depth) {
// data processing
const dataTensor = ndarray(new Float32Array(data), [width, height, depth]);
const dataProcessedTensor = ndarray(new Float32Array(width * height * depth), [1, depth, width, height]);
ops.divseq(dataTensor, 128.0);
ops.subseq(dataTensor, 1.0);
ops.assign(dataProcessedTensor.pick(0, 0, null, null), dataTensor.pick(null, null, 2));
ops.assign(dataProcessedTensor.pick(0, 1, null, null), dataTensor.pick(null, null, 1));
ops.assign(dataProcessedTensor.pick(0, 2, null, null), dataTensor.pick(null, null, 0));
const tensor = new onnx.Tensor('float32', dataProcessedTensor.data, [1, depth, width, height]);
return tensor;
}
endProfiling() {
this.session.endProfiling();
}
}
console.log(`browser: ${browser}`)
describe('Running benchmark', ()=> {
let benchmarksDone = 0;
before('Reading config', async ()=> {
await readTextFileAsync(`${SERVER_BASE_PATH}/` + __karma__.config.runConfig);
runIteration = run_config.runIteration;
console.log(`Starting benchmark: ${run_config.benchmarkName}`);
console.log(`Start benchmark: ${config.benchmarkName}`);
});
it('benchmark run ', async ()=> {
if(benchmarksDone >= run_config.frameworksToTest.length) {
return;
}
for(const backend of run_config.backendsToTest) {
results.push(await runBenchmark(run_config.frameworksToTest[benchmarksDone], backend));
}
await config.frameworksToTest.reduce(async (frameworkPromise, framework) => {
await frameworkPromise;
await config.backendsToTest.reduce(async (backendPromise, backend) => {
await backendPromise;
const result = await runBenchmark(config, framework, backend, profile);
results.push(result);
await new Promise(r => setTimeout(r, 1000));
}, Promise.resolve);
}, Promise.resolve);
});
it('benchmark run ', async ()=> {
if(benchmarksDone >= run_config.frameworksToTest.length) {
return;
}
for(const backend of run_config.backendsToTest) {
results.push(await runBenchmark(run_config.frameworksToTest[benchmarksDone], backend));
}
});
afterEach('Coninuing...', () => {
benchmarksDone++;
});
after('printing results', ()=> {
console.log(JSON.stringify(results));
});
});
});

112
src/index_utils.js
Просмотреть файл

@ -1,112 +0,0 @@
export function tensorDataTypeFromProto(typeProto, TensorProtoLibrary) {
switch (typeProto) {
case TensorProtoLibrary.DataType.INT8:
return 'int8';
case TensorProtoLibrary.DataType.UINT8:
return 'uint8';
case TensorProtoLibrary.DataType.BOOL:
return 'bool';
case TensorProtoLibrary.DataType.INT16:
return 'int16';
case TensorProtoLibrary.DataType.UINT16:
return 'uint16';
case TensorProtoLibrary.DataType.INT32:
return 'int32';
case TensorProtoLibrary.DataType.UINT32:
return 'uint32';
case TensorProtoLibrary.DataType.FLOAT:
return 'float32';
case TensorProtoLibrary.DataType.DOUBLE:
return 'float64';
case TensorProtoLibrary.DataType.STRING:
return 'string';
// For INT64/UINT64, reduce their value to 32-bits.
// Should throw exception when overflow
case TensorProtoLibrary.DataType.INT64:
return 'int32';
case TensorProtoLibrary.DataType.UINT64:
return 'uint32';
default:
throw new Error(`unsupported data type: ${TensorProtoLibrary.DataType[typeProto]}`);
}
}
export function tensorDimsFromProto(dims) {
// get rid of Long type for dims
return dims.map(d => d);
}
export function sizeofProto(type, TensorProtoLibrary) {
switch (type) {
case TensorProtoLibrary.DataType.UINT8:
case TensorProtoLibrary.DataType.INT8:
case TensorProtoLibrary.DataType.BOOL:
return 1;
case TensorProtoLibrary.DataType.UINT16:
case TensorProtoLibrary.DataType.INT16:
return 2;
case TensorProtoLibrary.DataType.FLOAT:
case TensorProtoLibrary.DataType.INT32:
case TensorProtoLibrary.DataType.UINT32:
return 4;
case TensorProtoLibrary.DataType.INT64:
case TensorProtoLibrary.DataType.DOUBLE:
case TensorProtoLibrary.DataType.UINT64:
return 8;
default:
throw new Error(`cannot calculate sizeof() on type ${TensorProtoLibrary.DataType[type]}`);
}
}
// read one value from TensorProto
export function readProto(view, type, byteOffset, TensorProtoLibrary) {
switch (type) {
case TensorProtoLibrary.DataType.BOOL:
case TensorProtoLibrary.DataType.UINT8:
return view.getUint8(byteOffset);
case TensorProtoLibrary.DataType.INT8:
return view.getInt8(byteOffset);
case TensorProtoLibrary.DataType.UINT16:
return view.getUint16(byteOffset, true);
case TensorProtoLibrary.DataType.INT16:
return view.getInt16(byteOffset, true);
case TensorProtoLibrary.DataType.FLOAT:
return view.getFloat32(byteOffset, true);
case TensorProtoLibrary.DataType.INT32:
return view.getInt32(byteOffset, true);
case TensorProtoLibrary.DataType.UINT32:
return view.getUint32(byteOffset, true);
case TensorProtoLibrary.DataType.INT64:
return longToNumber(
Long.fromBits(view.getUint32(byteOffset, true), view.getUint32(byteOffset + 4, true), false), type);
case TensorProtoLibrary.DataType.DOUBLE:
return view.getFloat64(byteOffset, true);
case TensorProtoLibrary.DataType.UINT64:
return longToNumber(
Long.fromBits(view.getUint32(byteOffset, true), view.getUint32(byteOffset + 4, true), true), type);
default:
throw new Error(`cannot read from DataView for type ${TensorProtoLibrary.DataType[type]}`);
}
}
export function longToNumber(i, type, TensorProtoLibrary) {
// INT64, UINT32, UINT64
if (type === TensorProtoLibrary.DataType.INT64) {
if (i.greaterThanOrEqual(2147483648) || i.lessThan(-2147483648)) {
throw new TypeError('int64 is not supported');
}
} else if (
type === TensorProtoLibrary.DataType.UINT32 ||
type === TensorProtoLibrary.DataType.UINT64) {
if (i.greaterThanOrEqual(4294967296) || i.lessThan(0)) {
throw new TypeError('uint64 is not supported');
}
} else {
throw new TypeError(`not a LONG type: ${TensorProtoLibrary.DataType[type]}`);
}
return i.toNumber();
}

171
src/ort-web-benchmark.ts Normal file
Просмотреть файл

@ -0,0 +1,171 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
'use strict';
import {Benchmark, BenchmarkBasePath} from './benchmark';
import Long from 'long';
import {onnx} from 'onnx-proto';
import * as ort from 'onnxruntime-web';
export class OrtWebBenchmark implements Benchmark {
#modelPath: string;
#session: ort.InferenceSession;
#input: ort.SessionHandler.FeedsType;
async init(config: any, backend: string, profile: boolean): Promise<void> {
ort.env.logLevel = profile ? 'verbose' : config.logLevel;
if (config.ortweb.webgl.pack !== undefined) {
ort.env.webgl.pack = config.ortweb.webgl.pack;
}
if (config.ortweb.webgl.contextId !== undefined) {
ort.env.webgl.contextId = config.ortweb.webgl.contextId;
}
if (config.ortweb.wasm.numThreads !== undefined) {
ort.env.wasm.numThreads = config.ortweb.wasm.numThreads
}
if (config.ortweb.wasm.simd !== undefined) {
ort.env.wasm.simd = config.ortweb.wasm.simd
}
if (config.ortweb.wasm.initTimeout !== undefined) {
ort.env.wasm.initTimeout = config.ortweb.wasm.initTimeout
}
console.log(`ort-web Pack mode enabled: ${ort.env.webgl.pack}`);
this.#modelPath= `${BenchmarkBasePath}/${config.ortweb.path}`;
this.#session = await ort.InferenceSession.create(this.#modelPath,
{
executionProviders: [backend],
enableProfiling: profile
});
console.log(`Session initialized with: ${backend} backend(s).`);
if (profile) {
this.#session.startProfiling();
}
this.#input = await generateInputs(this.#modelPath, config.ortweb.shape);
await new Promise(r => setTimeout(r, 3000));
}
async run(): Promise<ort.InferenceSession.OnnxValueMapType> {
const outputData = await this.#session.run(this.#input);
return outputData;
}
endProfiling() {
this.#session.endProfiling();
}
}
interface Metadata {
name: string;
dataType: number;
shape: number[];
}
type ShapeConfig = {[name: string]: number[]};
const generateInputs = async (uri: string, shapeConfig: ShapeConfig): Promise<ort.SessionHandler.FeedsType> => {
const metadata = await loadModel(uri);
let inputs: ort.SessionHandler.FeedsType = {};
metadata.forEach(data => {
let shape;
if (shapeConfig !== undefined && shapeConfig.hasOwnProperty(data.name)) {
shape = shapeConfig[data.name];
} else {
shape = data.shape.map((value, index) => {
if (value <= 0) {
// Only batch size is allowed to set
if (index !== 0) {
throw new Error("Input shape must be manually defined.");
}
return 1;
} else {
return value;
}
});
}
let size = 1;
shape.map(i => size *= i);
inputs[data.name] = generateTensor(data.dataType, size, shape);
});
return inputs;
}
const loadModel = async (uri: string): Promise<Metadata[]> => {
const response = await fetch(uri);
const arrayBuffer = await response.arrayBuffer();
const buffer = new Uint8Array(arrayBuffer);
const modelProto = onnx.ModelProto.decode(buffer);
const graph = modelProto.graph!;
if (!graph.initializer) {
throw new Error("Missing graph initializer");
}
const initializers = new Set<string>();
for (const initializer of graph.initializer!) {
initializers.add(initializer.name!);
}
const metadata: Metadata[] = [];
for (const input of graph.input!) {
if (initializers.has(input.name!)) {
continue;
}
const shape: number[] = input.type!.tensorType!.shape!.dim!.map((d, i) => {
let value = d.dimValue!;
return Long.isLong(value) ? value.toNumber(): value;
});
metadata.push({
name: input.name!,
dataType: input.type!.tensorType!.elemType!,
shape
});
}
return metadata;
}
const generateTensor = (dataType: number, size: number, shape: number[]): ort.Tensor => {
switch (dataType) {
case onnx.TensorProto.DataType.FLOAT:
return new ort.Tensor(Float32Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.DOUBLE:
return new ort.Tensor(Float64Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.UINT8:
return new ort.Tensor(Uint8Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.INT8:
return new ort.Tensor(Int8Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.UINT16:
return new ort.Tensor(Uint16Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.INT16:
return new ort.Tensor(Int16Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.UINT32:
return new ort.Tensor(Uint32Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.INT32:
return new ort.Tensor(Int32Array.from({length: size}, () => 1), shape);
case onnx.TensorProto.DataType.UINT64:
return new ort.Tensor(BigUint64Array.from({length: size}, () => BigInt(1)), shape);
case onnx.TensorProto.DataType.INT64:
return new ort.Tensor(BigInt64Array.from({length: size}, () => BigInt(1)), shape);
case onnx.TensorProto.DataType.STRING:
throw new Error("Can't support STRING tensor");
case onnx.TensorProto.DataType.FLOAT16:
throw new Error("Can't support FLOAT16 tensor");
case onnx.TensorProto.DataType.BFLOAT16:
throw new Error("Can't support BFLOAT16 tensor");
case onnx.TensorProto.DataType.COMPLEX64:
throw new Error("Can't support COMPLEX64 tensor");
case onnx.TensorProto.DataType.COMPLEX128:
throw new Error("Can't support COMPLEX128 tensor");
}
throw new Error("Input tensor type is unknown");
}

136
src/tfjs-benchmark.ts Normal file
Просмотреть файл

@ -0,0 +1,136 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
'use strict';
import * as tf from '@tensorflow/tfjs';
import '@tensorflow/tfjs-backend-wasm';
import {Benchmark, BenchmarkBasePath} from './benchmark';
type TensorflowModelType = tf.GraphModel|tf.LayersModel;
type TensorflowIOType = tf.Tensor<tf.Rank>|tf.Tensor<tf.Rank>[];
type TensorflowExecType = 'execute'|'executeAsync'|'predict'|undefined;
export class TensorFlowBenchmark implements Benchmark {
#model: TensorflowModelType;
#input: TensorflowIOType;
#execType: TensorflowExecType;
async init(config: any, backend: string, profile: boolean): Promise<void> {
let modelPath = isHttpUrl(config.tfjs.path) ? config.tfjs.path : `${BenchmarkBasePath}/${config.tfjs.path}`;
// first try to load it as layers model
try {
this.#model = await tf.loadLayersModel(modelPath);
}
catch (e) {
// then try loading as graph model
this.#model = await tf.loadGraphModel(modelPath);
}
tf.env().set('WEBGL_PACK', !!config.tfjs.webgl.pack);
console.log(`Tfjs pack mode enabled: ${tf.env().getBool('WEBGL_PACK')}`);
console.log(`Setting the backend to ${backend}`);
if (config.tfjs.wasm.threading !== undefined) {
tf.env().set('WASM_HAS_MULTITHREAD_SUPPORT', config.tfjs.wasm.threading);
}
if (config.tfjs.wasm.simd !== undefined) {
tf.env().set('WASM_HAS_SIMD_SUPPORT', config.tfjs.wasm.simd);
}
await tf.setBackend(backend);
await tf.ready().then(() => {
console.log('Set the backend to' + JSON.stringify(tf.getBackend()));
});
this.#input = generateInputs(this.#model, config.tfjs.shape);
this.#execType = await getExecType(this.#model, this.#input);
}
async run(): Promise<any[]|Uint8Array|Float32Array|Int32Array> {
const output = await run(this.#model, this.#input, this.#execType);
let outputData;
if (!Array.isArray(output)) {
outputData = output.dataSync();
} else {
outputData = new Array(output.length);
output.forEach(o => {
outputData.push(o.dataSync());
})
}
return outputData;
}
endProfiling() {}
}
type ShapeConfig = {[name: string]: number[]};
const getExecType = async(model: TensorflowModelType, input: TensorflowIOType): Promise<TensorflowExecType> => {
if (model instanceof tf.GraphModel) {
try {
model.execute(input);
return 'execute';
} catch (e) {
await model.executeAsync(input);
return 'executeAsync';
}
} else if (model instanceof tf.LayersModel) {
model.predict(input);
return 'predict';
} else {
throw new Error(
'Predict function was not found. Please provide a tf.GraphModel or ' +
'tf.LayersModel');
}
}
const run = async(model: TensorflowModelType, input: TensorflowIOType, execType: TensorflowExecType): Promise<TensorflowIOType> => {
switch (execType) {
case 'execute':
return (model as tf.GraphModel).execute(input);
case 'executeAsync':
return await (model as tf.GraphModel).executeAsync(input);
case 'predict':
return (model as tf.LayersModel).predict(input);
default:
throw new Error('Wrong execution type is given: ' + execType)
}
}
const generateInputs = (model: TensorflowModelType, shapeConfig: ShapeConfig): TensorflowIOType => {
const inputs: tf.Tensor<tf.Rank>[] = [];
model.inputs.forEach((node: any) => {
let shape;
if (shapeConfig !== undefined && shapeConfig.hasOwnProperty(node.name)) {
shape = shapeConfig[node.name];
} else {
shape = node.shape!.map((value: number, index: number) => {
if (value === null || value <= 0) {
// Only batch size is allowed to set
if (index !== 0) {
throw new Error("Input shape must be manually defined.");
}
return 1;
} else {
return value;
}
});
}
const tensor = tf.ones(shape, node.dtype);
inputs.push(tensor);
});
return inputs;
}
const isHttpUrl = (path: string): boolean => {
try {
const uri = new URL(path);
return uri.protocol === "http:" || uri.protocol === "https:";
} catch (_) {
return false;
}
}

18
tsconfig.json
Просмотреть файл

@ -1,17 +1,22 @@
{
"include": ["utils"],
"exclude": ["src"],
"include": [
"src/",
"utils/"
],
"exclude": [
"node_modules/"
],
"compileOnSave": true,
"compilerOptions": {
"module": "commonjs",
"moduleResolution": "node",
"esModuleInterop": true,
"target": "es5",
"target": "es2015",
"noImplicitAny": true,
"declaration": true,
"sourceMap": true,
"preserveConstEnums": true,
"lib": ["es2015", "dom"],
"lib": ["es2017", "dom"],
"noUnusedLocals": true,
"noImplicitReturns": true,
"noImplicitThis": true,
@ -22,7 +27,8 @@
"allowUnreachableCode": false,
"experimentalDecorators": true,
"declarationDir": "./types",
"downlevelIteration": true
"downlevelIteration": true,
"skipLibCheck": true
}
}

105
webpack.conf.js
Просмотреть файл

@ -1,3 +1,8 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
'use strict';
const webpack = require("webpack");
const path = require('path');
const fs = require('fs');
@ -8,79 +13,43 @@ if (!fs.existsSync('dist')){
fs.mkdirSync('dist');
}
if(!process.env.ORT_BASE) {
//TODO set ort-web base to the official release in node_modules
//process.env.ORT_BASE = './node_modules/onnxruntime-web';
throw new Error("ORT_BASE variable is not set!");
if (fs.existsSync('./node_modules/onnxruntime-common')) {
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-common/dist', 'ort-common.min.js'), path.resolve('./dist', 'ort-common.min.js'));
} else {
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/node_modules/onnxruntime-common/dist', 'ort-common.min.js'), path.resolve('./dist', 'ort-common.min.js'));
}
const ort_wasm_path = path.resolve(process.env.ORT_BASE, "./js/web/dist/ort-wasm.wasm");
const ort_wasm_simd_path = path.resolve(process.env.ORT_BASE, "./js/web/dist/ort-wasm-simd.wasm");
const ort_wasm_threaded_path = path.resolve(process.env.ORT_BASE, "./js/web/dist/ort-wasm-threaded.wasm");
const ort_wasm_threaded_simd_path = path.resolve(process.env.ORT_BASE, "./js/web/dist/ort-wasm-simd-threaded.wasm");
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/dist', 'ort-web.min.js'), path.resolve('./dist', 'ort-web.min.js'));
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/dist', 'ort-wasm-threaded.js'), path.resolve('./dist', 'ort-wasm-threaded.js'));
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/dist', 'ort-wasm-threaded.worker.js'), path.resolve('./dist', 'ort-wasm-threaded.worker.js'));
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/dist', 'ort-wasm.wasm'), path.resolve('./dist', 'ort-wasm.wasm'));
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/dist', 'ort-wasm-simd.wasm'), path.resolve('./dist', 'ort-wasm-simd.wasm'));
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/dist', 'ort-wasm-threaded.wasm'), path.resolve('./dist', 'ort-wasm-threaded.wasm'));
fs.copyFileSync(path.resolve('./node_modules/onnxruntime-web/dist', 'ort-wasm-simd-threaded.wasm'), path.resolve('./dist', 'ort-wasm-simd-threaded.wasm'));
if (!fs.existsSync(ort_wasm_path)){
console.log(`${ort_wasm_path} does not exist. Build will continue without benchmarking support for wasm.`);
}
else {
fs.createReadStream(ort_wasm_path).pipe(fs.createWriteStream('dist/ort-wasm.wasm'));
}
if (!fs.existsSync(ort_wasm_simd_path)){
console.log(`${ort_wasm_simd_path} does not exist. Build will continue without benchmarking support for simd wasm.`);
}
else {
fs.createReadStream(ort_wasm_simd_path).pipe(fs.createWriteStream('dist/ort-wasm-simd.wasm'));
}
if (!fs.existsSync(ort_wasm_threaded_path)){
console.log(`${ort_wasm_threaded_path} does not exist. Build will continue without benchmarking support for threaded wasm.`);
}
else {
fs.createReadStream(ort_wasm_threaded_path).pipe(fs.createWriteStream('dist/ort-wasm-threaded.wasm'));
}
if (!fs.existsSync(ort_wasm_threaded_simd_path)){
console.log(`${ort_wasm_threaded_simd_path} does not exist. Build will continue without benchmarking support for threaded simd wasm.`);
}
else {
fs.createReadStream(ort_wasm_threaded_simd_path).pipe(fs.createWriteStream('dist/ort-wasm-simd-threaded.wasm'));
}
const tfjs_wasm_path = path.resolve(__dirname, './node_modules/@tensorflow/tfjs-backend-wasm/wasm-out/tfjs-backend-wasm.wasm');
const tfjs_wasm_simd_path = path.resolve(__dirname, './node_modules/@tensorflow/tfjs-backend-wasm/wasm-out/tfjs-backend-wasm-simd.wasm');
const tfjs_wasm_threaded_simd_path = path.resolve(__dirname, './node_modules/@tensorflow/tfjs-backend-wasm/wasm-out/tfjs-backend-wasm-threaded-simd.wasm');
if (!fs.existsSync(tfjs_wasm_path)){
console.log(`${tfjs_wasm_path} does not exist. Build will continue without benchmarking support for wasm.`);
}
else {
fs.createReadStream(tfjs_wasm_path).pipe(fs.createWriteStream('dist/tfjs-backend-wasm.wasm'));
}
if (!fs.existsSync(tfjs_wasm_simd_path)){
console.log(`${tfjs_wasm_simd_path} does not exist. Build will continue without benchmarking support for wasm.`);
}
else {
fs.createReadStream(tfjs_wasm_simd_path).pipe(fs.createWriteStream('dist/tfjs-backend-wasm-simd.wasm'));
}
if (!fs.existsSync(tfjs_wasm_threaded_simd_path)){
console.log(`${tfjs_wasm_threaded_simd_path} does not exist. Build will continue without benchmarking support for wasm.`);
}
else {
fs.createReadStream(tfjs_wasm_threaded_simd_path).pipe(fs.createWriteStream('dist/tfjs-backend-wasm-threaded-simd.wasm'));
}
fs.copyFileSync(path.resolve('./node_modules/@tensorflow/tfjs/dist', 'tf.min.js'), path.resolve('./dist', 'tf.min.js'));
fs.copyFileSync(path.resolve('./node_modules/@tensorflow/tfjs-backend-wasm/dist', 'tf-backend-wasm.min.js'), path.resolve('./dist', 'tf-backend-wasm.min.js'));
fs.copyFileSync(path.resolve('./node_modules/@tensorflow/tfjs-backend-wasm/dist', 'tfjs-backend-wasm.wasm'), path.resolve('./dist', 'tfjs-backend-wasm.wasm'));
fs.copyFileSync(path.resolve('./node_modules/@tensorflow/tfjs-backend-wasm/dist', 'tfjs-backend-wasm-simd.wasm'), path.resolve('./dist', 'tfjs-backend-wasm-simd.wasm'));
fs.copyFileSync(path.resolve('./node_modules/@tensorflow/tfjs-backend-wasm/dist', 'tfjs-backend-wasm-threaded-simd.wasm'), path.resolve('./dist', 'tfjs-backend-wasm-threaded-simd.wasm'));
module.exports = (env, argv) => {
const config = {
entry: [APP_DIR + "/index.js", APP_DIR + "/index_utils.js"],
entry: [APP_DIR + "/index.js"],
output : {
path : DIST_DIR,
filename: "main.js"
},
node: {fs: 'empty'},
resolve: {
extensions: ['.js', '.ts'],
node: {
fs: 'empty'
},
plugins: [new webpack.WatchIgnorePlugin([/\.js$/, /\.d\.ts$/])],
resolve: {
extensions: ['.js', '.ts']
},
externals: {
},
plugins: [
new webpack.WatchIgnorePlugin([/\.js$/, /\.d\.ts$/])
],
module: {
rules: [
{
@ -93,8 +62,10 @@ module.exports = (env, argv) => {
],
},
{
test: /\.tsx?$/,
loader: 'ts-loader'}
test: /\.ts$/,
exclude: /node_modules/,
loader: 'ts-loader'
}
],
}, };
if (argv.mode === 'production') {
@ -102,7 +73,7 @@ module.exports = (env, argv) => {
config.devtool = 'source-map';
} else {
config.mode = 'development';
config.devtool = 'eval-source-map';
config.devtool = 'inline-source-map';
}
return config;
};
};