Граф коммитов

6 Коммитов

Автор SHA1 Сообщение Дата
guoshzhao 7595d79434
Runner: Add Feature - Generate summarized output files. (#157)
**Description**
Generate the summarized output files from all nodes. For each metric, do the reduce operation according to the `reduce_op`

**Major Revision**
- Generate the summarized json file per node:
For microbenchmark, the format is `{benchmark_name}/[{run_count}/]{metric_name}[:rank]`
For modelbenchmark, the format is `{benchmark_name}/{sub_benchmark_name}/[{run_count}/]{metric_name}`
`[]` means optional.
```
{
  "kernel-launch/overhead_event:0": 0.00583,
  "kernel-launch/overhead_event:1": 0.00545,
  "kernel-launch/overhead_event:2": 0.00581,
  "kernel-launch/overhead_event:3": 0.00572,
  "kernel-launch/overhead_event:4": 0.00559,
  "kernel-launch/overhead_event:5": 0.00591,
  "kernel-launch/overhead_event:6": 0.00562,
  "kernel-launch/overhead_event:7": 0.00586,
  "resnet_models/pytorch-resnet50/steptime-train-float32": 544.0827468410134,
  "resnet_models/pytorch-resnet50/throughput-train-float32": 353.7607016465773,
  "resnet_models/pytorch-resnet50/steptime-train-float16": 425.40482617914677,
  "resnet_models/pytorch-resnet50/throughput-train-float16": 454.0142363793973,
  "pytorch-sharding-matmul/0/allreduce": 10.561786651611328,
  "pytorch-sharding-matmul/1/allreduce": 10.561786651611328,
  "pytorch-sharding-matmul/0/allgather": 10.088025093078613,
  "pytorch-sharding-matmul/1/allgather": 10.088025093078613
}
```
- Generate the summarized jsonl file for all nodes, each line is the result from one node in json format.
2021-08-20 16:48:40 +08:00
guoshzhao e41b1f6225
Benchmarks: Add Feature - Add reduce function support for output summary. (#147)
**Description**
Add reduce function support for output summary.

**Major Revision**
- Add reducer class to maintain all reduce functions.
- Save reduce type of each metric into `BenchmarkResult`
- Fix UT.
2021-08-05 16:52:49 +08:00
guoshzhao 216c5b5c71
Benchmarks: Add Feature - Add DistributedImpl and DistributedBackend arguments for micro benchmark. (#100) 2021-06-21 23:34:05 +08:00
Yifan Xiong 57ce473a02
Utils - Support lazy import (#67)
__Major Revision__

* Support lazy import.
* Not importing benchmarks when running `help`, `version`, `deploy` commands, etc.
2021-05-11 10:49:22 +08:00
guoshzhao 8d24d03d9a
Benchmarks: Code Revision - Move benchmarks auto-registration from registry.py to __init__.py (#24)
* move benchmarks registration from registry.py to __init__.py

* revise __init__.

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>
2021-03-22 23:27:21 +08:00
guoshzhao 4c87a3e419
Benchmarks: Initialization - Add base class, registry, and result (#1)
* benchmarks init.

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>
2021-02-24 12:43:24 +08:00