Swinging some of the parallelization over to HTKDeserializers
This commit is contained in:
Mark Hillebrand
Родитель
fbb8687224
Коммит
05db810634
|
|
@ -78,7 +78,7 @@ speechTrain = [
|
|||
]
|
||||
]
|
||||
reader = [
|
||||
readerType = "HTKMLFReader"
|
||||
readerType = "HTKDeserializers"
|
||||
readMethod = "blockRandomize"
|
||||
miniBatchMode = "partial"
|
||||
randomize = "auto"
|
||||
|
|
@ -99,7 +99,7 @@ speechTrain = [
|
|||
]
|
||||
]
|
||||
cvreader = [
|
||||
readerType = "HTKMLFReader"
|
||||
readerType = "HTKDeserializers"
|
||||
readMethod = "blockRandomize"
|
||||
miniBatchMode = "partial"
|
||||
randomize = "auto"
|
||||
|
|
|
|||
|
|
@ -78,7 +78,7 @@ speechTrain = [
|
|||
]
|
||||
]
|
||||
reader = [
|
||||
readerType = "HTKMLFReader"
|
||||
readerType = "HTKDeserializers"
|
||||
readMethod = "blockRandomize"
|
||||
miniBatchMode = "partial"
|
||||
randomize = "auto"
|
||||
|
|
|
|||
|
|
@ -1,438 +0,0 @@
|
|||
CPU info:
|
||||
CPU Model Name: Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
|
||||
Hardware threads: 4
|
||||
Total Memory: 12580404 kB
|
||||
-------------------------------------------------------------------
|
||||
=== Running C:\Program Files\Microsoft MPI\Bin\/mpiexec.exe -n 3 C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:43:27
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (1) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:43:27
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (2) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:43:28
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (0) are in (participating)
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr_speechTrain.logrank0
|
||||
MPI Rank 0: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:43:28
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Using 1 CPU threads.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:28: ##############################################################################
|
||||
MPI Rank 0: 12/15/2016 08:43:28: # #
|
||||
MPI Rank 0: 12/15/2016 08:43:28: # speechTrain command (train action) #
|
||||
MPI Rank 0: 12/15/2016 08:43:28: # #
|
||||
MPI Rank 0: 12/15/2016 08:43:28: ##############################################################################
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:28:
|
||||
MPI Rank 0: Creating virgin network.
|
||||
MPI Rank 0: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 0: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 0: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 0: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 0: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 0: label set 0: 129 classes
|
||||
MPI Rank 0: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 0: 12/15/2016 08:43:28:
|
||||
MPI Rank 0: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { H1 : [512 x 1 x *]
|
||||
MPI Rank 0: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 0: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 0: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 0: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 0: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { H2 : [512 x 1 x *]
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { HLast : [132 x 1 x *]
|
||||
MPI Rank 0: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Initializing dataParallelSGD for 1-bit quantization.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:28: MeanOfFeatures = Mean()
|
||||
MPI Rank 0: 12/15/2016 08:43:28: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 0: 12/15/2016 08:43:28: Prior = Mean()
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:31: Precomputing --> Completed.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:32: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:32: Starting minibatch loop.
|
||||
MPI Rank 0: 12/15/2016 08:43:32: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2935s; samplesPerSecond = 2180.9
|
||||
MPI Rank 0: 12/15/2016 08:43:32: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2502s; samplesPerSecond = 2557.8
|
||||
MPI Rank 0: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2488s; samplesPerSecond = 2572.3
|
||||
MPI Rank 0: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2510s; samplesPerSecond = 2550.0
|
||||
MPI Rank 0: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2515s; samplesPerSecond = 2544.9
|
||||
MPI Rank 0: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2520s; samplesPerSecond = 2539.6
|
||||
MPI Rank 0: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2508s; samplesPerSecond = 2552.3
|
||||
MPI Rank 0: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2526s; samplesPerSecond = 2533.3
|
||||
MPI Rank 0: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2518s; samplesPerSecond = 2541.2
|
||||
MPI Rank 0: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2526s; samplesPerSecond = 2534.1
|
||||
MPI Rank 0: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2518s; samplesPerSecond = 2541.8
|
||||
MPI Rank 0: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2514s; samplesPerSecond = 2545.8
|
||||
MPI Rank 0: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2504s; samplesPerSecond = 2556.4
|
||||
MPI Rank 0: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2733s; samplesPerSecond = 2341.6
|
||||
MPI Rank 0: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2691s; samplesPerSecond = 2378.0
|
||||
MPI Rank 0: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2697s; samplesPerSecond = 2372.6
|
||||
MPI Rank 0: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2651s; samplesPerSecond = 2414.3
|
||||
MPI Rank 0: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2580s; samplesPerSecond = 2480.4
|
||||
MPI Rank 0: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2583s; samplesPerSecond = 2477.8
|
||||
MPI Rank 0: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2602s; samplesPerSecond = 2459.3
|
||||
MPI Rank 0: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2686s; samplesPerSecond = 2382.6
|
||||
MPI Rank 0: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2696s; samplesPerSecond = 2374.2
|
||||
MPI Rank 0: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2573s; samplesPerSecond = 2487.5
|
||||
MPI Rank 0: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2532s; samplesPerSecond = 2528.1
|
||||
MPI Rank 0: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2533s; samplesPerSecond = 2526.8
|
||||
MPI Rank 0: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2547s; samplesPerSecond = 2512.3
|
||||
MPI Rank 0: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2627s; samplesPerSecond = 2436.6
|
||||
MPI Rank 0: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2555s; samplesPerSecond = 2504.4
|
||||
MPI Rank 0: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2635s; samplesPerSecond = 2428.4
|
||||
MPI Rank 0: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2645s; samplesPerSecond = 2419.2
|
||||
MPI Rank 0: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2686s; samplesPerSecond = 2382.5
|
||||
MPI Rank 0: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2491s; samplesPerSecond = 2568.9
|
||||
MPI Rank 0: 12/15/2016 08:43:40: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.29563s
|
||||
MPI Rank 0: 12/15/2016 08:43:40: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/models/cntkSpeech.dnn.1'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:40: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:40: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 1), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:43:41: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.20280589 * 2560; EvalClassificationError = 0.60234375 * 2560; time = 0.5844s; samplesPerSecond = 4380.2
|
||||
MPI Rank 0: 12/15/2016 08:43:41: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.16401892 * 2560; EvalClassificationError = 0.56992188 * 2560; time = 0.5734s; samplesPerSecond = 4464.6
|
||||
MPI Rank 0: 12/15/2016 08:43:42: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.10520889 * 2560; EvalClassificationError = 0.56640625 * 2560; time = 0.5672s; samplesPerSecond = 4513.7
|
||||
MPI Rank 0: 12/15/2016 08:43:42: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.07595031 * 2560; EvalClassificationError = 0.56875000 * 2560; time = 0.5632s; samplesPerSecond = 4545.8
|
||||
MPI Rank 0: 12/15/2016 08:43:43: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.09291426 * 2560; EvalClassificationError = 0.57148438 * 2560; time = 0.5727s; samplesPerSecond = 4470.1
|
||||
MPI Rank 0: 12/15/2016 08:43:44: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.02267717 * 2560; EvalClassificationError = 0.55585938 * 2560; time = 0.5532s; samplesPerSecond = 4627.9
|
||||
MPI Rank 0: 12/15/2016 08:43:44: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.00026902 * 2560; EvalClassificationError = 0.54492188 * 2560; time = 0.5579s; samplesPerSecond = 4588.9
|
||||
MPI Rank 0: 12/15/2016 08:43:45: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.00974791 * 2560; EvalClassificationError = 0.55820313 * 2560; time = 0.5667s; samplesPerSecond = 4517.4
|
||||
MPI Rank 0: 12/15/2016 08:43:45: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.08419905 * 20480; EvalClassificationError = 0.56723633 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=4.56497s
|
||||
MPI Rank 0: 12/15/2016 08:43:45: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/models/cntkSpeech.dnn.2'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:45: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:45: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 1), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:43:46: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.96836606 * 10240; EvalClassificationError = 0.53740234 * 10240; time = 1.4889s; samplesPerSecond = 6877.4
|
||||
MPI Rank 0: 12/15/2016 08:43:48: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.98822385 * 10240; EvalClassificationError = 0.55458984 * 10240; time = 1.4599s; samplesPerSecond = 7014.1
|
||||
MPI Rank 0: 12/15/2016 08:43:48: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.97829495 * 20480; EvalClassificationError = 0.54599609 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.975s
|
||||
MPI Rank 0: 12/15/2016 08:43:48: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/models/cntkSpeech.dnn'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:48: Action "train" complete.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:43:48: __COMPLETED__
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr_speechTrain.logrank1
|
||||
MPI Rank 1: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:43:27
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Using 1 CPU threads.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:28: ##############################################################################
|
||||
MPI Rank 1: 12/15/2016 08:43:28: # #
|
||||
MPI Rank 1: 12/15/2016 08:43:28: # speechTrain command (train action) #
|
||||
MPI Rank 1: 12/15/2016 08:43:28: # #
|
||||
MPI Rank 1: 12/15/2016 08:43:28: ##############################################################################
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:28:
|
||||
MPI Rank 1: Creating virgin network.
|
||||
MPI Rank 1: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 1: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 1: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 1: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 1: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 1: label set 0: 129 classes
|
||||
MPI Rank 1: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 1: 12/15/2016 08:43:28:
|
||||
MPI Rank 1: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: { H2 : [512 x 1 x *]
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { HLast : [132 x 1 x *]
|
||||
MPI Rank 1: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 1: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 1: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 1: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { H1 : [512 x 1 x *]
|
||||
MPI Rank 1: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 1: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Initializing dataParallelSGD for 1-bit quantization.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:28: MeanOfFeatures = Mean()
|
||||
MPI Rank 1: 12/15/2016 08:43:28: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 1: 12/15/2016 08:43:28: Prior = Mean()
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:31: Precomputing --> Completed.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:32: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:32: Starting minibatch loop.
|
||||
MPI Rank 1: 12/15/2016 08:43:32: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2766s; samplesPerSecond = 2314.2
|
||||
MPI Rank 1: 12/15/2016 08:43:32: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2508s; samplesPerSecond = 2552.1
|
||||
MPI Rank 1: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2515s; samplesPerSecond = 2545.1
|
||||
MPI Rank 1: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2519s; samplesPerSecond = 2541.0
|
||||
MPI Rank 1: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2515s; samplesPerSecond = 2544.3
|
||||
MPI Rank 1: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2520s; samplesPerSecond = 2539.8
|
||||
MPI Rank 1: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2483s; samplesPerSecond = 2577.4
|
||||
MPI Rank 1: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2529s; samplesPerSecond = 2531.1
|
||||
MPI Rank 1: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2534s; samplesPerSecond = 2525.4
|
||||
MPI Rank 1: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2519s; samplesPerSecond = 2541.0
|
||||
MPI Rank 1: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2508s; samplesPerSecond = 2551.7
|
||||
MPI Rank 1: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2587s; samplesPerSecond = 2474.2
|
||||
MPI Rank 1: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2536s; samplesPerSecond = 2523.3
|
||||
MPI Rank 1: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2777s; samplesPerSecond = 2304.7
|
||||
MPI Rank 1: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2543s; samplesPerSecond = 2516.6
|
||||
MPI Rank 1: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2608s; samplesPerSecond = 2454.2
|
||||
MPI Rank 1: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2635s; samplesPerSecond = 2429.0
|
||||
MPI Rank 1: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2577s; samplesPerSecond = 2483.7
|
||||
MPI Rank 1: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2585s; samplesPerSecond = 2475.9
|
||||
MPI Rank 1: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2595s; samplesPerSecond = 2466.8
|
||||
MPI Rank 1: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2588s; samplesPerSecond = 2472.5
|
||||
MPI Rank 1: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2557s; samplesPerSecond = 2503.4
|
||||
MPI Rank 1: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2604s; samplesPerSecond = 2457.8
|
||||
MPI Rank 1: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2561s; samplesPerSecond = 2499.3
|
||||
MPI Rank 1: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2628s; samplesPerSecond = 2435.0
|
||||
MPI Rank 1: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2638s; samplesPerSecond = 2426.1
|
||||
MPI Rank 1: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2621s; samplesPerSecond = 2441.8
|
||||
MPI Rank 1: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2566s; samplesPerSecond = 2494.5
|
||||
MPI Rank 1: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2629s; samplesPerSecond = 2434.2
|
||||
MPI Rank 1: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2545s; samplesPerSecond = 2514.8
|
||||
MPI Rank 1: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2542s; samplesPerSecond = 2518.1
|
||||
MPI Rank 1: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2584s; samplesPerSecond = 2477.0
|
||||
MPI Rank 1: 12/15/2016 08:43:40: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.255s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:40: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:40: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 1), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:43:41: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.20280589 * 2560; EvalClassificationError = 0.60234375 * 2560; time = 0.5976s; samplesPerSecond = 4283.8
|
||||
MPI Rank 1: 12/15/2016 08:43:41: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.16401892 * 2560; EvalClassificationError = 0.56992188 * 2560; time = 0.5734s; samplesPerSecond = 4464.3
|
||||
MPI Rank 1: 12/15/2016 08:43:42: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.10520889 * 2560; EvalClassificationError = 0.56640625 * 2560; time = 0.5670s; samplesPerSecond = 4514.6
|
||||
MPI Rank 1: 12/15/2016 08:43:42: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.07595031 * 2560; EvalClassificationError = 0.56875000 * 2560; time = 0.5633s; samplesPerSecond = 4544.9
|
||||
MPI Rank 1: 12/15/2016 08:43:43: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.09291426 * 2560; EvalClassificationError = 0.57148438 * 2560; time = 0.5726s; samplesPerSecond = 4471.1
|
||||
MPI Rank 1: 12/15/2016 08:43:44: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.02267717 * 2560; EvalClassificationError = 0.55585938 * 2560; time = 0.5532s; samplesPerSecond = 4627.9
|
||||
MPI Rank 1: 12/15/2016 08:43:44: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.00026902 * 2560; EvalClassificationError = 0.54492188 * 2560; time = 0.5580s; samplesPerSecond = 4587.5
|
||||
MPI Rank 1: 12/15/2016 08:43:45: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.00974791 * 2560; EvalClassificationError = 0.55820313 * 2560; time = 0.5667s; samplesPerSecond = 4517.5
|
||||
MPI Rank 1: 12/15/2016 08:43:45: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.08419905 * 20480; EvalClassificationError = 0.56723633 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=4.57669s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:45: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:45: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 1), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:43:46: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.96836606 * 10240; EvalClassificationError = 0.53740234 * 10240; time = 1.4927s; samplesPerSecond = 6859.8
|
||||
MPI Rank 1: 12/15/2016 08:43:48: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.98822385 * 10240; EvalClassificationError = 0.55458984 * 10240; time = 1.4597s; samplesPerSecond = 7015.3
|
||||
MPI Rank 1: 12/15/2016 08:43:48: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.97829495 * 20480; EvalClassificationError = 0.54599609 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.97679s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:48: Action "train" complete.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:43:48: __COMPLETED__
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr_speechTrain.logrank2
|
||||
MPI Rank 2: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:43:27
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_Parallel1BitQuantization@release_cpu/stderr
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Using 1 CPU threads.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:29: ##############################################################################
|
||||
MPI Rank 2: 12/15/2016 08:43:29: # #
|
||||
MPI Rank 2: 12/15/2016 08:43:29: # speechTrain command (train action) #
|
||||
MPI Rank 2: 12/15/2016 08:43:29: # #
|
||||
MPI Rank 2: 12/15/2016 08:43:29: ##############################################################################
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:29:
|
||||
MPI Rank 2: Creating virgin network.
|
||||
MPI Rank 2: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 2: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 2: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 2: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 2: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 2: label set 0: 129 classes
|
||||
MPI Rank 2: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 2: 12/15/2016 08:43:29:
|
||||
MPI Rank 2: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 2: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 2: { H2 : [512 x 1 x *]
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 2: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { HLast : [132 x 1 x *]
|
||||
MPI Rank 2: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 2: { H1 : [512 x 1 x *]
|
||||
MPI Rank 2: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Initializing dataParallelSGD for 1-bit quantization.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:29: MeanOfFeatures = Mean()
|
||||
MPI Rank 2: 12/15/2016 08:43:29: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 2: 12/15/2016 08:43:29: Prior = Mean()
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:32: Precomputing --> Completed.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:32: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:32: Starting minibatch loop.
|
||||
MPI Rank 2: 12/15/2016 08:43:32: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2979s; samplesPerSecond = 2148.6
|
||||
MPI Rank 2: 12/15/2016 08:43:32: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2500s; samplesPerSecond = 2559.5
|
||||
MPI Rank 2: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2493s; samplesPerSecond = 2567.7
|
||||
MPI Rank 2: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2524s; samplesPerSecond = 2535.9
|
||||
MPI Rank 2: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2519s; samplesPerSecond = 2540.5
|
||||
MPI Rank 2: 12/15/2016 08:43:33: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2524s; samplesPerSecond = 2535.7
|
||||
MPI Rank 2: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2509s; samplesPerSecond = 2551.1
|
||||
MPI Rank 2: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2536s; samplesPerSecond = 2523.7
|
||||
MPI Rank 2: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2564s; samplesPerSecond = 2496.2
|
||||
MPI Rank 2: 12/15/2016 08:43:34: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2523s; samplesPerSecond = 2536.5
|
||||
MPI Rank 2: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2508s; samplesPerSecond = 2551.4
|
||||
MPI Rank 2: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2539s; samplesPerSecond = 2520.8
|
||||
MPI Rank 2: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2521s; samplesPerSecond = 2538.6
|
||||
MPI Rank 2: 12/15/2016 08:43:35: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2586s; samplesPerSecond = 2474.7
|
||||
MPI Rank 2: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2692s; samplesPerSecond = 2377.6
|
||||
MPI Rank 2: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2642s; samplesPerSecond = 2422.4
|
||||
MPI Rank 2: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2559s; samplesPerSecond = 2500.5
|
||||
MPI Rank 2: 12/15/2016 08:43:36: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2514s; samplesPerSecond = 2545.5
|
||||
MPI Rank 2: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2543s; samplesPerSecond = 2516.6
|
||||
MPI Rank 2: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2587s; samplesPerSecond = 2474.3
|
||||
MPI Rank 2: 12/15/2016 08:43:37: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2696s; samplesPerSecond = 2374.1
|
||||
MPI Rank 2: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2697s; samplesPerSecond = 2373.3
|
||||
MPI Rank 2: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2606s; samplesPerSecond = 2455.6
|
||||
MPI Rank 2: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2574s; samplesPerSecond = 2486.1
|
||||
MPI Rank 2: 12/15/2016 08:43:38: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2608s; samplesPerSecond = 2454.2
|
||||
MPI Rank 2: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2607s; samplesPerSecond = 2455.2
|
||||
MPI Rank 2: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2525s; samplesPerSecond = 2534.4
|
||||
MPI Rank 2: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2501s; samplesPerSecond = 2559.3
|
||||
MPI Rank 2: 12/15/2016 08:43:39: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2584s; samplesPerSecond = 2476.8
|
||||
MPI Rank 2: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2644s; samplesPerSecond = 2420.6
|
||||
MPI Rank 2: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2678s; samplesPerSecond = 2390.1
|
||||
MPI Rank 2: 12/15/2016 08:43:40: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2574s; samplesPerSecond = 2486.5
|
||||
MPI Rank 2: 12/15/2016 08:43:40: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.27847s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:40: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:40: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 1), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:43:41: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.20280589 * 2560; EvalClassificationError = 0.60234375 * 2560; time = 0.5974s; samplesPerSecond = 4285.5
|
||||
MPI Rank 2: 12/15/2016 08:43:41: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.16401892 * 2560; EvalClassificationError = 0.56992188 * 2560; time = 0.5734s; samplesPerSecond = 4464.3
|
||||
MPI Rank 2: 12/15/2016 08:43:42: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.10520889 * 2560; EvalClassificationError = 0.56640625 * 2560; time = 0.5671s; samplesPerSecond = 4514.4
|
||||
MPI Rank 2: 12/15/2016 08:43:42: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.07595031 * 2560; EvalClassificationError = 0.56875000 * 2560; time = 0.5631s; samplesPerSecond = 4546.0
|
||||
MPI Rank 2: 12/15/2016 08:43:43: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.09291426 * 2560; EvalClassificationError = 0.57148438 * 2560; time = 0.5726s; samplesPerSecond = 4471.0
|
||||
MPI Rank 2: 12/15/2016 08:43:44: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.02267717 * 2560; EvalClassificationError = 0.55585938 * 2560; time = 0.5530s; samplesPerSecond = 4629.2
|
||||
MPI Rank 2: 12/15/2016 08:43:44: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.00026902 * 2560; EvalClassificationError = 0.54492188 * 2560; time = 0.5580s; samplesPerSecond = 4587.5
|
||||
MPI Rank 2: 12/15/2016 08:43:45: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.00974791 * 2560; EvalClassificationError = 0.55820313 * 2560; time = 0.5666s; samplesPerSecond = 4518.0
|
||||
MPI Rank 2: 12/15/2016 08:43:45: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.08419905 * 20480; EvalClassificationError = 0.56723633 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=4.5763s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:45: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:45: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 1), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:43:46: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.96836606 * 10240; EvalClassificationError = 0.53740234 * 10240; time = 1.4771s; samplesPerSecond = 6932.6
|
||||
MPI Rank 2: 12/15/2016 08:43:48: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.98822385 * 10240; EvalClassificationError = 0.55458984 * 10240; time = 1.4597s; samplesPerSecond = 7015.3
|
||||
MPI Rank 2: 12/15/2016 08:43:48: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.97829495 * 20480; EvalClassificationError = 0.54599609 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.96064s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:48: Action "train" complete.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:43:48: __COMPLETED__
|
||||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
|
|
@ -1,23 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
. $TEST_ROOT_DIR/run-test-common
|
||||
|
||||
OriginalTestDir=../../../DNN/Parallel1BitQuantization
|
||||
ConfigDir=$TEST_DIR/../../../DNN
|
||||
LogFileName=stderr
|
||||
Instances=3
|
||||
NumCPUThreads=$(threadsPerInstance $Instances)
|
||||
|
||||
(cd $TEST_DIR/$OriginalTestDir && md5sum baseline*) | (cd $TEST_DIR && md5sum --status -c -)
|
||||
if [ $? != 0 ]; then
|
||||
echo Error: Baselines must match original test. Copy from $OriginalTestDir.
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# cntkmpirun <MPI args> <CNTK config file name> <additional CNTK args>
|
||||
cntkmpirun "-n $Instances" cntk.cntk "speechTrain=[reader=[readerType=HTKDeserializers]] numCPUThreads=$NumCPUThreads precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]]"
|
||||
ExitCode=$?
|
||||
sed 's/^/MPI Rank 0: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank0
|
||||
sed 's/^/MPI Rank 1: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank1
|
||||
sed 's/^/MPI Rank 2: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank2
|
||||
exit $ExitCode
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
dataDir: ../../../Data
|
||||
tags:
|
||||
# - bvt-s (build_sku == '1bitsgd') and ((flavor == 'release') if (os == 'windows') else ((flavor == 'debug') ^ (device == 'cpu')))
|
||||
- nightly-s (build_sku == '1bitsgd')
|
||||
|
||||
testCases:
|
||||
Must train epochs in exactly same order and parameters for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting Epoch {{integer}}
|
||||
- learning rate per sample = {{float}}
|
||||
- momentum = {{float}}
|
||||
|
||||
Epochs must be finished with expected results for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Finished Epoch[{{integer}} of {{integer}}]
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0%}}
|
||||
- EvalClassificationError = {{float,tolerance=0%}}
|
||||
- learningRatePerSample = {{float,tolerance=0.001%}}
|
||||
|
||||
Per-minibatch training results must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Epoch[{{integer}} of {{integer}}]-Minibatch[{{integer}}-{{integer}}
|
||||
- " * {{integer}}; "
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0%}}
|
||||
- EvalClassificationError = {{float,tolerance=0%}}
|
||||
|
||||
DataParallelSGD training parameters must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting minibatch loop
|
||||
- DataParallelSGD training
|
||||
- myRank = {{integer}}
|
||||
- numNodes = 3
|
||||
- numGradientBits = 1
|
||||
- distributed reading is ENABLED
|
||||
|
|
@ -1,613 +0,0 @@
|
|||
CPU info:
|
||||
CPU Model Name: Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
|
||||
Hardware threads: 4
|
||||
Total Memory: 12580404 kB
|
||||
-------------------------------------------------------------------
|
||||
=== Running C:\Program Files\Microsoft MPI\Bin\/mpiexec.exe -n 3 C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:45:01
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (1) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:45:01
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (2) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:45:01
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (0) are in (participating)
|
||||
MPI Rank 0: 12/15/2016 08:45:01: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr_speechTrain.logrank0
|
||||
MPI Rank 0: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:45:01
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
MPI Rank 0: 12/15/2016 08:45:01: Using 1 CPU threads.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:01: ##############################################################################
|
||||
MPI Rank 0: 12/15/2016 08:45:01: # #
|
||||
MPI Rank 0: 12/15/2016 08:45:01: # speechTrain command (train action) #
|
||||
MPI Rank 0: 12/15/2016 08:45:01: # #
|
||||
MPI Rank 0: 12/15/2016 08:45:01: ##############################################################################
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:01:
|
||||
MPI Rank 0: Creating virgin network.
|
||||
MPI Rank 0: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 0: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 0: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 0: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 0: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 0: label set 0: 129 classes
|
||||
MPI Rank 0: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 0: 12/15/2016 08:45:02:
|
||||
MPI Rank 0: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 0: { HLast : [132 x 1 x *]
|
||||
MPI Rank 0: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 0: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 0: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 0: { H2 : [512 x 1 x *]
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { H1 : [512 x 1 x *]
|
||||
MPI Rank 0: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Initializing dataParallelSGD for 1-bit quantization.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:02: MeanOfFeatures = Mean()
|
||||
MPI Rank 0: 12/15/2016 08:45:02: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 0: 12/15/2016 08:45:02: Prior = Mean()
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:05: Precomputing --> Completed.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:06: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:06: Starting minibatch loop.
|
||||
MPI Rank 0: 12/15/2016 08:45:06: Epoch[ 1 of 4]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2622s; samplesPerSecond = 2440.9
|
||||
MPI Rank 0: 12/15/2016 08:45:06: Epoch[ 1 of 4]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2583s; samplesPerSecond = 2477.8
|
||||
MPI Rank 0: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2540s; samplesPerSecond = 2520.0
|
||||
MPI Rank 0: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2483s; samplesPerSecond = 2577.9
|
||||
MPI Rank 0: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2509s; samplesPerSecond = 2551.2
|
||||
MPI Rank 0: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2546s; samplesPerSecond = 2514.0
|
||||
MPI Rank 0: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2567s; samplesPerSecond = 2493.6
|
||||
MPI Rank 0: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2573s; samplesPerSecond = 2487.3
|
||||
MPI Rank 0: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2603s; samplesPerSecond = 2459.0
|
||||
MPI Rank 0: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2683s; samplesPerSecond = 2385.0
|
||||
MPI Rank 0: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2675s; samplesPerSecond = 2392.1
|
||||
MPI Rank 0: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2573s; samplesPerSecond = 2486.9
|
||||
MPI Rank 0: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2748s; samplesPerSecond = 2329.1
|
||||
MPI Rank 0: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2580s; samplesPerSecond = 2480.5
|
||||
MPI Rank 0: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2510s; samplesPerSecond = 2549.4
|
||||
MPI Rank 0: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2493s; samplesPerSecond = 2567.7
|
||||
MPI Rank 0: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2492s; samplesPerSecond = 2568.6
|
||||
MPI Rank 0: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2504s; samplesPerSecond = 2556.3
|
||||
MPI Rank 0: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2555s; samplesPerSecond = 2504.4
|
||||
MPI Rank 0: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2535s; samplesPerSecond = 2524.9
|
||||
MPI Rank 0: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2536s; samplesPerSecond = 2523.8
|
||||
MPI Rank 0: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2634s; samplesPerSecond = 2429.6
|
||||
MPI Rank 0: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2542s; samplesPerSecond = 2517.6
|
||||
MPI Rank 0: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2641s; samplesPerSecond = 2423.0
|
||||
MPI Rank 0: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2655s; samplesPerSecond = 2410.5
|
||||
MPI Rank 0: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2630s; samplesPerSecond = 2433.9
|
||||
MPI Rank 0: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2682s; samplesPerSecond = 2386.6
|
||||
MPI Rank 0: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2565s; samplesPerSecond = 2495.5
|
||||
MPI Rank 0: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2532s; samplesPerSecond = 2527.9
|
||||
MPI Rank 0: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2553s; samplesPerSecond = 2507.0
|
||||
MPI Rank 0: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2611s; samplesPerSecond = 2451.3
|
||||
MPI Rank 0: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2518s; samplesPerSecond = 2541.8
|
||||
MPI Rank 0: 12/15/2016 08:45:14: Finished Epoch[ 1 of 4]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.26s
|
||||
MPI Rank 0: 12/15/2016 08:45:14: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn.1'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:14: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:14: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.017159
|
||||
MPI Rank 0: Async gradient aggregation wait time: 4e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.022209
|
||||
MPI Rank 0: 12/15/2016 08:45:15: Epoch[ 2 of 4]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.23258828 * 2304; EvalClassificationError = 0.61414931 * 2304; time = 0.6861s; samplesPerSecond = 3358.2
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.007336
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.062504
|
||||
MPI Rank 0: Async gradient aggregation wait time: 4e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.017296
|
||||
MPI Rank 0: 12/15/2016 08:45:15: Epoch[ 2 of 4]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.23901091 * 2560; EvalClassificationError = 0.58320313 * 2560; time = 0.5216s; samplesPerSecond = 4908.2
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.022689
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.041876
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.045387
|
||||
MPI Rank 0: 12/15/2016 08:45:16: Epoch[ 2 of 4]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.16822363 * 2560; EvalClassificationError = 0.57773438 * 2560; time = 0.6671s; samplesPerSecond = 3837.3
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.00641
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.043876
|
||||
MPI Rank 0: Async gradient aggregation wait time: 4e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.027536
|
||||
MPI Rank 0: 12/15/2016 08:45:17: Epoch[ 2 of 4]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.19927971 * 2560; EvalClassificationError = 0.62187500 * 2560; time = 0.5975s; samplesPerSecond = 4284.5
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.006883
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.045305
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.062512
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.05298
|
||||
MPI Rank 0: 12/15/2016 08:45:17: Epoch[ 2 of 4]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.22075939 * 2560; EvalClassificationError = 0.59648437 * 2560; time = 0.5649s; samplesPerSecond = 4531.5
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.012483
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.049937
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.010496
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.046163
|
||||
MPI Rank 0: 12/15/2016 08:45:18: Epoch[ 2 of 4]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.11227615 * 2560; EvalClassificationError = 0.57382813 * 2560; time = 0.5057s; samplesPerSecond = 5062.7
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.025722
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.018403
|
||||
MPI Rank 0: 12/15/2016 08:45:18: Epoch[ 2 of 4]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.17322591 * 2560; EvalClassificationError = 0.61914063 * 2560; time = 0.5979s; samplesPerSecond = 4281.6
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.026228
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.027207
|
||||
MPI Rank 0: 12/15/2016 08:45:19: Epoch[ 2 of 4]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.13027284 * 2560; EvalClassificationError = 0.60820312 * 2560; time = 0.6650s; samplesPerSecond = 3849.4
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.015012
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.024411
|
||||
MPI Rank 0: 12/15/2016 08:45:19: Finished Epoch[ 2 of 4]: [Training] CrossEntropyWithSoftmax = 2.18324277 * 20480; EvalClassificationError = 0.59892578 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=4.85012s
|
||||
MPI Rank 0: 12/15/2016 08:45:19: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn.2'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:19: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:19: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.01979
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.017679
|
||||
MPI Rank 0: 12/15/2016 08:45:20: Epoch[ 3 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 2.20597127 * 9216; EvalClassificationError = 0.58593750 * 9216; time = 1.3442s; samplesPerSecond = 6856.0
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.065355
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.066516
|
||||
MPI Rank 0: 12/15/2016 08:45:22: Epoch[ 3 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 2.14626719 * 10240; EvalClassificationError = 0.58886719 * 10240; time = 1.2738s; samplesPerSecond = 8039.0
|
||||
MPI Rank 0: 12/15/2016 08:45:22: Finished Epoch[ 3 of 4]: [Training] CrossEntropyWithSoftmax = 2.16917041 * 20480; EvalClassificationError = 0.58637695 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.73964s
|
||||
MPI Rank 0: 12/15/2016 08:45:22: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn.3'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:22: Starting Epoch 4: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 3: frames [61440..81920] (first utterance at frame 61440), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:22: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.04443
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.012333
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.126445
|
||||
MPI Rank 0: 12/15/2016 08:45:23: Epoch[ 4 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.99117050 * 9216; EvalClassificationError = 0.54427083 * 9216; time = 1.2882s; samplesPerSecond = 7154.4
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.029068
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.108564
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.006866
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.123917
|
||||
MPI Rank 0: 12/15/2016 08:45:24: Epoch[ 4 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.97438950 * 10240; EvalClassificationError = 0.54345703 * 10240; time = 1.3228s; samplesPerSecond = 7741.2
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.022913
|
||||
MPI Rank 0: 12/15/2016 08:45:25: Finished Epoch[ 4 of 4]: [Training] CrossEntropyWithSoftmax = 1.98353069 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-005; epochTime=2.76533s
|
||||
MPI Rank 0: 12/15/2016 08:45:25: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:25: Action "train" complete.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:45:25: __COMPLETED__
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr_speechTrain.logrank1
|
||||
MPI Rank 1: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:45:01
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Using 1 CPU threads.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:02: ##############################################################################
|
||||
MPI Rank 1: 12/15/2016 08:45:02: # #
|
||||
MPI Rank 1: 12/15/2016 08:45:02: # speechTrain command (train action) #
|
||||
MPI Rank 1: 12/15/2016 08:45:02: # #
|
||||
MPI Rank 1: 12/15/2016 08:45:02: ##############################################################################
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:02:
|
||||
MPI Rank 1: Creating virgin network.
|
||||
MPI Rank 1: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 1: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 1: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 1: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 1: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 1: label set 0: 129 classes
|
||||
MPI Rank 1: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 1: 12/15/2016 08:45:02:
|
||||
MPI Rank 1: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 1: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 1: { H1 : [512 x 1 x *]
|
||||
MPI Rank 1: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 1: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { H2 : [512 x 1 x *]
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 1: { HLast : [132 x 1 x *]
|
||||
MPI Rank 1: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Initializing dataParallelSGD for 1-bit quantization.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:02: MeanOfFeatures = Mean()
|
||||
MPI Rank 1: 12/15/2016 08:45:02: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 1: 12/15/2016 08:45:02: Prior = Mean()
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:05: Precomputing --> Completed.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:06: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:06: Starting minibatch loop.
|
||||
MPI Rank 1: 12/15/2016 08:45:06: Epoch[ 1 of 4]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2913s; samplesPerSecond = 2197.1
|
||||
MPI Rank 1: 12/15/2016 08:45:06: Epoch[ 1 of 4]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2671s; samplesPerSecond = 2396.1
|
||||
MPI Rank 1: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2515s; samplesPerSecond = 2544.4
|
||||
MPI Rank 1: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2473s; samplesPerSecond = 2588.5
|
||||
MPI Rank 1: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2596s; samplesPerSecond = 2465.4
|
||||
MPI Rank 1: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2641s; samplesPerSecond = 2423.6
|
||||
MPI Rank 1: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2588s; samplesPerSecond = 2472.9
|
||||
MPI Rank 1: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2569s; samplesPerSecond = 2491.1
|
||||
MPI Rank 1: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2589s; samplesPerSecond = 2471.6
|
||||
MPI Rank 1: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2676s; samplesPerSecond = 2391.9
|
||||
MPI Rank 1: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2688s; samplesPerSecond = 2381.2
|
||||
MPI Rank 1: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2586s; samplesPerSecond = 2475.0
|
||||
MPI Rank 1: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2710s; samplesPerSecond = 2361.7
|
||||
MPI Rank 1: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2622s; samplesPerSecond = 2440.6
|
||||
MPI Rank 1: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2595s; samplesPerSecond = 2466.3
|
||||
MPI Rank 1: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2516s; samplesPerSecond = 2543.9
|
||||
MPI Rank 1: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2499s; samplesPerSecond = 2560.6
|
||||
MPI Rank 1: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2585s; samplesPerSecond = 2476.0
|
||||
MPI Rank 1: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2595s; samplesPerSecond = 2466.6
|
||||
MPI Rank 1: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2609s; samplesPerSecond = 2452.9
|
||||
MPI Rank 1: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2680s; samplesPerSecond = 2387.9
|
||||
MPI Rank 1: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2573s; samplesPerSecond = 2487.7
|
||||
MPI Rank 1: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2515s; samplesPerSecond = 2544.7
|
||||
MPI Rank 1: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2533s; samplesPerSecond = 2526.8
|
||||
MPI Rank 1: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2568s; samplesPerSecond = 2492.4
|
||||
MPI Rank 1: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2552s; samplesPerSecond = 2507.7
|
||||
MPI Rank 1: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2580s; samplesPerSecond = 2480.8
|
||||
MPI Rank 1: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2551s; samplesPerSecond = 2508.4
|
||||
MPI Rank 1: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2533s; samplesPerSecond = 2526.6
|
||||
MPI Rank 1: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2562s; samplesPerSecond = 2497.6
|
||||
MPI Rank 1: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2600s; samplesPerSecond = 2461.6
|
||||
MPI Rank 1: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2475s; samplesPerSecond = 2585.6
|
||||
MPI Rank 1: 12/15/2016 08:45:14: Finished Epoch[ 1 of 4]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.30884s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:14: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:14: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.042853
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.025159
|
||||
MPI Rank 1: 12/15/2016 08:45:15: Epoch[ 2 of 4]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.23258828 * 2304; EvalClassificationError = 0.61414931 * 2304; time = 0.7137s; samplesPerSecond = 3228.3
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.011636
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.05291
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.06182
|
||||
MPI Rank 1: 12/15/2016 08:45:15: Epoch[ 2 of 4]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.23901091 * 2560; EvalClassificationError = 0.58320313 * 2560; time = 0.5390s; samplesPerSecond = 4749.4
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.021365
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.03829
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.067366
|
||||
MPI Rank 1: 12/15/2016 08:45:16: Epoch[ 2 of 4]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.16822363 * 2560; EvalClassificationError = 0.57773438 * 2560; time = 0.6390s; samplesPerSecond = 4006.2
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.016716
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.046773
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.035357
|
||||
MPI Rank 1: 12/15/2016 08:45:17: Epoch[ 2 of 4]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.19927971 * 2560; EvalClassificationError = 0.62187500 * 2560; time = 0.6239s; samplesPerSecond = 4103.2
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.01908
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.019362
|
||||
MPI Rank 1: 12/15/2016 08:45:17: Epoch[ 2 of 4]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.22075939 * 2560; EvalClassificationError = 0.59648437 * 2560; time = 0.5690s; samplesPerSecond = 4499.2
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.019096
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.028039
|
||||
MPI Rank 1: 12/15/2016 08:45:18: Epoch[ 2 of 4]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.11227615 * 2560; EvalClassificationError = 0.57382813 * 2560; time = 0.4738s; samplesPerSecond = 5403.2
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.051405
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.050967
|
||||
MPI Rank 1: 12/15/2016 08:45:18: Epoch[ 2 of 4]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.17322591 * 2560; EvalClassificationError = 0.61914063 * 2560; time = 0.6206s; samplesPerSecond = 4125.0
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.023017
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.03727
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.037849
|
||||
MPI Rank 1: 12/15/2016 08:45:19: Epoch[ 2 of 4]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.13027284 * 2560; EvalClassificationError = 0.60820312 * 2560; time = 0.6112s; samplesPerSecond = 4188.6
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.04738
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.01977
|
||||
MPI Rank 1: 12/15/2016 08:45:19: Finished Epoch[ 2 of 4]: [Training] CrossEntropyWithSoftmax = 2.18324277 * 20480; EvalClassificationError = 0.59892578 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=4.86625s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:19: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:19: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.048671
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.162308
|
||||
MPI Rank 1: 12/15/2016 08:45:20: Epoch[ 3 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 2.20597127 * 9216; EvalClassificationError = 0.58593750 * 9216; time = 1.4064s; samplesPerSecond = 6553.0
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.029329
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.028563
|
||||
MPI Rank 1: 12/15/2016 08:45:22: Epoch[ 3 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 2.14626719 * 10240; EvalClassificationError = 0.58886719 * 10240; time = 1.2755s; samplesPerSecond = 8028.1
|
||||
MPI Rank 1: 12/15/2016 08:45:22: Finished Epoch[ 3 of 4]: [Training] CrossEntropyWithSoftmax = 2.16917041 * 20480; EvalClassificationError = 0.58637695 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.74129s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:22: Starting Epoch 4: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 3: frames [61440..81920] (first utterance at frame 61440), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:22: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.078784
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.020759
|
||||
MPI Rank 1: 12/15/2016 08:45:23: Epoch[ 4 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.99117050 * 9216; EvalClassificationError = 0.54427083 * 9216; time = 1.3985s; samplesPerSecond = 6589.8
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.022602
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.019065
|
||||
MPI Rank 1: 12/15/2016 08:45:25: Epoch[ 4 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.97438950 * 10240; EvalClassificationError = 0.54345703 * 10240; time = 1.3151s; samplesPerSecond = 7786.7
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.022045
|
||||
MPI Rank 1: 12/15/2016 08:45:25: Finished Epoch[ 4 of 4]: [Training] CrossEntropyWithSoftmax = 1.98353069 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-005; epochTime=2.76637s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:25: Action "train" complete.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:45:25: __COMPLETED__
|
||||
MPI Rank 2: 12/15/2016 08:45:02: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr_speechTrain.logrank2
|
||||
MPI Rank 2: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:31:19) on cntk-muc03 at 2016/12/15 08:45:01
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215084323.697388\Speech\DNN_ParallelBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
MPI Rank 2: 12/15/2016 08:45:02: Using 1 CPU threads.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:02: ##############################################################################
|
||||
MPI Rank 2: 12/15/2016 08:45:02: # #
|
||||
MPI Rank 2: 12/15/2016 08:45:02: # speechTrain command (train action) #
|
||||
MPI Rank 2: 12/15/2016 08:45:02: # #
|
||||
MPI Rank 2: 12/15/2016 08:45:02: ##############################################################################
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:02:
|
||||
MPI Rank 2: Creating virgin network.
|
||||
MPI Rank 2: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 2: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 2: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 2: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 2: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 2: label set 0: 129 classes
|
||||
MPI Rank 2: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 2: 12/15/2016 08:45:03:
|
||||
MPI Rank 2: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: { H1 : [512 x 1 x *]
|
||||
MPI Rank 2: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 2: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 2: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 2: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 2: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { H2 : [512 x 1 x *]
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { HLast : [132 x 1 x *]
|
||||
MPI Rank 2: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Initializing dataParallelSGD for 1-bit quantization.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:03: MeanOfFeatures = Mean()
|
||||
MPI Rank 2: 12/15/2016 08:45:03: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 2: 12/15/2016 08:45:03: Prior = Mean()
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:06: Precomputing --> Completed.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:06: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:06: Starting minibatch loop.
|
||||
MPI Rank 2: 12/15/2016 08:45:06: Epoch[ 1 of 4]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2718s; samplesPerSecond = 2354.8
|
||||
MPI Rank 2: 12/15/2016 08:45:06: Epoch[ 1 of 4]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2665s; samplesPerSecond = 2401.4
|
||||
MPI Rank 2: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2495s; samplesPerSecond = 2564.7
|
||||
MPI Rank 2: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2478s; samplesPerSecond = 2583.0
|
||||
MPI Rank 2: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2617s; samplesPerSecond = 2445.2
|
||||
MPI Rank 2: 12/15/2016 08:45:07: Epoch[ 1 of 4]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2624s; samplesPerSecond = 2439.0
|
||||
MPI Rank 2: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2531s; samplesPerSecond = 2529.1
|
||||
MPI Rank 2: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2515s; samplesPerSecond = 2545.2
|
||||
MPI Rank 2: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2510s; samplesPerSecond = 2549.4
|
||||
MPI Rank 2: 12/15/2016 08:45:08: Epoch[ 1 of 4]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2601s; samplesPerSecond = 2460.8
|
||||
MPI Rank 2: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2549s; samplesPerSecond = 2511.1
|
||||
MPI Rank 2: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2507s; samplesPerSecond = 2553.2
|
||||
MPI Rank 2: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2712s; samplesPerSecond = 2359.7
|
||||
MPI Rank 2: 12/15/2016 08:45:09: Epoch[ 1 of 4]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2609s; samplesPerSecond = 2453.0
|
||||
MPI Rank 2: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2888s; samplesPerSecond = 2216.0
|
||||
MPI Rank 2: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2517s; samplesPerSecond = 2543.0
|
||||
MPI Rank 2: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2497s; samplesPerSecond = 2563.3
|
||||
MPI Rank 2: 12/15/2016 08:45:10: Epoch[ 1 of 4]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2535s; samplesPerSecond = 2524.7
|
||||
MPI Rank 2: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2557s; samplesPerSecond = 2503.3
|
||||
MPI Rank 2: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2673s; samplesPerSecond = 2394.1
|
||||
MPI Rank 2: 12/15/2016 08:45:11: Epoch[ 1 of 4]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2666s; samplesPerSecond = 2401.0
|
||||
MPI Rank 2: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2671s; samplesPerSecond = 2396.5
|
||||
MPI Rank 2: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2558s; samplesPerSecond = 2502.1
|
||||
MPI Rank 2: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2641s; samplesPerSecond = 2423.5
|
||||
MPI Rank 2: 12/15/2016 08:45:12: Epoch[ 1 of 4]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2622s; samplesPerSecond = 2441.3
|
||||
MPI Rank 2: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2661s; samplesPerSecond = 2405.1
|
||||
MPI Rank 2: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2646s; samplesPerSecond = 2418.5
|
||||
MPI Rank 2: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2509s; samplesPerSecond = 2550.4
|
||||
MPI Rank 2: 12/15/2016 08:45:13: Epoch[ 1 of 4]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2504s; samplesPerSecond = 2555.4
|
||||
MPI Rank 2: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2533s; samplesPerSecond = 2527.0
|
||||
MPI Rank 2: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2514s; samplesPerSecond = 2545.2
|
||||
MPI Rank 2: 12/15/2016 08:45:14: Epoch[ 1 of 4]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2466s; samplesPerSecond = 2594.9
|
||||
MPI Rank 2: 12/15/2016 08:45:14: Finished Epoch[ 1 of 4]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.29154s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:14: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:14: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.045254
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.01563
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.034825
|
||||
MPI Rank 2: 12/15/2016 08:45:15: Epoch[ 2 of 4]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.23258828 * 2304; EvalClassificationError = 0.61414931 * 2304; time = 0.7033s; samplesPerSecond = 3275.8
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.025709
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.062272
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.008646
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.074517
|
||||
MPI Rank 2: 12/15/2016 08:45:15: Epoch[ 2 of 4]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.23901091 * 2560; EvalClassificationError = 0.58320313 * 2560; time = 0.5417s; samplesPerSecond = 4725.6
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.005925
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.04346
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.058485
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.06787
|
||||
MPI Rank 2: 12/15/2016 08:45:16: Epoch[ 2 of 4]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.16822363 * 2560; EvalClassificationError = 0.57773438 * 2560; time = 0.6751s; samplesPerSecond = 3791.9
|
||||
MPI Rank 2: Async gradient aggregation wait time: 4e-006
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.038691
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.014519
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.044701
|
||||
MPI Rank 2: 12/15/2016 08:45:17: Epoch[ 2 of 4]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.19927971 * 2560; EvalClassificationError = 0.62187500 * 2560; time = 0.5667s; samplesPerSecond = 4517.2
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.026562
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.04628
|
||||
MPI Rank 2: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.048649
|
||||
MPI Rank 2: 12/15/2016 08:45:17: Epoch[ 2 of 4]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.22075939 * 2560; EvalClassificationError = 0.59648437 * 2560; time = 0.5674s; samplesPerSecond = 4511.8
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.026443
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.049981
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.017769
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.037914
|
||||
MPI Rank 2: 12/15/2016 08:45:18: Epoch[ 2 of 4]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.11227615 * 2560; EvalClassificationError = 0.57382813 * 2560; time = 0.5163s; samplesPerSecond = 4958.2
|
||||
MPI Rank 2: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.054407
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.009165
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.06387
|
||||
MPI Rank 2: 12/15/2016 08:45:18: Epoch[ 2 of 4]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.17322591 * 2560; EvalClassificationError = 0.61914063 * 2560; time = 0.5886s; samplesPerSecond = 4349.3
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.008188
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.039783
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.140833
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.047561
|
||||
MPI Rank 2: 12/15/2016 08:45:19: Epoch[ 2 of 4]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.13027284 * 2560; EvalClassificationError = 0.60820312 * 2560; time = 0.6432s; samplesPerSecond = 3979.9
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.035015
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.022108
|
||||
MPI Rank 2: 12/15/2016 08:45:19: Finished Epoch[ 2 of 4]: [Training] CrossEntropyWithSoftmax = 2.18324277 * 20480; EvalClassificationError = 0.59892578 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=4.86587s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:19: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:19: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.058171
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.126685
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.054589
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.248473
|
||||
MPI Rank 2: 12/15/2016 08:45:20: Epoch[ 3 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 2.20597127 * 9216; EvalClassificationError = 0.58593750 * 9216; time = 1.3300s; samplesPerSecond = 6929.2
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.030365
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.141699
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.04912
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.131064
|
||||
MPI Rank 2: 12/15/2016 08:45:22: Epoch[ 3 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 2.14626719 * 10240; EvalClassificationError = 0.58886719 * 10240; time = 1.2240s; samplesPerSecond = 8366.0
|
||||
MPI Rank 2: 12/15/2016 08:45:22: Finished Epoch[ 3 of 4]: [Training] CrossEntropyWithSoftmax = 2.16917041 * 20480; EvalClassificationError = 0.58637695 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.72378s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:22: Starting Epoch 4: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 3: frames [61440..81920] (first utterance at frame 61440), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:22: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 1), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 2: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.129446
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.040623
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.126456
|
||||
MPI Rank 2: 12/15/2016 08:45:23: Epoch[ 4 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.99117050 * 9216; EvalClassificationError = 0.54427083 * 9216; time = 1.2583s; samplesPerSecond = 7324.0
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.054145
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.120512
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.044203
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.115423
|
||||
MPI Rank 2: 12/15/2016 08:45:24: Epoch[ 4 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.97438950 * 10240; EvalClassificationError = 0.54345703 * 10240; time = 1.3223s; samplesPerSecond = 7744.0
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.019428
|
||||
MPI Rank 2: 12/15/2016 08:45:25: Finished Epoch[ 4 of 4]: [Training] CrossEntropyWithSoftmax = 1.98353069 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-005; epochTime=2.73407s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:25: Action "train" complete.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:45:25: __COMPLETED__
|
||||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
|
|
@ -1,23 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
. $TEST_ROOT_DIR/run-test-common
|
||||
|
||||
OriginalTestDir=../../../DNN/ParallelBufferedAsyncGradientAggregation
|
||||
ConfigDir=$TEST_DIR/../../../DNN
|
||||
LogFileName=stderr
|
||||
Instances=3
|
||||
NumCPUThreads=$(threadsPerInstance $Instances)
|
||||
|
||||
(cd $TEST_DIR/$OriginalTestDir && md5sum baseline*) | (cd $TEST_DIR && md5sum --status -c -)
|
||||
if [ $? != 0 ]; then
|
||||
echo Error: Baselines must match original test. Copy from $OriginalTestDir.
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# cntkmpirun <MPI args> <CNTK config file name> <additional CNTK args>
|
||||
cntkmpirun "-n $Instances" cntk.cntk "speechTrain=[reader=[readerType=HTKDeserializers]] numCPUThreads=$NumCPUThreads precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=1]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]]"
|
||||
ExitCode=$?
|
||||
sed 's/^/MPI Rank 0: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank0
|
||||
sed 's/^/MPI Rank 1: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank1
|
||||
sed 's/^/MPI Rank 2: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank2
|
||||
exit $ExitCode
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
dataDir: ../../../Data
|
||||
tags:
|
||||
# - bvt-s (build_sku == 'gpu') and ((flavor == 'release') if (os == 'windows') else ((flavor == 'debug') ^ (device == 'cpu')))
|
||||
- nightly-s (build_sku == '1bitsgd')
|
||||
|
||||
testCases:
|
||||
Must train epochs in exactly same order and parameters for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting Epoch {{integer}}
|
||||
- learning rate per sample = {{float}}
|
||||
- momentum = {{float}}
|
||||
|
||||
Epochs must be finished with expected results for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Finished Epoch[{{integer}} of {{integer}}]
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0%}}
|
||||
- EvalClassificationError = {{float,tolerance=0.00000001}}
|
||||
- learningRatePerSample = {{float,tolerance=0%}}
|
||||
|
||||
Per-minibatch training results must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Epoch[{{integer}} of {{integer}}]-Minibatch[{{integer}}-{{integer}}
|
||||
- " * {{integer}}; "
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0%}}
|
||||
- EvalClassificationError = {{float,tolerance=0.00000001}}
|
||||
|
||||
DataParallelSGD training parameters must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting minibatch loop
|
||||
- DataParallelSGD training
|
||||
- myRank = {{integer}}
|
||||
- numNodes = 3
|
||||
- numGradientBits = 1
|
||||
- distributed reading is ENABLED
|
||||
- BufferedAsyncGradientAggregation is ENABLED
|
||||
|
|
@ -1,438 +0,0 @@
|
|||
CPU info:
|
||||
CPU Model Name: Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
|
||||
Hardware threads: 4
|
||||
Total Memory: 12580404 kB
|
||||
-------------------------------------------------------------------
|
||||
=== Running C:\Program Files\Microsoft MPI\Bin\/mpiexec.exe -n 3 C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:16
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (1) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:16
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (2) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:16
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (0) are in (participating)
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr_speechTrain.logrank0
|
||||
MPI Rank 0: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:16
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Using 1 CPU threads.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:16: ##############################################################################
|
||||
MPI Rank 0: 12/15/2016 08:31:16: # #
|
||||
MPI Rank 0: 12/15/2016 08:31:16: # speechTrain command (train action) #
|
||||
MPI Rank 0: 12/15/2016 08:31:16: # #
|
||||
MPI Rank 0: 12/15/2016 08:31:16: ##############################################################################
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:16:
|
||||
MPI Rank 0: Creating virgin network.
|
||||
MPI Rank 0: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 0: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 0: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 0: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 0: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 0: label set 0: 129 classes
|
||||
MPI Rank 0: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 0: 12/15/2016 08:31:16:
|
||||
MPI Rank 0: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { HLast : [132 x 1 x *]
|
||||
MPI Rank 0: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 0: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 0: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 0: { H2 : [512 x 1 x *]
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { H1 : [512 x 1 x *]
|
||||
MPI Rank 0: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 0: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Initializing dataParallelSGD with FP32 aggregation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:16: MeanOfFeatures = Mean()
|
||||
MPI Rank 0: 12/15/2016 08:31:16: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 0: 12/15/2016 08:31:16: Prior = Mean()
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:19: Precomputing --> Completed.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755209 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.1225s; samplesPerSecond = 5225.1
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610347 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.1122s; samplesPerSecond = 5704.3
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222493 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.1140s; samplesPerSecond = 5613.7
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152761 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.1134s; samplesPerSecond = 5643.9
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818495 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.1116s; samplesPerSecond = 5737.0
|
||||
MPI Rank 0: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641133 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.1124s; samplesPerSecond = 5694.5
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802654 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.1131s; samplesPerSecond = 5657.7
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832811 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.1120s; samplesPerSecond = 5716.2
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50627956 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.1143s; samplesPerSecond = 5600.5
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478094 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.1109s; samplesPerSecond = 5770.2
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031055 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.1135s; samplesPerSecond = 5638.0
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365293 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.1114s; samplesPerSecond = 5743.3
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20931888 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.1114s; samplesPerSecond = 5747.3
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460312 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.1083s; samplesPerSecond = 5911.6
|
||||
MPI Rank 0: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97528860 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.1105s; samplesPerSecond = 5791.6
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968648 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.1123s; samplesPerSecond = 5696.6
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84171867 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.1130s; samplesPerSecond = 5664.7
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031476 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1095s; samplesPerSecond = 5845.3
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83857843 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.1108s; samplesPerSecond = 5774.1
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632032 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.1098s; samplesPerSecond = 5831.0
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61032974 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.1097s; samplesPerSecond = 5831.9
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330475 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.1111s; samplesPerSecond = 5761.5
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591535 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.1095s; samplesPerSecond = 5843.4
|
||||
MPI Rank 0: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566229 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1103s; samplesPerSecond = 5803.4
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164700 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.1096s; samplesPerSecond = 5838.5
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954552 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.1142s; samplesPerSecond = 5605.4
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27033979 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.1105s; samplesPerSecond = 5792.2
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112142 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1101s; samplesPerSecond = 5810.8
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800742 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.1102s; samplesPerSecond = 5809.0
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783400 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.1086s; samplesPerSecond = 5891.6
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590123 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.1105s; samplesPerSecond = 5791.8
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415391 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.1092s; samplesPerSecond = 5860.6
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.04696796 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=3.58853s
|
||||
MPI Rank 0: 12/15/2016 08:31:23: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/models/cntkSpeech.dnn.1'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:23: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.14624175 * 2560; EvalClassificationError = 0.56953125 * 2560; time = 0.2418s; samplesPerSecond = 10587.8
|
||||
MPI Rank 0: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.06174128 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 0.2326s; samplesPerSecond = 11004.5
|
||||
MPI Rank 0: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.04994338 * 2560; EvalClassificationError = 0.55351562 * 2560; time = 0.2293s; samplesPerSecond = 11166.7
|
||||
MPI Rank 0: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.03695538 * 2560; EvalClassificationError = 0.56132812 * 2560; time = 0.2298s; samplesPerSecond = 11138.5
|
||||
MPI Rank 0: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.03086227 * 2560; EvalClassificationError = 0.55664063 * 2560; time = 0.2347s; samplesPerSecond = 10907.4
|
||||
MPI Rank 0: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 1.97306193 * 2560; EvalClassificationError = 0.53671875 * 2560; time = 0.2221s; samplesPerSecond = 11524.3
|
||||
MPI Rank 0: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 1.96746064 * 2560; EvalClassificationError = 0.53164062 * 2560; time = 0.2241s; samplesPerSecond = 11425.0
|
||||
MPI Rank 0: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 1.95498165 * 2560; EvalClassificationError = 0.53750000 * 2560; time = 0.2250s; samplesPerSecond = 11378.8
|
||||
MPI Rank 0: 12/15/2016 08:31:25: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.02765603 * 20480; EvalClassificationError = 0.55053711 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=1.84777s
|
||||
MPI Rank 0: 12/15/2016 08:31:25: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/models/cntkSpeech.dnn.2'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:25: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:25: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:31:26: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.95358449 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 0.7052s; samplesPerSecond = 14520.5
|
||||
MPI Rank 0: 12/15/2016 08:31:27: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.97540911 * 10240; EvalClassificationError = 0.55253906 * 10240; time = 0.6838s; samplesPerSecond = 14975.6
|
||||
MPI Rank 0: 12/15/2016 08:31:27: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.96449680 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=1.39589s
|
||||
MPI Rank 0: 12/15/2016 08:31:27: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/models/cntkSpeech.dnn'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:27: Action "train" complete.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:27: __COMPLETED__
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr_speechTrain.logrank1
|
||||
MPI Rank 1: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:16
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Using 1 CPU threads.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:17: ##############################################################################
|
||||
MPI Rank 1: 12/15/2016 08:31:17: # #
|
||||
MPI Rank 1: 12/15/2016 08:31:17: # speechTrain command (train action) #
|
||||
MPI Rank 1: 12/15/2016 08:31:17: # #
|
||||
MPI Rank 1: 12/15/2016 08:31:17: ##############################################################################
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:17:
|
||||
MPI Rank 1: Creating virgin network.
|
||||
MPI Rank 1: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 1: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 1: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 1: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 1: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 1: label set 0: 129 classes
|
||||
MPI Rank 1: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 1: 12/15/2016 08:31:17:
|
||||
MPI Rank 1: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: { HLast : [132 x 1 x *]
|
||||
MPI Rank 1: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 1: { H1 : [512 x 1 x *]
|
||||
MPI Rank 1: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 1: { H2 : [512 x 1 x *]
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 1: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 1: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Initializing dataParallelSGD with FP32 aggregation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:17: MeanOfFeatures = Mean()
|
||||
MPI Rank 1: 12/15/2016 08:31:17: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 1: 12/15/2016 08:31:17: Prior = Mean()
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:19: Precomputing --> Completed.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755209 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.1266s; samplesPerSecond = 5054.1
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610347 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.1122s; samplesPerSecond = 5704.8
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222493 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.1140s; samplesPerSecond = 5613.1
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152761 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.1135s; samplesPerSecond = 5641.1
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818495 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.1115s; samplesPerSecond = 5738.9
|
||||
MPI Rank 1: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641133 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.1124s; samplesPerSecond = 5694.4
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802654 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.1131s; samplesPerSecond = 5658.0
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832811 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.1120s; samplesPerSecond = 5716.0
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50627956 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.1143s; samplesPerSecond = 5601.3
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478094 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.1109s; samplesPerSecond = 5770.2
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031055 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.1135s; samplesPerSecond = 5637.7
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365293 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.1114s; samplesPerSecond = 5743.2
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20931888 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.1114s; samplesPerSecond = 5747.1
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460312 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.1083s; samplesPerSecond = 5910.5
|
||||
MPI Rank 1: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97528860 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.1105s; samplesPerSecond = 5791.5
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968648 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.1123s; samplesPerSecond = 5697.8
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84171867 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.1130s; samplesPerSecond = 5664.4
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031476 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1095s; samplesPerSecond = 5845.0
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83857843 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.1108s; samplesPerSecond = 5773.8
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632032 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.1097s; samplesPerSecond = 5832.7
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61032974 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.1098s; samplesPerSecond = 5829.6
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330475 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.1111s; samplesPerSecond = 5761.0
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591535 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.1096s; samplesPerSecond = 5841.4
|
||||
MPI Rank 1: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566229 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1103s; samplesPerSecond = 5800.9
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164700 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.1097s; samplesPerSecond = 5835.1
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954552 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.1141s; samplesPerSecond = 5607.5
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27033979 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.1105s; samplesPerSecond = 5792.1
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112142 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1102s; samplesPerSecond = 5809.7
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800742 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.1102s; samplesPerSecond = 5809.4
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783400 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.1086s; samplesPerSecond = 5891.6
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590123 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.1105s; samplesPerSecond = 5791.0
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415391 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.1092s; samplesPerSecond = 5860.4
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.04696796 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=3.59272s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:23: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.14624175 * 2560; EvalClassificationError = 0.56953125 * 2560; time = 0.2419s; samplesPerSecond = 10583.8
|
||||
MPI Rank 1: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.06174128 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 0.2327s; samplesPerSecond = 11002.6
|
||||
MPI Rank 1: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.04994338 * 2560; EvalClassificationError = 0.55351562 * 2560; time = 0.2293s; samplesPerSecond = 11164.1
|
||||
MPI Rank 1: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.03695538 * 2560; EvalClassificationError = 0.56132812 * 2560; time = 0.2298s; samplesPerSecond = 11140.6
|
||||
MPI Rank 1: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.03086227 * 2560; EvalClassificationError = 0.55664063 * 2560; time = 0.2347s; samplesPerSecond = 10907.3
|
||||
MPI Rank 1: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 1.97306193 * 2560; EvalClassificationError = 0.53671875 * 2560; time = 0.2222s; samplesPerSecond = 11522.8
|
||||
MPI Rank 1: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 1.96746064 * 2560; EvalClassificationError = 0.53164062 * 2560; time = 0.2240s; samplesPerSecond = 11427.0
|
||||
MPI Rank 1: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 1.95498165 * 2560; EvalClassificationError = 0.53750000 * 2560; time = 0.2250s; samplesPerSecond = 11378.7
|
||||
MPI Rank 1: 12/15/2016 08:31:25: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.02765603 * 20480; EvalClassificationError = 0.55053711 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=1.84776s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:25: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:25: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:31:26: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.95358449 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 0.7053s; samplesPerSecond = 14519.2
|
||||
MPI Rank 1: 12/15/2016 08:31:27: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.97540911 * 10240; EvalClassificationError = 0.55253906 * 10240; time = 0.6838s; samplesPerSecond = 14975.4
|
||||
MPI Rank 1: 12/15/2016 08:31:27: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.96449680 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=1.39588s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:27: Action "train" complete.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:27: __COMPLETED__
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr_speechTrain.logrank2
|
||||
MPI Rank 2: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:16
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_cpu/stderr
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Using 1 CPU threads.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:17: ##############################################################################
|
||||
MPI Rank 2: 12/15/2016 08:31:17: # #
|
||||
MPI Rank 2: 12/15/2016 08:31:17: # speechTrain command (train action) #
|
||||
MPI Rank 2: 12/15/2016 08:31:17: # #
|
||||
MPI Rank 2: 12/15/2016 08:31:17: ##############################################################################
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:17:
|
||||
MPI Rank 2: Creating virgin network.
|
||||
MPI Rank 2: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 2: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 2: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 2: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 2: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 2: label set 0: 129 classes
|
||||
MPI Rank 2: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 2: 12/15/2016 08:31:17:
|
||||
MPI Rank 2: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 2: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { HLast : [132 x 1 x *]
|
||||
MPI Rank 2: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 2: { H2 : [512 x 1 x *]
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { H1 : [512 x 1 x *]
|
||||
MPI Rank 2: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 2: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 2: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 2: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Initializing dataParallelSGD with FP32 aggregation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:17: MeanOfFeatures = Mean()
|
||||
MPI Rank 2: 12/15/2016 08:31:17: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 2: 12/15/2016 08:31:17: Prior = Mean()
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Precomputing --> Completed.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755209 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.1259s; samplesPerSecond = 5082.3
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610347 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.1121s; samplesPerSecond = 5707.5
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222493 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.1140s; samplesPerSecond = 5613.8
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152761 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.1133s; samplesPerSecond = 5647.3
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818495 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.1115s; samplesPerSecond = 5739.4
|
||||
MPI Rank 2: 12/15/2016 08:31:20: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641133 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.1124s; samplesPerSecond = 5694.7
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802654 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.1131s; samplesPerSecond = 5659.7
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832811 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.1119s; samplesPerSecond = 5718.0
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50627956 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.1142s; samplesPerSecond = 5601.8
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478094 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.1109s; samplesPerSecond = 5770.7
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031055 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.1135s; samplesPerSecond = 5641.1
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365293 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.1114s; samplesPerSecond = 5744.4
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20931888 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.1113s; samplesPerSecond = 5748.1
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460312 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.1082s; samplesPerSecond = 5914.6
|
||||
MPI Rank 2: 12/15/2016 08:31:21: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97528860 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.1105s; samplesPerSecond = 5792.4
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968648 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.1123s; samplesPerSecond = 5699.5
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84171867 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.1130s; samplesPerSecond = 5665.8
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031476 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1094s; samplesPerSecond = 5850.0
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83857843 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.1108s; samplesPerSecond = 5775.1
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632032 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.1097s; samplesPerSecond = 5834.4
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61032974 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.1098s; samplesPerSecond = 5831.3
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330475 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.1110s; samplesPerSecond = 5763.8
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591535 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.1096s; samplesPerSecond = 5841.9
|
||||
MPI Rank 2: 12/15/2016 08:31:22: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566229 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1103s; samplesPerSecond = 5803.0
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164700 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.1096s; samplesPerSecond = 5837.8
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954552 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.1141s; samplesPerSecond = 5609.5
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27033979 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.1105s; samplesPerSecond = 5793.8
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112142 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.1102s; samplesPerSecond = 5810.0
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800742 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.1101s; samplesPerSecond = 5812.0
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783400 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.1086s; samplesPerSecond = 5892.2
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590123 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.1105s; samplesPerSecond = 5793.1
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415391 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.1091s; samplesPerSecond = 5863.6
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.04696796 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=3.59201s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:23: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.14624175 * 2560; EvalClassificationError = 0.56953125 * 2560; time = 0.2319s; samplesPerSecond = 11037.6
|
||||
MPI Rank 2: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.06174128 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 0.2326s; samplesPerSecond = 11006.3
|
||||
MPI Rank 2: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.04994338 * 2560; EvalClassificationError = 0.55351562 * 2560; time = 0.2292s; samplesPerSecond = 11167.0
|
||||
MPI Rank 2: 12/15/2016 08:31:24: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.03695538 * 2560; EvalClassificationError = 0.56132812 * 2560; time = 0.2298s; samplesPerSecond = 11140.3
|
||||
MPI Rank 2: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.03086227 * 2560; EvalClassificationError = 0.55664063 * 2560; time = 0.2347s; samplesPerSecond = 10908.7
|
||||
MPI Rank 2: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 1.97306193 * 2560; EvalClassificationError = 0.53671875 * 2560; time = 0.2221s; samplesPerSecond = 11524.1
|
||||
MPI Rank 2: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 1.96746064 * 2560; EvalClassificationError = 0.53164062 * 2560; time = 0.2240s; samplesPerSecond = 11428.0
|
||||
MPI Rank 2: 12/15/2016 08:31:25: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 1.95498165 * 2560; EvalClassificationError = 0.53750000 * 2560; time = 0.2250s; samplesPerSecond = 11377.8
|
||||
MPI Rank 2: 12/15/2016 08:31:25: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.02765603 * 20480; EvalClassificationError = 0.55053711 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=1.83772s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:25: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:25: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:31:26: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.95358449 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 0.6863s; samplesPerSecond = 14920.6
|
||||
MPI Rank 2: 12/15/2016 08:31:27: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.97540911 * 10240; EvalClassificationError = 0.55253906 * 10240; time = 0.6837s; samplesPerSecond = 14977.5
|
||||
MPI Rank 2: 12/15/2016 08:31:27: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.96449680 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=1.37654s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:27: Action "train" complete.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:27: __COMPLETED__
|
||||
|
|
@ -1,438 +0,0 @@
|
|||
CPU info:
|
||||
CPU Model Name: Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
|
||||
Hardware threads: 4
|
||||
Total Memory: 12580404 kB
|
||||
-------------------------------------------------------------------
|
||||
=== Running C:\Program Files\Microsoft MPI\Bin\/mpiexec.exe -n 3 C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DeviceId=0 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:31
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DeviceId=0 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (1) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:31
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DeviceId=0 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (2) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:31
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DeviceId=0 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (0) are in (participating)
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr_speechTrain.logrank0
|
||||
MPI Rank 0: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:31
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DeviceId=0 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Using 1 CPU threads.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:32: ##############################################################################
|
||||
MPI Rank 0: 12/15/2016 08:31:32: # #
|
||||
MPI Rank 0: 12/15/2016 08:31:32: # speechTrain command (train action) #
|
||||
MPI Rank 0: 12/15/2016 08:31:32: # #
|
||||
MPI Rank 0: 12/15/2016 08:31:32: ##############################################################################
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:32:
|
||||
MPI Rank 0: Creating virgin network.
|
||||
MPI Rank 0: SimpleNetworkBuilder Using GPU 0
|
||||
MPI Rank 0: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 0: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 0: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 0: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 0: label set 0: 129 classes
|
||||
MPI Rank 0: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 0: 12/15/2016 08:31:32:
|
||||
MPI Rank 0: Model has 25 nodes. Using GPU 0.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { HLast : [132 x 1 x *]
|
||||
MPI Rank 0: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 0: { H2 : [512 x 1 x *]
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { H1 : [512 x 1 x *]
|
||||
MPI Rank 0: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 0: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 0: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 0: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Initializing dataParallelSGD with FP32 aggregation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:32: MeanOfFeatures = Mean()
|
||||
MPI Rank 0: 12/15/2016 08:31:32: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 0: 12/15/2016 08:31:32: Prior = Mean()
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:36: Precomputing --> Completed.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.53638635 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.1094s; samplesPerSecond = 5850.0
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.32517799 * 640; EvalClassificationError = 0.92500000 * 640; time = 0.0937s; samplesPerSecond = 6827.8
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98246287 * 640; EvalClassificationError = 0.87187500 * 640; time = 0.0945s; samplesPerSecond = 6774.2
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.73673607 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.0929s; samplesPerSecond = 6886.2
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.84021876 * 640; EvalClassificationError = 0.86406250 * 640; time = 0.0889s; samplesPerSecond = 7201.9
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.69831379 * 640; EvalClassificationError = 0.86250000 * 640; time = 0.0903s; samplesPerSecond = 7090.6
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.39593109 * 640; EvalClassificationError = 0.77031250 * 640; time = 0.0864s; samplesPerSecond = 7406.6
|
||||
MPI Rank 0: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.49749690 * 640; EvalClassificationError = 0.82968750 * 640; time = 0.0897s; samplesPerSecond = 7133.4
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.47295705 * 640; EvalClassificationError = 0.81093750 * 640; time = 0.0892s; samplesPerSecond = 7170.9
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.36483701 * 640; EvalClassificationError = 0.79843750 * 640; time = 0.0895s; samplesPerSecond = 7152.0
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.46790699 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.0918s; samplesPerSecond = 6971.0
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.22104761 * 640; EvalClassificationError = 0.75625000 * 640; time = 0.0919s; samplesPerSecond = 6963.3
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.12504348 * 640; EvalClassificationError = 0.75312500 * 640; time = 0.0914s; samplesPerSecond = 7004.8
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 2.99508080 * 640; EvalClassificationError = 0.71875000 * 640; time = 0.0911s; samplesPerSecond = 7024.9
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.89602893 * 640; EvalClassificationError = 0.70000000 * 640; time = 0.0908s; samplesPerSecond = 7050.7
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.04740234 * 640; EvalClassificationError = 0.74218750 * 640; time = 0.0922s; samplesPerSecond = 6944.6
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.75064617 * 640; EvalClassificationError = 0.69375000 * 640; time = 0.0884s; samplesPerSecond = 7239.7
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.65538372 * 640; EvalClassificationError = 0.63750000 * 640; time = 0.0895s; samplesPerSecond = 7148.4
|
||||
MPI Rank 0: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.74816083 * 640; EvalClassificationError = 0.69062500 * 640; time = 0.0881s; samplesPerSecond = 7261.0
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.68736721 * 640; EvalClassificationError = 0.68593750 * 640; time = 0.0911s; samplesPerSecond = 7028.6
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.53268728 * 640; EvalClassificationError = 0.64375000 * 640; time = 0.0901s; samplesPerSecond = 7099.4
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.53923335 * 640; EvalClassificationError = 0.63750000 * 640; time = 0.0903s; samplesPerSecond = 7090.4
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.48909476 * 640; EvalClassificationError = 0.64218750 * 640; time = 0.0900s; samplesPerSecond = 7114.0
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.50033044 * 640; EvalClassificationError = 0.65156250 * 640; time = 0.0882s; samplesPerSecond = 7256.9
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.43569647 * 640; EvalClassificationError = 0.63125000 * 640; time = 0.0899s; samplesPerSecond = 7116.3
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.34293090 * 640; EvalClassificationError = 0.61562500 * 640; time = 0.0888s; samplesPerSecond = 7208.4
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.20428060 * 640; EvalClassificationError = 0.57812500 * 640; time = 0.0909s; samplesPerSecond = 7044.3
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.46886817 * 640; EvalClassificationError = 0.65468750 * 640; time = 0.0901s; samplesPerSecond = 7101.4
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.22066720 * 640; EvalClassificationError = 0.58906250 * 640; time = 0.0897s; samplesPerSecond = 7131.5
|
||||
MPI Rank 0: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.21784279 * 640; EvalClassificationError = 0.60781250 * 640; time = 0.0910s; samplesPerSecond = 7036.7
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.20442240 * 640; EvalClassificationError = 0.57812500 * 640; time = 0.0897s; samplesPerSecond = 7133.3
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.18215676 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.0910s; samplesPerSecond = 7030.7
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 2.99321241 * 20480; EvalClassificationError = 0.72216797 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=2.92872s
|
||||
MPI Rank 0: 12/15/2016 08:31:40: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/models/cntkSpeech.dnn.1'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.08889867 * 2560; EvalClassificationError = 0.56367188 * 2560; time = 0.1253s; samplesPerSecond = 20437.7
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.00776227 * 2560; EvalClassificationError = 0.54218750 * 2560; time = 0.1002s; samplesPerSecond = 25542.3
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 1.99260187 * 2560; EvalClassificationError = 0.54257813 * 2560; time = 0.0969s; samplesPerSecond = 26424.2
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 1.98459924 * 2560; EvalClassificationError = 0.54648438 * 2560; time = 0.0954s; samplesPerSecond = 26820.9
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 1.97206449 * 2560; EvalClassificationError = 0.53984375 * 2560; time = 0.0955s; samplesPerSecond = 26809.9
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 1.91865552 * 2560; EvalClassificationError = 0.52109375 * 2560; time = 0.0978s; samplesPerSecond = 26175.9
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 1.91066643 * 2560; EvalClassificationError = 0.52148438 * 2560; time = 0.0953s; samplesPerSecond = 26854.1
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 1.89501440 * 2560; EvalClassificationError = 0.51992187 * 2560; time = 0.0961s; samplesPerSecond = 26633.4
|
||||
MPI Rank 0: 12/15/2016 08:31:40: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 1.97128286 * 20480; EvalClassificationError = 0.53715820 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=0.812502s
|
||||
MPI Rank 0: 12/15/2016 08:31:41: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/models/cntkSpeech.dnn.2'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:41: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:41: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 0: 12/15/2016 08:31:41: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.89820598 * 10240; EvalClassificationError = 0.52470703 * 10240; time = 0.1751s; samplesPerSecond = 58495.2
|
||||
MPI Rank 0: 12/15/2016 08:31:41: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.91958079 * 10240; EvalClassificationError = 0.53974609 * 10240; time = 0.1351s; samplesPerSecond = 75799.6
|
||||
MPI Rank 0: 12/15/2016 08:31:41: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.90889339 * 20480; EvalClassificationError = 0.53222656 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=0.319187s
|
||||
MPI Rank 0: 12/15/2016 08:31:41: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/models/cntkSpeech.dnn'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:41: Action "train" complete.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:41: __COMPLETED__
|
||||
MPI Rank 1: 12/15/2016 08:31:32: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr_speechTrain.logrank1
|
||||
MPI Rank 1: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:31
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DeviceId=0 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr
|
||||
MPI Rank 1: 12/15/2016 08:31:32: Using 1 CPU threads.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:32: ##############################################################################
|
||||
MPI Rank 1: 12/15/2016 08:31:32: # #
|
||||
MPI Rank 1: 12/15/2016 08:31:32: # speechTrain command (train action) #
|
||||
MPI Rank 1: 12/15/2016 08:31:32: # #
|
||||
MPI Rank 1: 12/15/2016 08:31:32: ##############################################################################
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:32:
|
||||
MPI Rank 1: Creating virgin network.
|
||||
MPI Rank 1: SimpleNetworkBuilder Using GPU 0
|
||||
MPI Rank 1: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 1: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 1: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 1: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 1: label set 0: 129 classes
|
||||
MPI Rank 1: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 1: 12/15/2016 08:31:33:
|
||||
MPI Rank 1: Model has 25 nodes. Using GPU 0.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { H1 : [512 x 1 x *]
|
||||
MPI Rank 1: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 1: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 1: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { H2 : [512 x 1 x *]
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 1: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 1: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { HLast : [132 x 1 x *]
|
||||
MPI Rank 1: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Initializing dataParallelSGD with FP32 aggregation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:33: MeanOfFeatures = Mean()
|
||||
MPI Rank 1: 12/15/2016 08:31:33: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 1: 12/15/2016 08:31:33: Prior = Mean()
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:36: Precomputing --> Completed.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.53638635 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.1091s; samplesPerSecond = 5868.8
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.32517799 * 640; EvalClassificationError = 0.92500000 * 640; time = 0.0950s; samplesPerSecond = 6739.8
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98246287 * 640; EvalClassificationError = 0.87187500 * 640; time = 0.0937s; samplesPerSecond = 6832.1
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.73673607 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.0930s; samplesPerSecond = 6879.0
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.84021876 * 640; EvalClassificationError = 0.86406250 * 640; time = 0.0888s; samplesPerSecond = 7206.4
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.69831379 * 640; EvalClassificationError = 0.86250000 * 640; time = 0.0903s; samplesPerSecond = 7089.0
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.39593109 * 640; EvalClassificationError = 0.77031250 * 640; time = 0.0863s; samplesPerSecond = 7411.8
|
||||
MPI Rank 1: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.49749690 * 640; EvalClassificationError = 0.82968750 * 640; time = 0.0894s; samplesPerSecond = 7157.7
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.47295705 * 640; EvalClassificationError = 0.81093750 * 640; time = 0.0901s; samplesPerSecond = 7104.1
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.36483701 * 640; EvalClassificationError = 0.79843750 * 640; time = 0.0894s; samplesPerSecond = 7155.7
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.46790699 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.0906s; samplesPerSecond = 7065.7
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.22104761 * 640; EvalClassificationError = 0.75625000 * 640; time = 0.0931s; samplesPerSecond = 6872.9
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.12504348 * 640; EvalClassificationError = 0.75312500 * 640; time = 0.0906s; samplesPerSecond = 7066.6
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 2.99508080 * 640; EvalClassificationError = 0.71875000 * 640; time = 0.0907s; samplesPerSecond = 7052.8
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.89602893 * 640; EvalClassificationError = 0.70000000 * 640; time = 0.0923s; samplesPerSecond = 6932.6
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.04740234 * 640; EvalClassificationError = 0.74218750 * 640; time = 0.0911s; samplesPerSecond = 7025.9
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.75064617 * 640; EvalClassificationError = 0.69375000 * 640; time = 0.0880s; samplesPerSecond = 7271.8
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.65538372 * 640; EvalClassificationError = 0.63750000 * 640; time = 0.0899s; samplesPerSecond = 7120.0
|
||||
MPI Rank 1: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.74816083 * 640; EvalClassificationError = 0.69062500 * 640; time = 0.0877s; samplesPerSecond = 7298.0
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.68736721 * 640; EvalClassificationError = 0.68593750 * 640; time = 0.0927s; samplesPerSecond = 6906.4
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.53268728 * 640; EvalClassificationError = 0.64375000 * 640; time = 0.0885s; samplesPerSecond = 7228.0
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.53923335 * 640; EvalClassificationError = 0.63750000 * 640; time = 0.0902s; samplesPerSecond = 7092.0
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.48909476 * 640; EvalClassificationError = 0.64218750 * 640; time = 0.0912s; samplesPerSecond = 7019.3
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.50033044 * 640; EvalClassificationError = 0.65156250 * 640; time = 0.0874s; samplesPerSecond = 7325.8
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.43569647 * 640; EvalClassificationError = 0.63125000 * 640; time = 0.0912s; samplesPerSecond = 7019.3
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.34293090 * 640; EvalClassificationError = 0.61562500 * 640; time = 0.0884s; samplesPerSecond = 7240.9
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.20428060 * 640; EvalClassificationError = 0.57812500 * 640; time = 0.0897s; samplesPerSecond = 7137.8
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.46886817 * 640; EvalClassificationError = 0.65468750 * 640; time = 0.0913s; samplesPerSecond = 7010.4
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.22066720 * 640; EvalClassificationError = 0.58906250 * 640; time = 0.0902s; samplesPerSecond = 7096.1
|
||||
MPI Rank 1: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.21784279 * 640; EvalClassificationError = 0.60781250 * 640; time = 0.0910s; samplesPerSecond = 7034.0
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.20442240 * 640; EvalClassificationError = 0.57812500 * 640; time = 0.0898s; samplesPerSecond = 7130.7
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.18215676 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.0894s; samplesPerSecond = 7157.3
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 2.99321241 * 20480; EvalClassificationError = 0.72216797 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=2.9298s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.08889867 * 2560; EvalClassificationError = 0.56367188 * 2560; time = 0.1257s; samplesPerSecond = 20373.6
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.00776227 * 2560; EvalClassificationError = 0.54218750 * 2560; time = 0.1007s; samplesPerSecond = 25418.5
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 1.99260187 * 2560; EvalClassificationError = 0.54257813 * 2560; time = 0.0953s; samplesPerSecond = 26873.8
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 1.98459924 * 2560; EvalClassificationError = 0.54648438 * 2560; time = 0.0954s; samplesPerSecond = 26825.4
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 1.97206449 * 2560; EvalClassificationError = 0.53984375 * 2560; time = 0.0955s; samplesPerSecond = 26815.5
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 1.91865552 * 2560; EvalClassificationError = 0.52109375 * 2560; time = 0.0990s; samplesPerSecond = 25855.5
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 1.91066643 * 2560; EvalClassificationError = 0.52148438 * 2560; time = 0.0941s; samplesPerSecond = 27206.3
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 1.89501440 * 2560; EvalClassificationError = 0.51992187 * 2560; time = 0.0976s; samplesPerSecond = 26220.6
|
||||
MPI Rank 1: 12/15/2016 08:31:40: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 1.97128286 * 20480; EvalClassificationError = 0.53715820 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=0.813236s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:41: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:41: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 1: 12/15/2016 08:31:41: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.89820598 * 10240; EvalClassificationError = 0.52470703 * 10240; time = 0.1746s; samplesPerSecond = 58641.3
|
||||
MPI Rank 1: 12/15/2016 08:31:41: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.91958079 * 10240; EvalClassificationError = 0.53974609 * 10240; time = 0.1351s; samplesPerSecond = 75822.1
|
||||
MPI Rank 1: 12/15/2016 08:31:41: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.90889339 * 20480; EvalClassificationError = 0.53222656 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=0.318744s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:41: Action "train" complete.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:41: __COMPLETED__
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr_speechTrain.logrank2
|
||||
MPI Rank 2: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:31
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu DeviceId=0 timestamping=true numCPUThreads=1 stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantization@release_gpu/stderr
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Using 1 CPU threads.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:33: ##############################################################################
|
||||
MPI Rank 2: 12/15/2016 08:31:33: # #
|
||||
MPI Rank 2: 12/15/2016 08:31:33: # speechTrain command (train action) #
|
||||
MPI Rank 2: 12/15/2016 08:31:33: # #
|
||||
MPI Rank 2: 12/15/2016 08:31:33: ##############################################################################
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:33:
|
||||
MPI Rank 2: Creating virgin network.
|
||||
MPI Rank 2: SimpleNetworkBuilder Using GPU 0
|
||||
MPI Rank 2: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 2: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 2: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 2: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 2: label set 0: 129 classes
|
||||
MPI Rank 2: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 2: 12/15/2016 08:31:33:
|
||||
MPI Rank 2: Model has 25 nodes. Using GPU 0.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 2: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 2: { H2 : [512 x 1 x *]
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { H1 : [512 x 1 x *]
|
||||
MPI Rank 2: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 2: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { HLast : [132 x 1 x *]
|
||||
MPI Rank 2: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 2: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Initializing dataParallelSGD with FP32 aggregation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:33: MeanOfFeatures = Mean()
|
||||
MPI Rank 2: 12/15/2016 08:31:33: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 2: 12/15/2016 08:31:33: Prior = Mean()
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Precomputing --> Completed.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.53638635 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.0989s; samplesPerSecond = 6468.4
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.32517799 * 640; EvalClassificationError = 0.92500000 * 640; time = 0.0938s; samplesPerSecond = 6825.4
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98246287 * 640; EvalClassificationError = 0.87187500 * 640; time = 0.0941s; samplesPerSecond = 6803.7
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.73673607 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.0930s; samplesPerSecond = 6884.7
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.84021876 * 640; EvalClassificationError = 0.86406250 * 640; time = 0.0888s; samplesPerSecond = 7204.3
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.69831379 * 640; EvalClassificationError = 0.86250000 * 640; time = 0.0915s; samplesPerSecond = 6996.5
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.39593109 * 640; EvalClassificationError = 0.77031250 * 640; time = 0.0864s; samplesPerSecond = 7409.8
|
||||
MPI Rank 2: 12/15/2016 08:31:37: Epoch[ 1 of 3]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.49749690 * 640; EvalClassificationError = 0.82968750 * 640; time = 0.0891s; samplesPerSecond = 7181.5
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.47295705 * 640; EvalClassificationError = 0.81093750 * 640; time = 0.0896s; samplesPerSecond = 7139.6
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.36483701 * 640; EvalClassificationError = 0.79843750 * 640; time = 0.0894s; samplesPerSecond = 7158.1
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.46790699 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.0918s; samplesPerSecond = 6973.9
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.22104761 * 640; EvalClassificationError = 0.75625000 * 640; time = 0.0920s; samplesPerSecond = 6959.7
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.12504348 * 640; EvalClassificationError = 0.75312500 * 640; time = 0.0909s; samplesPerSecond = 7038.5
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 2.99508080 * 640; EvalClassificationError = 0.71875000 * 640; time = 0.0915s; samplesPerSecond = 6992.9
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.89602893 * 640; EvalClassificationError = 0.70000000 * 640; time = 0.0915s; samplesPerSecond = 6993.1
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.04740234 * 640; EvalClassificationError = 0.74218750 * 640; time = 0.0911s; samplesPerSecond = 7025.1
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.75064617 * 640; EvalClassificationError = 0.69375000 * 640; time = 0.0888s; samplesPerSecond = 7209.2
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.65538372 * 640; EvalClassificationError = 0.63750000 * 640; time = 0.0904s; samplesPerSecond = 7082.5
|
||||
MPI Rank 2: 12/15/2016 08:31:38: Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.74816083 * 640; EvalClassificationError = 0.69062500 * 640; time = 0.0873s; samplesPerSecond = 7330.5
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.68736721 * 640; EvalClassificationError = 0.68593750 * 640; time = 0.0919s; samplesPerSecond = 6965.8
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.53268728 * 640; EvalClassificationError = 0.64375000 * 640; time = 0.0894s; samplesPerSecond = 7162.0
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.53923335 * 640; EvalClassificationError = 0.63750000 * 640; time = 0.0903s; samplesPerSecond = 7089.6
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.48909476 * 640; EvalClassificationError = 0.64218750 * 640; time = 0.0900s; samplesPerSecond = 7112.1
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.50033044 * 640; EvalClassificationError = 0.65156250 * 640; time = 0.0878s; samplesPerSecond = 7287.1
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.43569647 * 640; EvalClassificationError = 0.63125000 * 640; time = 0.0910s; samplesPerSecond = 7030.7
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.34293090 * 640; EvalClassificationError = 0.61562500 * 640; time = 0.0892s; samplesPerSecond = 7171.9
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.20428060 * 640; EvalClassificationError = 0.57812500 * 640; time = 0.0896s; samplesPerSecond = 7139.4
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.46886817 * 640; EvalClassificationError = 0.65468750 * 640; time = 0.0901s; samplesPerSecond = 7102.0
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.22066720 * 640; EvalClassificationError = 0.58906250 * 640; time = 0.0905s; samplesPerSecond = 7068.1
|
||||
MPI Rank 2: 12/15/2016 08:31:39: Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.21784279 * 640; EvalClassificationError = 0.60781250 * 640; time = 0.0910s; samplesPerSecond = 7034.0
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.20442240 * 640; EvalClassificationError = 0.57812500 * 640; time = 0.0898s; samplesPerSecond = 7130.6
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.18215676 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.0902s; samplesPerSecond = 7092.1
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 2.99321241 * 20480; EvalClassificationError = 0.72216797 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=2.92432s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.08889867 * 2560; EvalClassificationError = 0.56367188 * 2560; time = 0.1131s; samplesPerSecond = 22628.0
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.00776227 * 2560; EvalClassificationError = 0.54218750 * 2560; time = 0.1011s; samplesPerSecond = 25323.2
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 1.99260187 * 2560; EvalClassificationError = 0.54257813 * 2560; time = 0.0961s; samplesPerSecond = 26645.6
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 1.98459924 * 2560; EvalClassificationError = 0.54648438 * 2560; time = 0.0955s; samplesPerSecond = 26819.8
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 1.97206449 * 2560; EvalClassificationError = 0.53984375 * 2560; time = 0.0955s; samplesPerSecond = 26808.8
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 1.91865552 * 2560; EvalClassificationError = 0.52109375 * 2560; time = 0.0978s; samplesPerSecond = 26169.4
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 1.91066643 * 2560; EvalClassificationError = 0.52148438 * 2560; time = 0.0953s; samplesPerSecond = 26865.9
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Epoch[ 2 of 3]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 1.89501440 * 2560; EvalClassificationError = 0.51992187 * 2560; time = 0.0968s; samplesPerSecond = 26434.8
|
||||
MPI Rank 2: 12/15/2016 08:31:40: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 1.97128286 * 20480; EvalClassificationError = 0.53715820 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=0.811981s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:41: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:41: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 32), distributed reading is ENABLED.
|
||||
MPI Rank 2: 12/15/2016 08:31:41: Epoch[ 3 of 3]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.89820598 * 10240; EvalClassificationError = 0.52470703 * 10240; time = 0.1679s; samplesPerSecond = 60975.6
|
||||
MPI Rank 2: 12/15/2016 08:31:41: Epoch[ 3 of 3]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.91958079 * 10240; EvalClassificationError = 0.53974609 * 10240; time = 0.1350s; samplesPerSecond = 75841.2
|
||||
MPI Rank 2: 12/15/2016 08:31:41: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.90889339 * 20480; EvalClassificationError = 0.53222656 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=0.317066s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:41: Action "train" complete.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:41: __COMPLETED__
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
. $TEST_ROOT_DIR/run-test-common
|
||||
|
||||
OriginalTestDir=../../../DNN/ParallelNoQuantization
|
||||
ConfigDir=$TEST_DIR/../../../DNN
|
||||
LogFileName=stderr
|
||||
Instances=3
|
||||
NumCPUThreads=$(threadsPerInstance $Instances)
|
||||
|
||||
(cd $TEST_DIR/$OriginalTestDir && md5sum baseline*) | (cd $TEST_DIR && md5sum --status -c -)
|
||||
if [ $? != 0 ]; then
|
||||
echo Error: Baselines must match original test. Copy from $OriginalTestDir.
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# cntkmpirun <MPI args> <CNTK config file name> <additional CNTK args>
|
||||
cntkmpirun "-n $Instances" cntk.cntk "speechTrain=[reader=[readerType=HTKDeserializers]] numCPUThreads=$NumCPUThreads"
|
||||
ExitCode=$?
|
||||
sed 's/^/MPI Rank 0: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank0
|
||||
sed 's/^/MPI Rank 1: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank1
|
||||
sed 's/^/MPI Rank 2: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank2
|
||||
exit $ExitCode
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
dataDir: ../../../Data
|
||||
tags:
|
||||
# running on every BVT job in 'P' (Speech) leg in Debug-GPU and Release-CPU configurations:
|
||||
# - bvt-p ((build_sku == 'gpu') or (build_sku == '1bitsgd')) and ((flavor == 'release') if (os == 'windows') else ((flavor == 'debug') ^ (device == 'cpu')))
|
||||
# running unconditionally on every Nightly job in 'P' leg
|
||||
- nightly-p ((build_sku == 'gpu') or (build_sku == '1bitsgd'))
|
||||
|
||||
testCases:
|
||||
Must train epochs in exactly same order and parameters for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting Epoch {{integer}}
|
||||
- learning rate per sample = {{float}}
|
||||
- momentum = {{float}}
|
||||
|
||||
Epochs must be finished with expected results for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Finished Epoch[{{integer}} of {{integer}}]
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0.01%}}
|
||||
- EvalClassificationError = {{float,tolerance=0.01%}}
|
||||
- learningRatePerSample = {{float,tolerance=0.001%}}
|
||||
|
||||
Per-minibatch training results must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Epoch[{{integer}} of {{integer}}]-Minibatch[{{integer}}-{{integer}}
|
||||
- " * {{integer}}; "
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0.01%}}
|
||||
- EvalClassificationError = {{float,tolerance=0.01%}}
|
||||
|
||||
DataParallelSGD training parameters must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting minibatch loop
|
||||
- DataParallelSGD training
|
||||
- myRank = {{integer}}
|
||||
- numNodes = 3
|
||||
- numGradientBits = 32
|
||||
- distributed reading is ENABLED
|
||||
|
|
@ -1,613 +0,0 @@
|
|||
CPU info:
|
||||
CPU Model Name: Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
|
||||
Hardware threads: 4
|
||||
Total Memory: 12580404 kB
|
||||
-------------------------------------------------------------------
|
||||
=== Running C:\Program Files\Microsoft MPI\Bin\/mpiexec.exe -n 3 C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:46
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (1) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:45
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (2) are in (participating)
|
||||
CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:46
|
||||
|
||||
C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
Changed current directory to C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data
|
||||
requestnodes [MPIWrapper]: using 3 out of 3 MPI nodes on a single host (3 requested); we (0) are in (participating)
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr_speechTrain.logrank0
|
||||
MPI Rank 0: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:46
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Using 1 CPU threads.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:46: ##############################################################################
|
||||
MPI Rank 0: 12/15/2016 08:31:46: # #
|
||||
MPI Rank 0: 12/15/2016 08:31:46: # speechTrain command (train action) #
|
||||
MPI Rank 0: 12/15/2016 08:31:46: # #
|
||||
MPI Rank 0: 12/15/2016 08:31:46: ##############################################################################
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:46:
|
||||
MPI Rank 0: Creating virgin network.
|
||||
MPI Rank 0: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 0: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 0: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 0: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 0: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 0: label set 0: 129 classes
|
||||
MPI Rank 0: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 0: 12/15/2016 08:31:46:
|
||||
MPI Rank 0: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 0: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 0: { H2 : [512 x 1 x *]
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 0: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 0: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 0: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 0: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 0: { H1 : [512 x 1 x *]
|
||||
MPI Rank 0: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 0: { HLast : [132 x 1 x *]
|
||||
MPI Rank 0: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: Initializing dataParallelSGD with FP64 aggregation.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:46: MeanOfFeatures = Mean()
|
||||
MPI Rank 0: 12/15/2016 08:31:46: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 0: 12/15/2016 08:31:46: Prior = Mean()
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:49: Precomputing --> Completed.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:50: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:50: Starting minibatch loop.
|
||||
MPI Rank 0: 12/15/2016 08:31:50: Epoch[ 1 of 4]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.3011s; samplesPerSecond = 2125.9
|
||||
MPI Rank 0: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2555s; samplesPerSecond = 2505.3
|
||||
MPI Rank 0: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2622s; samplesPerSecond = 2441.2
|
||||
MPI Rank 0: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2643s; samplesPerSecond = 2421.2
|
||||
MPI Rank 0: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2635s; samplesPerSecond = 2428.9
|
||||
MPI Rank 0: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2628s; samplesPerSecond = 2435.1
|
||||
MPI Rank 0: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2533s; samplesPerSecond = 2526.9
|
||||
MPI Rank 0: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2528s; samplesPerSecond = 2532.0
|
||||
MPI Rank 0: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2544s; samplesPerSecond = 2516.2
|
||||
MPI Rank 0: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2536s; samplesPerSecond = 2523.9
|
||||
MPI Rank 0: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2549s; samplesPerSecond = 2510.6
|
||||
MPI Rank 0: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2537s; samplesPerSecond = 2522.5
|
||||
MPI Rank 0: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2479s; samplesPerSecond = 2581.4
|
||||
MPI Rank 0: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2515s; samplesPerSecond = 2544.8
|
||||
MPI Rank 0: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2818s; samplesPerSecond = 2270.8
|
||||
MPI Rank 0: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2569s; samplesPerSecond = 2491.5
|
||||
MPI Rank 0: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2631s; samplesPerSecond = 2432.4
|
||||
MPI Rank 0: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2709s; samplesPerSecond = 2362.6
|
||||
MPI Rank 0: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2640s; samplesPerSecond = 2424.3
|
||||
MPI Rank 0: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2571s; samplesPerSecond = 2489.6
|
||||
MPI Rank 0: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2532s; samplesPerSecond = 2527.7
|
||||
MPI Rank 0: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2541s; samplesPerSecond = 2518.4
|
||||
MPI Rank 0: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2543s; samplesPerSecond = 2516.2
|
||||
MPI Rank 0: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2625s; samplesPerSecond = 2437.8
|
||||
MPI Rank 0: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2599s; samplesPerSecond = 2462.6
|
||||
MPI Rank 0: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2540s; samplesPerSecond = 2519.3
|
||||
MPI Rank 0: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2558s; samplesPerSecond = 2502.1
|
||||
MPI Rank 0: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2505s; samplesPerSecond = 2555.2
|
||||
MPI Rank 0: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2525s; samplesPerSecond = 2534.8
|
||||
MPI Rank 0: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2509s; samplesPerSecond = 2550.6
|
||||
MPI Rank 0: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2521s; samplesPerSecond = 2538.7
|
||||
MPI Rank 0: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2508s; samplesPerSecond = 2551.9
|
||||
MPI Rank 0: 12/15/2016 08:31:58: Finished Epoch[ 1 of 4]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.28871s
|
||||
MPI Rank 0: 12/15/2016 08:31:58: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn.1'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:58: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:31:58: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.029024
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.011366
|
||||
MPI Rank 0: 12/15/2016 08:31:59: Epoch[ 2 of 4]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.19109241 * 2304; EvalClassificationError = 0.58246528 * 2304; time = 0.4068s; samplesPerSecond = 5663.4
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.009713
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.007983
|
||||
MPI Rank 0: 12/15/2016 08:31:59: Epoch[ 2 of 4]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.20697464 * 2560; EvalClassificationError = 0.59453125 * 2560; time = 0.4134s; samplesPerSecond = 6193.0
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.008095
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.009539
|
||||
MPI Rank 0: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.23618716 * 2560; EvalClassificationError = 0.60039062 * 2560; time = 0.4040s; samplesPerSecond = 6336.0
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.007831
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.007867
|
||||
MPI Rank 0: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.21810382 * 2560; EvalClassificationError = 0.59609375 * 2560; time = 0.4054s; samplesPerSecond = 6314.1
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.007854
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.007794
|
||||
MPI Rank 0: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.17778205 * 2560; EvalClassificationError = 0.59414062 * 2560; time = 0.3964s; samplesPerSecond = 6458.2
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.013175
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.012979
|
||||
MPI Rank 0: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.13452559 * 2560; EvalClassificationError = 0.57734375 * 2560; time = 0.3897s; samplesPerSecond = 6568.9
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.007794
|
||||
MPI Rank 0: Async gradient aggregation wait time: 1e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.007836
|
||||
MPI Rank 0: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.13087789 * 2560; EvalClassificationError = 0.57265625 * 2560; time = 0.3849s; samplesPerSecond = 6650.9
|
||||
MPI Rank 0: Async gradient aggregation wait time: 1e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.011382
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.008394
|
||||
MPI Rank 0: 12/15/2016 08:32:02: Epoch[ 2 of 4]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.11200101 * 2560; EvalClassificationError = 0.58632812 * 2560; time = 0.3899s; samplesPerSecond = 6566.0
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.00391
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.006681
|
||||
MPI Rank 0: 12/15/2016 08:32:02: Finished Epoch[ 2 of 4]: [Training] CrossEntropyWithSoftmax = 2.17402050 * 20480; EvalClassificationError = 0.58750000 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=3.21196s
|
||||
MPI Rank 0: 12/15/2016 08:32:02: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn.2'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:32:02: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:32:02: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.00922
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.009044
|
||||
MPI Rank 0: 12/15/2016 08:32:03: Epoch[ 3 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 2.15723941 * 9216; EvalClassificationError = 0.56488715 * 9216; time = 1.3161s; samplesPerSecond = 7002.6
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.022458
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.016563
|
||||
MPI Rank 0: 12/15/2016 08:32:04: Epoch[ 3 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 2.02453665 * 10240; EvalClassificationError = 0.55771484 * 10240; time = 1.2215s; samplesPerSecond = 8383.3
|
||||
MPI Rank 0: 12/15/2016 08:32:04: Finished Epoch[ 3 of 4]: [Training] CrossEntropyWithSoftmax = 2.08437881 * 20480; EvalClassificationError = 0.56079102 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.55852s
|
||||
MPI Rank 0: 12/15/2016 08:32:04: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn.3'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:32:04: Starting Epoch 4: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 0: minibatchiterator: epoch 3: frames [61440..81920] (first utterance at frame 61440), data subset 0 of 3, with 1 datapasses
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:32:04: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.012264
|
||||
MPI Rank 0: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.037037
|
||||
MPI Rank 0: 12/15/2016 08:32:05: Epoch[ 4 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.96502938 * 9216; EvalClassificationError = 0.53190104 * 9216; time = 1.2404s; samplesPerSecond = 7430.0
|
||||
MPI Rank 0: Async gradient aggregation wait time: 4e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.048557
|
||||
MPI Rank 0: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 0: Actual gradient aggregation time: 0.135479
|
||||
MPI Rank 0: 12/15/2016 08:32:07: Epoch[ 4 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.95947098 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 1.1937s; samplesPerSecond = 8578.4
|
||||
MPI Rank 0: Async gradient aggregation wait time: 0.008052
|
||||
MPI Rank 0: 12/15/2016 08:32:07: Finished Epoch[ 4 of 4]: [Training] CrossEntropyWithSoftmax = 1.96369079 * 20480; EvalClassificationError = 0.53471680 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-005; epochTime=2.5704s
|
||||
MPI Rank 0: 12/15/2016 08:32:07: SGD: Saving checkpoint model 'C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/models/cntkSpeech.dnn'
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:32:07: Action "train" complete.
|
||||
MPI Rank 0:
|
||||
MPI Rank 0: 12/15/2016 08:32:07: __COMPLETED__
|
||||
MPI Rank 1: 12/15/2016 08:31:46: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr_speechTrain.logrank1
|
||||
MPI Rank 1: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:46
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
MPI Rank 1: 12/15/2016 08:31:46: Using 1 CPU threads.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:46: ##############################################################################
|
||||
MPI Rank 1: 12/15/2016 08:31:46: # #
|
||||
MPI Rank 1: 12/15/2016 08:31:46: # speechTrain command (train action) #
|
||||
MPI Rank 1: 12/15/2016 08:31:46: # #
|
||||
MPI Rank 1: 12/15/2016 08:31:46: ##############################################################################
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:46:
|
||||
MPI Rank 1: Creating virgin network.
|
||||
MPI Rank 1: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 1: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 1: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 1: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 1: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 1: label set 0: 129 classes
|
||||
MPI Rank 1: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 1: 12/15/2016 08:31:47:
|
||||
MPI Rank 1: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 1: { H1 : [512 x 1 x *]
|
||||
MPI Rank 1: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 1: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 1: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 1: { H2 : [512 x 1 x *]
|
||||
MPI Rank 1: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 1: { HLast : [132 x 1 x *]
|
||||
MPI Rank 1: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 1: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 1: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 1: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 1: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: Initializing dataParallelSGD with FP64 aggregation.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:47: MeanOfFeatures = Mean()
|
||||
MPI Rank 1: 12/15/2016 08:31:47: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 1: 12/15/2016 08:31:47: Prior = Mean()
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:49: Precomputing --> Completed.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:50: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:50: Starting minibatch loop.
|
||||
MPI Rank 1: 12/15/2016 08:31:50: Epoch[ 1 of 4]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2852s; samplesPerSecond = 2244.1
|
||||
MPI Rank 1: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2670s; samplesPerSecond = 2396.8
|
||||
MPI Rank 1: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2622s; samplesPerSecond = 2440.9
|
||||
MPI Rank 1: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2643s; samplesPerSecond = 2421.9
|
||||
MPI Rank 1: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2647s; samplesPerSecond = 2417.9
|
||||
MPI Rank 1: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2584s; samplesPerSecond = 2476.9
|
||||
MPI Rank 1: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2513s; samplesPerSecond = 2546.6
|
||||
MPI Rank 1: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2509s; samplesPerSecond = 2550.5
|
||||
MPI Rank 1: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2514s; samplesPerSecond = 2546.1
|
||||
MPI Rank 1: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2542s; samplesPerSecond = 2517.8
|
||||
MPI Rank 1: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2542s; samplesPerSecond = 2517.8
|
||||
MPI Rank 1: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2548s; samplesPerSecond = 2511.4
|
||||
MPI Rank 1: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2502s; samplesPerSecond = 2558.4
|
||||
MPI Rank 1: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2521s; samplesPerSecond = 2538.3
|
||||
MPI Rank 1: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2861s; samplesPerSecond = 2237.3
|
||||
MPI Rank 1: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2576s; samplesPerSecond = 2484.3
|
||||
MPI Rank 1: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2539s; samplesPerSecond = 2520.4
|
||||
MPI Rank 1: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2546s; samplesPerSecond = 2513.7
|
||||
MPI Rank 1: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2671s; samplesPerSecond = 2396.0
|
||||
MPI Rank 1: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2600s; samplesPerSecond = 2461.4
|
||||
MPI Rank 1: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2583s; samplesPerSecond = 2478.2
|
||||
MPI Rank 1: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2632s; samplesPerSecond = 2431.4
|
||||
MPI Rank 1: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2588s; samplesPerSecond = 2473.3
|
||||
MPI Rank 1: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2669s; samplesPerSecond = 2398.2
|
||||
MPI Rank 1: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2586s; samplesPerSecond = 2474.4
|
||||
MPI Rank 1: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2501s; samplesPerSecond = 2559.0
|
||||
MPI Rank 1: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2513s; samplesPerSecond = 2547.1
|
||||
MPI Rank 1: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2536s; samplesPerSecond = 2523.9
|
||||
MPI Rank 1: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2534s; samplesPerSecond = 2525.9
|
||||
MPI Rank 1: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2532s; samplesPerSecond = 2527.3
|
||||
MPI Rank 1: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2518s; samplesPerSecond = 2541.6
|
||||
MPI Rank 1: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2511s; samplesPerSecond = 2548.7
|
||||
MPI Rank 1: 12/15/2016 08:31:58: Finished Epoch[ 1 of 4]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.28342s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:58: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:31:58: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.030247
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.017114
|
||||
MPI Rank 1: 12/15/2016 08:31:59: Epoch[ 2 of 4]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.19109241 * 2304; EvalClassificationError = 0.58246528 * 2304; time = 0.3927s; samplesPerSecond = 5866.6
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.028061
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.005708
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.044276
|
||||
MPI Rank 1: 12/15/2016 08:31:59: Epoch[ 2 of 4]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.20697464 * 2560; EvalClassificationError = 0.59453125 * 2560; time = 0.3941s; samplesPerSecond = 6495.0
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.009357
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.039388
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.005254
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.036219
|
||||
MPI Rank 1: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.23618716 * 2560; EvalClassificationError = 0.60039062 * 2560; time = 0.4050s; samplesPerSecond = 6321.0
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.002649
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.043082
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.006427
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.038851
|
||||
MPI Rank 1: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.21810382 * 2560; EvalClassificationError = 0.59609375 * 2560; time = 0.4063s; samplesPerSecond = 6301.0
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.00754
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.040924
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.008271
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.040546
|
||||
MPI Rank 1: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.17778205 * 2560; EvalClassificationError = 0.59414062 * 2560; time = 0.4008s; samplesPerSecond = 6387.6
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.000114
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.038789
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.035122
|
||||
MPI Rank 1: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.13452559 * 2560; EvalClassificationError = 0.57734375 * 2560; time = 0.3869s; samplesPerSecond = 6617.2
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.00517
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.037537
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.003319
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.037987
|
||||
MPI Rank 1: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.13087789 * 2560; EvalClassificationError = 0.57265625 * 2560; time = 0.3933s; samplesPerSecond = 6509.3
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.00473
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.038653
|
||||
MPI Rank 1: Async gradient aggregation wait time: 7e-005
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.034378
|
||||
MPI Rank 1: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.11200101 * 2560; EvalClassificationError = 0.58632812 * 2560; time = 0.3829s; samplesPerSecond = 6686.4
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.032732
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.012683
|
||||
MPI Rank 1: 12/15/2016 08:32:02: Finished Epoch[ 2 of 4]: [Training] CrossEntropyWithSoftmax = 2.17402050 * 20480; EvalClassificationError = 0.58750000 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=3.21197s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:32:02: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:32:02: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 1: Async gradient aggregation wait time: 1e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.037301
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.082708
|
||||
MPI Rank 1: 12/15/2016 08:32:03: Epoch[ 3 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 2.15723941 * 9216; EvalClassificationError = 0.56488715 * 9216; time = 1.2386s; samplesPerSecond = 7440.4
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.112156
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.075956
|
||||
MPI Rank 1: 12/15/2016 08:32:04: Epoch[ 3 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 2.02453665 * 10240; EvalClassificationError = 0.55771484 * 10240; time = 1.2680s; samplesPerSecond = 8075.8
|
||||
MPI Rank 1: 12/15/2016 08:32:04: Finished Epoch[ 3 of 4]: [Training] CrossEntropyWithSoftmax = 2.08437881 * 20480; EvalClassificationError = 0.56079102 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.55944s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:32:04: Starting Epoch 4: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 1: minibatchiterator: epoch 3: frames [61440..81920] (first utterance at frame 61440), data subset 1 of 3, with 1 datapasses
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:32:04: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 1: Async gradient aggregation wait time: 4e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.023745
|
||||
MPI Rank 1: Async gradient aggregation wait time: 2e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.008013
|
||||
MPI Rank 1: 12/15/2016 08:32:06: Epoch[ 4 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.96502938 * 9216; EvalClassificationError = 0.53190104 * 9216; time = 1.2758s; samplesPerSecond = 7223.5
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.01483
|
||||
MPI Rank 1: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 1: Actual gradient aggregation time: 0.028558
|
||||
MPI Rank 1: 12/15/2016 08:32:07: Epoch[ 4 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.95947098 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 1.2730s; samplesPerSecond = 8044.2
|
||||
MPI Rank 1: Async gradient aggregation wait time: 0.005241
|
||||
MPI Rank 1: 12/15/2016 08:32:07: Finished Epoch[ 4 of 4]: [Training] CrossEntropyWithSoftmax = 1.96369079 * 20480; EvalClassificationError = 0.53471680 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-005; epochTime=2.57039s
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:32:07: Action "train" complete.
|
||||
MPI Rank 1:
|
||||
MPI Rank 1: 12/15/2016 08:32:07: __COMPLETED__
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Redirecting stderr to file C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr_speechTrain.logrank2
|
||||
MPI Rank 2: CNTK 2.0.beta6.0+ (HEAD 5f1fab, Dec 15 2016 06:29:34) on cntk-muc03 at 2016/12/15 08:31:45
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: C:\jenkins\workspace\CNTK-Test-Windows-W1\x64\release\cntk.exe configFile=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN/cntk.cntk currentDirectory=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data RunDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DataDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data ConfigDir=C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\DNN OutputDir=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu DeviceId=-1 timestamping=true numCPUThreads=1 precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]] stderr=C:\Users\svcphil\AppData\Local\Temp\cntk-test-20161215082748.614918\Speech\DNN_ParallelNoQuantizationBufferedAsyncGradientAggregation@release_cpu/stderr
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Using 1 CPU threads.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:47: ##############################################################################
|
||||
MPI Rank 2: 12/15/2016 08:31:47: # #
|
||||
MPI Rank 2: 12/15/2016 08:31:47: # speechTrain command (train action) #
|
||||
MPI Rank 2: 12/15/2016 08:31:47: # #
|
||||
MPI Rank 2: 12/15/2016 08:31:47: ##############################################################################
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:47:
|
||||
MPI Rank 2: Creating virgin network.
|
||||
MPI Rank 2: SimpleNetworkBuilder Using CPU
|
||||
MPI Rank 2: reading script file glob_0000.scp ... 948 entries
|
||||
MPI Rank 2: total 132 state names in state list C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/state.list
|
||||
MPI Rank 2: htkmlfreader: reading MLF file C:\jenkins\workspace\CNTK-Test-Windows-W1\Tests\EndToEndTests\Speech\Data/glob_0000.mlf ... total 948 entries
|
||||
MPI Rank 2: ...............................................................................................feature set 0: 252734 frames in 948 out of 948 utterances
|
||||
MPI Rank 2: label set 0: 129 classes
|
||||
MPI Rank 2: minibatchutterancesource: 948 utterances grouped into 3 chunks, av. chunk size: 316.0 utterances, 84244.7 frames
|
||||
MPI Rank 2: 12/15/2016 08:31:47:
|
||||
MPI Rank 2: Model has 25 nodes. Using CPU.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Training criterion: CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Evaluation criterion: EvalClassificationError = ClassificationError
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Allocating matrices for forward and/or backward propagation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Memory Sharing: Out of 40 matrices, 19 are shared as 8, and 21 are not shared.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: { HLast : [132 x 1 x *]
|
||||
MPI Rank 2: W2 : [132 x 512] (gradient) }
|
||||
MPI Rank 2: { W0 : [512 x 363] (gradient)
|
||||
MPI Rank 2: W0*features+B0 : [512 x 1 x *] }
|
||||
MPI Rank 2: { W0*features+B0 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { H2 : [512 x 1 x *]
|
||||
MPI Rank 2: W1*H1 : [512 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { B1 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H2 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: HLast : [132 x 1 x *] (gradient) }
|
||||
MPI Rank 2: { W1 : [512 x 512] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] }
|
||||
MPI Rank 2: { H1 : [512 x 1 x *]
|
||||
MPI Rank 2: W0*features : [512 x *] (gradient) }
|
||||
MPI Rank 2: { B0 : [512 x 1] (gradient)
|
||||
MPI Rank 2: H1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W1*H1+B1 : [512 x 1 x *] (gradient)
|
||||
MPI Rank 2: W2*H1 : [132 x 1 x *] }
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Node 'B0' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Node 'B1' (LearnableParameter operation) : [512 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Node 'B2' (LearnableParameter operation) : [132 x 1]
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Node 'W0' (LearnableParameter operation) : [512 x 363]
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Node 'W1' (LearnableParameter operation) : [512 x 512]
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Node 'W2' (LearnableParameter operation) : [132 x 512]
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: Initializing dataParallelSGD with FP64 aggregation.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Precomputing --> 3 PreCompute nodes found.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:47: MeanOfFeatures = Mean()
|
||||
MPI Rank 2: 12/15/2016 08:31:47: InvStdOfFeatures = InvStdDev()
|
||||
MPI Rank 2: 12/15/2016 08:31:47: Prior = Mean()
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..252734] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2: requiredata: determined feature kind as 33-dimensional 'USER' with frame shift 10.0 ms
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:50: Precomputing --> Completed.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:50: Starting Epoch 1: learning rate per sample = 0.015625 effective momentum = 0.900000 momentum as time constant = 607.4 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 0: frames [0..20480] (first utterance at frame 0), data subset 0 of 1, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:50: Starting minibatch loop.
|
||||
MPI Rank 2: 12/15/2016 08:31:50: Epoch[ 1 of 4]-Minibatch[ 1- 10, 3.13%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 0.2984s; samplesPerSecond = 2145.1
|
||||
MPI Rank 2: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 11- 20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 0.2685s; samplesPerSecond = 2383.9
|
||||
MPI Rank 2: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 21- 30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 0.2603s; samplesPerSecond = 2459.1
|
||||
MPI Rank 2: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 31- 40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.2523s; samplesPerSecond = 2536.2
|
||||
MPI Rank 2: 12/15/2016 08:31:51: Epoch[ 1 of 4]-Minibatch[ 41- 50, 15.63%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 0.2533s; samplesPerSecond = 2526.6
|
||||
MPI Rank 2: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 51- 60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 0.2614s; samplesPerSecond = 2448.8
|
||||
MPI Rank 2: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 61- 70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 0.2527s; samplesPerSecond = 2532.5
|
||||
MPI Rank 2: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 71- 80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2533s; samplesPerSecond = 2527.1
|
||||
MPI Rank 2: 12/15/2016 08:31:52: Epoch[ 1 of 4]-Minibatch[ 81- 90, 28.13%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.2543s; samplesPerSecond = 2516.4
|
||||
MPI Rank 2: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 0.2539s; samplesPerSecond = 2521.0
|
||||
MPI Rank 2: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 0.2545s; samplesPerSecond = 2514.5
|
||||
MPI Rank 2: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 0.2522s; samplesPerSecond = 2537.9
|
||||
MPI Rank 2: 12/15/2016 08:31:53: Epoch[ 1 of 4]-Minibatch[ 121- 130, 40.63%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 0.2487s; samplesPerSecond = 2573.4
|
||||
MPI Rank 2: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460534 * 640; EvalClassificationError = 0.75468750 * 640; time = 0.2520s; samplesPerSecond = 2539.7
|
||||
MPI Rank 2: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 0.2805s; samplesPerSecond = 2281.6
|
||||
MPI Rank 2: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968882 * 640; EvalClassificationError = 0.74531250 * 640; time = 0.2580s; samplesPerSecond = 2480.2
|
||||
MPI Rank 2: 12/15/2016 08:31:54: Epoch[ 1 of 4]-Minibatch[ 161- 170, 53.13%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 0.2642s; samplesPerSecond = 2422.4
|
||||
MPI Rank 2: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2820s; samplesPerSecond = 2269.5
|
||||
MPI Rank 2: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 0.2554s; samplesPerSecond = 2506.1
|
||||
MPI Rank 2: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 0.2599s; samplesPerSecond = 2462.9
|
||||
MPI Rank 2: 12/15/2016 08:31:55: Epoch[ 1 of 4]-Minibatch[ 201- 210, 65.63%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 0.2579s; samplesPerSecond = 2481.4
|
||||
MPI Rank 2: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 0.2629s; samplesPerSecond = 2434.6
|
||||
MPI Rank 2: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 0.2602s; samplesPerSecond = 2460.1
|
||||
MPI Rank 2: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2617s; samplesPerSecond = 2445.7
|
||||
MPI Rank 2: 12/15/2016 08:31:56: Epoch[ 1 of 4]-Minibatch[ 241- 250, 78.13%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 0.2518s; samplesPerSecond = 2541.9
|
||||
MPI Rank 2: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954796 * 640; EvalClassificationError = 0.62812500 * 640; time = 0.2529s; samplesPerSecond = 2530.4
|
||||
MPI Rank 2: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 0.2549s; samplesPerSecond = 2510.5
|
||||
MPI Rank 2: 12/15/2016 08:31:57: Epoch[ 1 of 4]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.2542s; samplesPerSecond = 2517.4
|
||||
MPI Rank 2: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 281- 290, 90.63%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 0.2546s; samplesPerSecond = 2513.8
|
||||
MPI Rank 2: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 0.2511s; samplesPerSecond = 2548.9
|
||||
MPI Rank 2: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 0.2517s; samplesPerSecond = 2542.9
|
||||
MPI Rank 2: 12/15/2016 08:31:58: Epoch[ 1 of 4]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 0.2514s; samplesPerSecond = 2545.7
|
||||
MPI Rank 2: 12/15/2016 08:31:58: Finished Epoch[ 1 of 4]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=8.29391s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:58: Starting Epoch 2: learning rate per sample = 0.001953 effective momentum = 0.656119 momentum as time constant = 607.5 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 1: frames [20480..40960] (first utterance at frame 20480), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:31:58: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.014339
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.02282
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.034024
|
||||
MPI Rank 2: 12/15/2016 08:31:59: Epoch[ 2 of 4]-Minibatch[ 1- 10, 12.50%]: CrossEntropyWithSoftmax = 2.19109241 * 2304; EvalClassificationError = 0.58246528 * 2304; time = 0.3624s; samplesPerSecond = 6357.5
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.011787
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.044361
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.017445
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.042474
|
||||
MPI Rank 2: 12/15/2016 08:31:59: Epoch[ 2 of 4]-Minibatch[ 11- 20, 25.00%]: CrossEntropyWithSoftmax = 2.20697464 * 2560; EvalClassificationError = 0.59453125 * 2560; time = 0.4107s; samplesPerSecond = 6233.9
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.016332
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.037706
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.015016
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.037001
|
||||
MPI Rank 2: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 21- 30, 37.50%]: CrossEntropyWithSoftmax = 2.23618716 * 2560; EvalClassificationError = 0.60039062 * 2560; time = 0.3980s; samplesPerSecond = 6431.9
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.018632
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.044457
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.015279
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.03699
|
||||
MPI Rank 2: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 31- 40, 50.00%]: CrossEntropyWithSoftmax = 2.21810382 * 2560; EvalClassificationError = 0.59609375 * 2560; time = 0.4133s; samplesPerSecond = 6193.6
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.016877
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.039
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.012806
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.040168
|
||||
MPI Rank 2: 12/15/2016 08:32:00: Epoch[ 2 of 4]-Minibatch[ 41- 50, 62.50%]: CrossEntropyWithSoftmax = 2.17778205 * 2560; EvalClassificationError = 0.59414062 * 2560; time = 0.3948s; samplesPerSecond = 6483.8
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.019087
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.041333
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.001464
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.0523
|
||||
MPI Rank 2: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 51- 60, 75.00%]: CrossEntropyWithSoftmax = 2.13452559 * 2560; EvalClassificationError = 0.57734375 * 2560; time = 0.3891s; samplesPerSecond = 6579.9
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.013255
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.035552
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.012247
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.039475
|
||||
MPI Rank 2: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 61- 70, 87.50%]: CrossEntropyWithSoftmax = 2.13087789 * 2560; EvalClassificationError = 0.57265625 * 2560; time = 0.3970s; samplesPerSecond = 6448.7
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.01151
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.038946
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.014233
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.03598
|
||||
MPI Rank 2: 12/15/2016 08:32:01: Epoch[ 2 of 4]-Minibatch[ 71- 80, 100.00%]: CrossEntropyWithSoftmax = 2.11200101 * 2560; EvalClassificationError = 0.58632812 * 2560; time = 0.3830s; samplesPerSecond = 6683.7
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.032617
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.009137
|
||||
MPI Rank 2: 12/15/2016 08:32:02: Finished Epoch[ 2 of 4]: [Training] CrossEntropyWithSoftmax = 2.17402050 * 20480; EvalClassificationError = 0.58750000 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=3.19815s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:32:02: Starting Epoch 3: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 2: frames [40960..61440] (first utterance at frame 40960), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:32:02: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.048113
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.121149
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.06524
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.122137
|
||||
MPI Rank 2: 12/15/2016 08:32:03: Epoch[ 3 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 2.15723941 * 9216; EvalClassificationError = 0.56488715 * 9216; time = 1.1785s; samplesPerSecond = 7820.3
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.02704
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.13324
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.041943
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.120703
|
||||
MPI Rank 2: 12/15/2016 08:32:04: Epoch[ 3 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 2.02453665 * 10240; EvalClassificationError = 0.55771484 * 10240; time = 1.2475s; samplesPerSecond = 8208.4
|
||||
MPI Rank 2: 12/15/2016 08:32:04: Finished Epoch[ 3 of 4]: [Training] CrossEntropyWithSoftmax = 2.08437881 * 20480; EvalClassificationError = 0.56079102 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-005; epochTime=2.54358s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:32:04: Starting Epoch 4: learning rate per sample = 0.000098 effective momentum = 0.656119 momentum as time constant = 2429.9 samples
|
||||
MPI Rank 2: minibatchiterator: epoch 3: frames [61440..81920] (first utterance at frame 61440), data subset 2 of 3, with 1 datapasses
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:32:04: Starting minibatch loop, DataParallelSGD training (myRank = 2, numNodes = 3, numGradientBits = 64), BufferedAsyncGradientAggregation is ENABLED, distributed reading is ENABLED.
|
||||
MPI Rank 2: Async gradient aggregation wait time: 3e-006
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.070919
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.030892
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.125888
|
||||
MPI Rank 2: 12/15/2016 08:32:05: Epoch[ 4 of 4]-Minibatch[ 1- 10, 50.00%]: CrossEntropyWithSoftmax = 1.96502938 * 9216; EvalClassificationError = 0.53190104 * 9216; time = 1.1196s; samplesPerSecond = 8231.4
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.034789
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.121302
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.035583
|
||||
MPI Rank 2: Actual gradient aggregation time: 0.138767
|
||||
MPI Rank 2: 12/15/2016 08:32:07: Epoch[ 4 of 4]-Minibatch[ 11- 20, 100.00%]: CrossEntropyWithSoftmax = 1.95947098 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 1.2808s; samplesPerSecond = 7994.8
|
||||
MPI Rank 2: Async gradient aggregation wait time: 0.008184
|
||||
MPI Rank 2: 12/15/2016 08:32:07: Finished Epoch[ 4 of 4]: [Training] CrossEntropyWithSoftmax = 1.96369079 * 20480; EvalClassificationError = 0.53471680 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-005; epochTime=2.53683s
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:32:07: Action "train" complete.
|
||||
MPI Rank 2:
|
||||
MPI Rank 2: 12/15/2016 08:32:07: __COMPLETED__
|
||||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
|
|
@ -1,23 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
. $TEST_ROOT_DIR/run-test-common
|
||||
|
||||
OriginalTestDir=../../../DNN/ParallelNoQuantizationBufferedAsyncGradientAggregation
|
||||
ConfigDir=$TEST_DIR/../../../DNN
|
||||
LogFileName=stderr
|
||||
Instances=3
|
||||
NumCPUThreads=$(threadsPerInstance $Instances)
|
||||
|
||||
(cd $TEST_DIR/$OriginalTestDir && md5sum baseline*) | (cd $TEST_DIR && md5sum --status -c -)
|
||||
if [ $? != 0 ]; then
|
||||
echo Error: Baselines must match original test. Copy from $OriginalTestDir.
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# cntkmpirun <MPI args> <CNTK config file name> <additional CNTK args>
|
||||
cntkmpirun "-n $Instances" cntk.cntk "speechTrain=[reader=[readerType=HTKDeserializers]] numCPUThreads=$NumCPUThreads precision=double speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[gradientBits=64]]]] speechTrain=[SGD=[ParallelTrain=[DataParallelSGD=[useBufferedAsyncGradientAggregation=true]]]] speechTrain=[SGD=[ParallelTrain=[parallelizationStartEpoch=2]]] speechTrain=[SGD=[maxEpochs=4]] speechTrain=[SGD=[ParallelTrain=[syncPerfStats=5]]]"
|
||||
ExitCode=$?
|
||||
sed 's/^/MPI Rank 0: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank0
|
||||
sed 's/^/MPI Rank 1: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank1
|
||||
sed 's/^/MPI Rank 2: /' $TEST_RUN_DIR/"$LogFileName"_speechTrain.logrank2
|
||||
exit $ExitCode
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
dataDir: ../../../Data
|
||||
tags:
|
||||
# - bvt-p ((build_sku == 'gpu') or (build_sku == '1bitsgd')) and ((flavor == 'release') if (os == 'windows') else ((flavor == 'debug') ^ (device == 'cpu')))
|
||||
- nightly-p ((build_sku == 'gpu') or (build_sku == '1bitsgd'))
|
||||
|
||||
testCases:
|
||||
Must train epochs in exactly same order and parameters for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting Epoch {{integer}}
|
||||
- learning rate per sample = {{float}}
|
||||
- momentum = {{float}}
|
||||
|
||||
Epochs must be finished with expected results for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Finished Epoch[{{integer}} of {{integer}}]
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0%}}
|
||||
- EvalClassificationError = {{float,tolerance=0%}}
|
||||
- learningRatePerSample = {{float,tolerance=0%}}
|
||||
|
||||
Per-minibatch training results must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Epoch[{{integer}} of {{integer}}]-Minibatch[{{integer}}-{{integer}}
|
||||
- " * {{integer}}; "
|
||||
- CrossEntropyWithSoftmax = {{float,tolerance=0%}}
|
||||
- EvalClassificationError = {{float,tolerance=0%}}
|
||||
|
||||
DataParallelSGD training parameters must match for each MPI Rank:
|
||||
patterns:
|
||||
- ^MPI Rank {{integer}}
|
||||
- Starting minibatch loop
|
||||
- DataParallelSGD training
|
||||
- myRank = {{integer}}
|
||||
- numNodes = 3
|
||||
- numGradientBits = 64
|
||||
- distributed reading is ENABLED
|
||||
- BufferedAsyncGradientAggregation is ENABLED
|
||||
Загрузка…
Ссылка в новой задаче