* Issue was that AssignSqrOfDifferenceOf(beta, input, mean, alpha)
assigns mean value to the gaps in input. These values are then reduced
within this function, leading to incorrect results. The fix is to
execute assign and reduce separately, and mask gaps to zero again before reducing.
* Update test baseline affected by this change (err is lowered by <1%).
Previously, to resolve issue of gather producing incorrect gradient
values, validity mask check was added to ensure we don't count non-valid
cells as 0.
However, this check is needed only for input that has dynamic axis, i.e.
inputs that have MBLayout.
In case other projects may use these header files, we added
them into API/Internals.
* ComputationGraphAlgorithms.h was moved from Source/ComputationNetworkLib
* PrimitiveOpType.h and EvaluatorWrapper.h were moved from Source/CNTKv2Library
* PrimitiveFunctionAttribute.h was extracted from PrimitiveFunction.h. It contains
a new class PrimitiveFunctionAttribute which is the collection of all attribute
names for PrimitiveFunction.
This change actually had a subtle side-effect. We had a global static variable
s_stateAttributes that depended on PrimitiveFunction::AttributeNameRngSeed and
PrimitiveFunction::AttributeNameRngOffset. After we moved those static
attribute-variables into another translation unit, s_stateAttributes can be
initialized with empty wstring, because PrimitiveFunctionAttribute::AttributeNameRngSeed
PrimitiveFunctionAttribute::AttributeNameRngSeedOffset were initialized after
s_stateAttributes. Note that the initialization order of global static variables
is not well-defined cross translation units. To fix the issue, we also moved
s_stateAttributes into PrimitiveFunctionAttribute class, and renamed it to
s_rngStateAttributes. I think it's reasonable to consider s_rngStateAttributes
to be part of the PrimitiveFunctionAttribute class.
* PrimitiveFunction.h was moved from Source/CNTKv2Library
adding convolution over sequential axis related tests.
adding convolution over sequential axis.
currently additional supported parameters:
auto padding
strides
groups
support for dilation needs to be tested on GPU.
updating PrimitiveOpType SerializationTests that is missing from other commits ..
convert tabs to spaces.
Refine cpp convolution unit tests. Add dilation tests to python convolution unit tests.
more detailed comments on shape change for 1d seq conv with reduction rank 0. And other minor tweaks.
add EndToEndTests of sequential convolution on MNIST
add init_bias tests for seq conv
minor change in comments
rename ConvolutionOverSequenceAxisNode. Add comment on cudnn failed new test.
add more comments, trim spaces
add more comments, remove magic number, add more boundary checks.
remove the last SetValue for outputSeqAxisDimValue as TensorView Unary Op has already updated the value.
fix bug in python seqconv default bias shape, and add related unit tests.
small tweak in seq conv to avoid additional gpu memory allocation and increase performance.
Example: seq MNIST, and profiling
adjust conv c++ value unit test channel size.
small update on python seq mnist
Sequential convolution v2.
* re-designed ConvolutionSequenceShapeNode: refactored to separate out computing output sequence length from v1 node design. And refactored ConvolutionNodeBaseExtended as their common base class. (Since "ConvolutionNodeBase" is not only base class of ConvolutionNode but also PoolingNode).
* Performance increase against v1.
- compute sequence length by MBLayout instead of mask output from unpack. Avoiding the unnecessary cpu/gpu memory copy.
not include py sequence example for now .. need to find they a correct location.
add check for truncated sequences in sequential convolution
improve code style.
Moving sequential convolution in python to a new high level api, to maintain compatibility with previous implementation (special case 1d sequential convolution).
Add ConvolutionSequenceShape OP.
nit
update conv_attribute test for updated convolution parameter
move sequential parameter to the last
update test shortcircuit for CPU convolution dilation.
update endtoendtest - unittest baseline file for new convolution unittests.
update makefile to include new unittest file for linux
nit
Update ConvolutionNode initialization code to handle TransformerNode Initialization.
nit
nit
* the error was due to that we pad 0 as default value for missing gaps. All these then each contribute 1 to the gradient of reference at index 0. The fix is to mask missing values in indices matrix to negative, and in Matrix scatter implementation to check and skip negative indices. (previous Matrix CPU implementation already checks for negative indices)
Working example in ./Examples/Image/Classification/MLP/Python/SimpleMNIST.py
Note that node timing would be added to profiler details when profiler is enabled, i.e.
import cntk as C
C.debugging.debug.set_node_timing(True)
C.debugging.start_profiler()
C.debugging.enable_profiler()
trainer|evaluator|function executions
trainer|evaluator|function.print_node_timing()
C.debugging.stop_profiler()
CNTK now supports CUDA 9/cuDNN 7. This requires an update to build environment to Ubuntu 16/GCC 5 for Linux, and Visual Studio 2017/VCTools 14.11 for Windows. With CUDA 9, CNTK also added a preview for 16-bit floating point (a.k.a FP16) computation.
Please check out the example of FP16 in ResNet50 at /Examples/Image/Classification/ResNet/Python/TrainResNet_ImageNet_Distributed.py
Notes on FP16 preview:
* FP16 implementation on CPU is not optimized, and it's not supposed to be used in CPU inference directly. User needs to convert the model to 32-bit floating point before running on CPU.
* Loss/Criterion for FP16 training needs to be 32bit for accumulation without overflow, using cast function. Please check the example above.
* Readers do not have FP16 output unless using numpy to feed data, cast from FP32 to FP16 is needed. Please check the example above.
* FP16 gradient aggregation is currently only implemented on GPU using NCCL2. Distributed training with FP16 with MPI is not supported.
* FP16 math is a subset of current FP32 implementation. Some model may get Feature Not Implemented exception using FP16.
* FP16 is currently not supported in BrainScript. Please use Python for FP16.
To setup build and runtime environment on Windows:
* Install [Visual Studio 2017](https://www.visualstudio.com/downloads/) with following workloads and components. From command line (use Community version installer as example):
vs_community.exe --add Microsoft.VisualStudio.Workload.NativeDesktop --add Microsoft.VisualStudio.Workload.ManagedDesktop --add Microsoft.VisualStudio.Workload.Universal --add Microsoft.Component.PythonTools --add Microsoft.VisualStudio.Component.VC.Tools.14.11
* Install [NVidia CUDA 9](https://developer.nvidia.com/cuda-90-download-archive?target_os=Windows&target_arch=x86_64)
* From PowerShell, run:
/Tools/devInstall/Windows/DevInstall.ps1
* Start VCTools 14.11 command line, run:
cmd /k "%VS2017INSTALLDIR%\VC\Auxiliary\Build\vcvarsall.bat" x64 --vcvars_ver=14.11
* Open /CNTK.sln from the VCTools 14.11 command line. Note that starting CNTK.sln other than VCTools 14.11 command line, would causes CUDA 9 [build error](https://developercommunity.visualstudio.com/content/problem/163758/vs-2017-155-doesnt-support-cuda-9.html).
To setup build and runtime environment on Linux using docker, please build Unbuntu 16.04 docker image using Dockerfiles /Tools/docker. For other Linux systems, please refer to the Dockerfiles to setup dependent libraries for CNTK.
liqfu
Bowen Bao
KeDengMS
Yang Chen
Vadim Mazalov
Binbin Zhang
kaituoxu
Spandan Tiwari
David Brownell
Sergii Dymchenko
Jie Zhu
Jaliya Ekanayake
Yuqing Tang
Project Philly
Eldar Akchurin
Chengji Yao