adding convolution over sequential axis related tests.
adding convolution over sequential axis.
currently additional supported parameters:
auto padding
strides
groups
support for dilation needs to be tested on GPU.
updating PrimitiveOpType SerializationTests that is missing from other commits ..
convert tabs to spaces.
Refine cpp convolution unit tests. Add dilation tests to python convolution unit tests.
more detailed comments on shape change for 1d seq conv with reduction rank 0. And other minor tweaks.
add EndToEndTests of sequential convolution on MNIST
add init_bias tests for seq conv
minor change in comments
rename ConvolutionOverSequenceAxisNode. Add comment on cudnn failed new test.
add more comments, trim spaces
add more comments, remove magic number, add more boundary checks.
remove the last SetValue for outputSeqAxisDimValue as TensorView Unary Op has already updated the value.
fix bug in python seqconv default bias shape, and add related unit tests.
small tweak in seq conv to avoid additional gpu memory allocation and increase performance.
Example: seq MNIST, and profiling
adjust conv c++ value unit test channel size.
small update on python seq mnist
Sequential convolution v2.
* re-designed ConvolutionSequenceShapeNode: refactored to separate out computing output sequence length from v1 node design. And refactored ConvolutionNodeBaseExtended as their common base class. (Since "ConvolutionNodeBase" is not only base class of ConvolutionNode but also PoolingNode).
* Performance increase against v1.
- compute sequence length by MBLayout instead of mask output from unpack. Avoiding the unnecessary cpu/gpu memory copy.
not include py sequence example for now .. need to find they a correct location.
add check for truncated sequences in sequential convolution
improve code style.
Moving sequential convolution in python to a new high level api, to maintain compatibility with previous implementation (special case 1d sequential convolution).
Add ConvolutionSequenceShape OP.
nit
update conv_attribute test for updated convolution parameter
move sequential parameter to the last
update test shortcircuit for CPU convolution dilation.
update endtoendtest - unittest baseline file for new convolution unittests.
update makefile to include new unittest file for linux
nit
Update ConvolutionNode initialization code to handle TransformerNode Initialization.
nit
nit
* the error was due to that we pad 0 as default value for missing gaps. All these then each contribute 1 to the gradient of reference at index 0. The fix is to mask missing values in indices matrix to negative, and in Matrix scatter implementation to check and skip negative indices. (previous Matrix CPU implementation already checks for negative indices)
Working example in ./Examples/Image/Classification/MLP/Python/SimpleMNIST.py
Note that node timing would be added to profiler details when profiler is enabled, i.e.
import cntk as C
C.debugging.debug.set_node_timing(True)
C.debugging.start_profiler()
C.debugging.enable_profiler()
trainer|evaluator|function executions
trainer|evaluator|function.print_node_timing()
C.debugging.stop_profiler()
CNTK now supports CUDA 9/cuDNN 7. This requires an update to build environment to Ubuntu 16/GCC 5 for Linux, and Visual Studio 2017/VCTools 14.11 for Windows. With CUDA 9, CNTK also added a preview for 16-bit floating point (a.k.a FP16) computation.
Please check out the example of FP16 in ResNet50 at /Examples/Image/Classification/ResNet/Python/TrainResNet_ImageNet_Distributed.py
Notes on FP16 preview:
* FP16 implementation on CPU is not optimized, and it's not supposed to be used in CPU inference directly. User needs to convert the model to 32-bit floating point before running on CPU.
* Loss/Criterion for FP16 training needs to be 32bit for accumulation without overflow, using cast function. Please check the example above.
* Readers do not have FP16 output unless using numpy to feed data, cast from FP32 to FP16 is needed. Please check the example above.
* FP16 gradient aggregation is currently only implemented on GPU using NCCL2. Distributed training with FP16 with MPI is not supported.
* FP16 math is a subset of current FP32 implementation. Some model may get Feature Not Implemented exception using FP16.
* FP16 is currently not supported in BrainScript. Please use Python for FP16.
To setup build and runtime environment on Windows:
* Install [Visual Studio 2017](https://www.visualstudio.com/downloads/) with following workloads and components. From command line (use Community version installer as example):
vs_community.exe --add Microsoft.VisualStudio.Workload.NativeDesktop --add Microsoft.VisualStudio.Workload.ManagedDesktop --add Microsoft.VisualStudio.Workload.Universal --add Microsoft.Component.PythonTools --add Microsoft.VisualStudio.Component.VC.Tools.14.11
* Install [NVidia CUDA 9](https://developer.nvidia.com/cuda-90-download-archive?target_os=Windows&target_arch=x86_64)
* From PowerShell, run:
/Tools/devInstall/Windows/DevInstall.ps1
* Start VCTools 14.11 command line, run:
cmd /k "%VS2017INSTALLDIR%\VC\Auxiliary\Build\vcvarsall.bat" x64 --vcvars_ver=14.11
* Open /CNTK.sln from the VCTools 14.11 command line. Note that starting CNTK.sln other than VCTools 14.11 command line, would causes CUDA 9 [build error](https://developercommunity.visualstudio.com/content/problem/163758/vs-2017-155-doesnt-support-cuda-9.html).
To setup build and runtime environment on Linux using docker, please build Unbuntu 16.04 docker image using Dockerfiles /Tools/docker. For other Linux systems, please refer to the Dockerfiles to setup dependent libraries for CNTK.
Peyman Manikashani
Sergii Dymchenko
Bowen Bao
Spandan Tiwari
Jie Zhu
Yuqing Tang
Jaliya Ekanayake
KeDengMS
Project Philly
Igor Macedo Quintanilha
Liqun Fu
Chengji Yao
Nikos Karampatziakis