Update developer guide to support new patch version of Spark (#109)

This commit is contained in:
Terry Kim 2019-06-29 10:34:52 +02:00 коммит произвёл GitHub
Родитель 574be7bdca
Коммит cd91a1360e
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
1 изменённых файлов: 29 добавлений и 2 удалений

Просмотреть файл

@ -1,8 +1,12 @@
# Developer Guide
## How to debug
## Table of Contents
- [How to Do Local Debugging](#how-to-do-local-debugging)
- [How to Support New Spark Releases](#how-to-support-new-spark-releases)
### Debugging .NET application
## How to Do Local Debugging
### Debugging Spark .NET Application
Open a new command prompt window, run the following:
```shell
@ -34,3 +38,26 @@ spark-submit \
<path-to-microsoft-spark-jar> \
<path-to-your-app-exe> <argument(s)-to-your-app>
```
## How to Support New Spark Releases
We encourage developers to first read Apache Spark's [Versioning Policy](https://spark.apache.org/versioning-policy.html) and [Semantic Versioning](https://semver.org/) to gain the most out of the instructions below.
At a high-level, Spark's versions are: **[MAJOR].[FEATURE].[MAINTENANCE]**. We will cover the upgrade path for each type of version separately below (in increasing order of effort required).
### [MAINTENANCE]: Upgrading for a Patch Release Version
Since Apache Spark's [MAINTENANCE] releases involve only internal changes (e.g., bug fixes etc.), it is straightforward to upgrade the code base to support a [MAINTENANCE] release. The steps to do this are below:
1. In the corresponding `pom.xml`, update the `spark.version` value to the newly released version.
* For example, if a new patch release is 2.4.3, you will update [src/scala/microsoft-spark-2.4.x/pom.xml](/src/scala/microsoft-spark-2.4.x/pom.xml) to have `<spark.version>2.4.3</spark.version>`.
2. Update `DotnetRunner.supportedSparkVersions` to include the newly released version.
* For example, if a new patch release is 2.4.3, you will update [src/scala/microsoft-spark-2.4.x/src/main/scala/org/apache/spark/deploy/DotnetRunner.scala](/src/scala/microsoft-spark-2.4.x/src/main/scala/org/apache/spark/deploy/DotnetRunner.scala).
3. Update the [azure-pipelines.yml](/azure-pipelines.yml) to include E2E testing for the newly released version.
Refer to [this commit](https://github.com/dotnet/spark/commit/eb26baa46200bfcbe3e1080e650f335853d9990e) for an example.
### [FEATURE]: Upgrading for a Minor Release Version
*WIP*
### [MAJOR]: Upgrading for a Major Release Version
*WIP*