From cd91a1360e70408862c40f04448216e6b0bd2ca6 Mon Sep 17 00:00:00 2001 From: Terry Kim Date: Sat, 29 Jun 2019 10:34:52 +0200 Subject: [PATCH] Update developer guide to support new patch version of Spark (#109) --- docs/developer-guide.md | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/docs/developer-guide.md b/docs/developer-guide.md index a6eb1859..9bdf89fb 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -1,8 +1,12 @@ # Developer Guide -## How to debug +## Table of Contents +- [How to Do Local Debugging](#how-to-do-local-debugging) +- [How to Support New Spark Releases](#how-to-support-new-spark-releases) -### Debugging .NET application +## How to Do Local Debugging + +### Debugging Spark .NET Application Open a new command prompt window, run the following: ```shell @@ -34,3 +38,26 @@ spark-submit \ \ ``` + +## How to Support New Spark Releases + +We encourage developers to first read Apache Spark's [Versioning Policy](https://spark.apache.org/versioning-policy.html) and [Semantic Versioning](https://semver.org/) to gain the most out of the instructions below. + +At a high-level, Spark's versions are: **[MAJOR].[FEATURE].[MAINTENANCE]**. We will cover the upgrade path for each type of version separately below (in increasing order of effort required). + +### [MAINTENANCE]: Upgrading for a Patch Release Version +Since Apache Spark's [MAINTENANCE] releases involve only internal changes (e.g., bug fixes etc.), it is straightforward to upgrade the code base to support a [MAINTENANCE] release. The steps to do this are below: + +1. In the corresponding `pom.xml`, update the `spark.version` value to the newly released version. + * For example, if a new patch release is 2.4.3, you will update [src/scala/microsoft-spark-2.4.x/pom.xml](/src/scala/microsoft-spark-2.4.x/pom.xml) to have `2.4.3`. +2. Update `DotnetRunner.supportedSparkVersions` to include the newly released version. + * For example, if a new patch release is 2.4.3, you will update [src/scala/microsoft-spark-2.4.x/src/main/scala/org/apache/spark/deploy/DotnetRunner.scala](/src/scala/microsoft-spark-2.4.x/src/main/scala/org/apache/spark/deploy/DotnetRunner.scala). +3. Update the [azure-pipelines.yml](/azure-pipelines.yml) to include E2E testing for the newly released version. + +Refer to [this commit](https://github.com/dotnet/spark/commit/eb26baa46200bfcbe3e1080e650f335853d9990e) for an example. + +### [FEATURE]: Upgrading for a Minor Release Version +*WIP* + +### [MAJOR]: Upgrading for a Major Release Version +*WIP*