Minor improvements to dev/README.md (#11525)
This commit is contained in:
Родитель
6c8cf6aebe
Коммит
2509d13096
136
dev/README.md
136
dev/README.md
|
@ -156,9 +156,9 @@ You can add and copy the tokens here:
|
||||||
* [Prod PyPI](https://pypi.org/manage/account/token/)
|
* [Prod PyPI](https://pypi.org/manage/account/token/)
|
||||||
|
|
||||||
|
|
||||||
Create a ~/.pypirc file:
|
Create a `~/.pypirc` file:
|
||||||
|
|
||||||
```shell script
|
```ini
|
||||||
[distutils]
|
[distutils]
|
||||||
index-servers =
|
index-servers =
|
||||||
pypi
|
pypi
|
||||||
|
@ -172,7 +172,6 @@ password=<API Upload Token>
|
||||||
repository=https://test.pypi.org/legacy/
|
repository=https://test.pypi.org/legacy/
|
||||||
username=__token__
|
username=__token__
|
||||||
password=<API Upload Token>
|
password=<API Upload Token>
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Set proper permissions for the pypirc file:
|
Set proper permissions for the pypirc file:
|
||||||
|
@ -193,7 +192,7 @@ pip install twine
|
||||||
- Set proper permissions for the pypirc file:
|
- Set proper permissions for the pypirc file:
|
||||||
`$ chmod 600 ~/.pypirc`
|
`$ chmod 600 ~/.pypirc`
|
||||||
|
|
||||||
- Confirm that airflow/version.py is set properly.
|
- Confirm that `airflow/version.py` is set properly.
|
||||||
|
|
||||||
|
|
||||||
## Hardware used to prepare and verify the packages
|
## Hardware used to prepare and verify the packages
|
||||||
|
@ -213,52 +212,67 @@ prepared using such hardware. More information can be found in this
|
||||||
The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release canidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
|
The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release canidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
|
||||||
|
|
||||||
- Set environment variables
|
- Set environment variables
|
||||||
```
|
|
||||||
# Set Version
|
```shell script
|
||||||
export VERSION=1.10.2rc3
|
# Set Version
|
||||||
|
export VERSION=1.10.2rc3
|
||||||
|
|
||||||
|
|
||||||
# Set AIRFLOW_REPO_ROOT to the path of your git repo
|
# Set AIRFLOW_REPO_ROOT to the path of your git repo
|
||||||
export AIRFLOW_REPO_ROOT=$(pwd)
|
export AIRFLOW_REPO_ROOT=$(pwd)
|
||||||
|
|
||||||
|
|
||||||
# Example after cloning
|
# Example after cloning
|
||||||
git clone https://github.com/apache/airflow.git airflow
|
git clone https://github.com/apache/airflow.git airflow
|
||||||
cd airflow
|
cd airflow
|
||||||
export AIRFLOW_REPO_ROOT=$(pwd)
|
export AIRFLOW_REPO_ROOT=$(pwd)
|
||||||
```
|
```
|
||||||
|
|
||||||
- set your version to 1.10.2 in airflow/version.py (without the RC tag)
|
- Set your version to 1.10.2 in `airflow/version.py` (without the RC tag)
|
||||||
- Commit the version change.
|
- Commit the version change.
|
||||||
|
|
||||||
- Tag your release
|
- Tag your release
|
||||||
|
|
||||||
`git tag ${VERSION}`
|
```shell script
|
||||||
|
git tag ${VERSION}
|
||||||
|
```
|
||||||
|
|
||||||
- Clean the checkout: the sdist step below will
|
- Clean the checkout: the sdist step below will
|
||||||
`git clean -fxd`
|
|
||||||
|
```shell script
|
||||||
|
git clean -fxd
|
||||||
|
```
|
||||||
|
|
||||||
- Tarball the repo
|
- Tarball the repo
|
||||||
`git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz`
|
|
||||||
|
```shell script
|
||||||
|
git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz`
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
- Generate sdist
|
- Generate sdist
|
||||||
|
|
||||||
NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included in the file produced.
|
NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included
|
||||||
|
in the file produced.
|
||||||
|
|
||||||
`python setup.py compile_assets sdist bdist_wheel`
|
```shell script
|
||||||
|
python setup.py compile_assets sdist bdist_wheel
|
||||||
|
```
|
||||||
|
|
||||||
- Rename the sdist
|
- Rename the sdist
|
||||||
```
|
|
||||||
mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz
|
```shell script
|
||||||
mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl
|
mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz
|
||||||
```
|
mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl
|
||||||
|
```
|
||||||
|
|
||||||
- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
|
- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
|
||||||
```
|
|
||||||
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz
|
```shell script
|
||||||
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz
|
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz
|
||||||
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl
|
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz
|
||||||
```
|
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl
|
||||||
|
```
|
||||||
|
|
||||||
- Push Tags
|
- Push Tags
|
||||||
`git push --tags`
|
`git push --tags`
|
||||||
|
@ -284,16 +298,25 @@ svn commit -m "Add artifacts for Airflow ${VERSION}"
|
||||||
At this point we have the artefact that we vote on, but as a convenience to developers we also want to
|
At this point we have the artefact that we vote on, but as a convenience to developers we also want to
|
||||||
publish "snapshots" of the RC builds to pypi for installing via pip. To do this we need to
|
publish "snapshots" of the RC builds to pypi for installing via pip. To do this we need to
|
||||||
|
|
||||||
- Edit the airflow/version.py to include the RC suffix.
|
- Edit the `airflow/version.py` to include the RC suffix.
|
||||||
|
|
||||||
- Build the package:
|
- Build the package:
|
||||||
`python setup.py compile_assets sdist bdist_wheel`
|
|
||||||
|
```shell script
|
||||||
|
python setup.py compile_assets sdist bdist_wheel
|
||||||
|
```
|
||||||
|
|
||||||
- Verify the artifacts that would be uploaded:
|
- Verify the artifacts that would be uploaded:
|
||||||
`twine check dist/*`
|
|
||||||
|
```shell script
|
||||||
|
twine check dist/*
|
||||||
|
```
|
||||||
|
|
||||||
- Upload the package to PyPi's test environment:
|
- Upload the package to PyPi's test environment:
|
||||||
`twine upload -r pypitest dist/*`
|
|
||||||
|
```shell script
|
||||||
|
twine upload -r pypitest dist/*
|
||||||
|
```
|
||||||
|
|
||||||
- Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at:
|
- Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at:
|
||||||
https://test.pypi.org/project/apache-airflow/#files
|
https://test.pypi.org/project/apache-airflow/#files
|
||||||
|
@ -549,13 +572,13 @@ downloaded from the SVN).
|
||||||
There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
|
There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
|
||||||
Running the following command will use tmux inside breeze, create `admin` user and run Webserver & Scheduler:
|
Running the following command will use tmux inside breeze, create `admin` user and run Webserver & Scheduler:
|
||||||
|
|
||||||
```
|
```shell script
|
||||||
./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres
|
./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres
|
||||||
```
|
```
|
||||||
|
|
||||||
For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
|
For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
|
||||||
|
|
||||||
```
|
```shell script
|
||||||
./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres --no-rbac-ui
|
./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres --no-rbac-ui
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -642,27 +665,41 @@ Verify that the packages appear in [airflow](https://dist.apache.org/repos/dist/
|
||||||
At this point we release an official package:
|
At this point we release an official package:
|
||||||
|
|
||||||
- Build the package:
|
- Build the package:
|
||||||
`python setup.py compile_assets sdist bdist_wheel`
|
|
||||||
|
```shell script
|
||||||
|
python setup.py compile_assets sdist bdist_wheel`
|
||||||
|
```
|
||||||
|
|
||||||
- Verify the artifacts that would be uploaded:
|
- Verify the artifacts that would be uploaded:
|
||||||
`twine check dist/*`
|
|
||||||
|
```shell script
|
||||||
|
twine check dist/*`
|
||||||
|
```
|
||||||
|
|
||||||
- Upload the package to PyPi's test environment:
|
- Upload the package to PyPi's test environment:
|
||||||
`twine upload -r pypitest dist/*`
|
|
||||||
|
```shell script
|
||||||
|
twine upload -r pypitest dist/*
|
||||||
|
```
|
||||||
|
|
||||||
- Verify that the test package looks good by downloading it and installing it into a virtual environment.
|
- Verify that the test package looks good by downloading it and installing it into a virtual environment.
|
||||||
The package download link is available at: https://test.pypi.org/project/apache-airflow/#files
|
The package download link is available at: https://test.pypi.org/project/apache-airflow/#files
|
||||||
|
|
||||||
- Upload the package to PyPi's production environment:
|
- Upload the package to PyPi's production environment:
|
||||||
`twine upload -r pypi dist/*`
|
|
||||||
|
|
||||||
- Again, confirm that the package is available here:
|
```shell script
|
||||||
https://pypi.python.org/pypi/apache-airflow
|
twine upload -r pypi dist/*
|
||||||
|
```
|
||||||
|
|
||||||
|
- Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow
|
||||||
|
|
||||||
### Update CHANGELOG.md
|
### Update CHANGELOG.md
|
||||||
|
|
||||||
- Get a diff between the last version and the current version:
|
- Get a diff between the last version and the current version:
|
||||||
`$ git log 1.8.0..1.9.0 --pretty=oneline`
|
|
||||||
|
```shell script
|
||||||
|
$ git log 1.8.0..1.9.0 --pretty=oneline
|
||||||
|
```
|
||||||
- Update CHANGELOG.md with the details, and commit it.
|
- Update CHANGELOG.md with the details, and commit it.
|
||||||
|
|
||||||
### Notify developers of release
|
### Notify developers of release
|
||||||
|
@ -753,14 +790,14 @@ Details about maintaining the SEMVER version are going to be discussed and imple
|
||||||
Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
|
Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
|
||||||
date for the packages.
|
date for the packages.
|
||||||
|
|
||||||
```
|
```shell script
|
||||||
./breeze --backports prepare-provider-readme YYYY.MM.DD [packages]
|
./breeze --backports prepare-provider-readme YYYY.MM.DD [packages]
|
||||||
```
|
```
|
||||||
|
|
||||||
If you iterate with merges and release candidates you can update the release date without providing
|
If you iterate with merges and release candidates you can update the release date without providing
|
||||||
the date (to update the existing release notes)
|
the date (to update the existing release notes)
|
||||||
|
|
||||||
```
|
```shell script
|
||||||
./breeze --backports prepare-provider-readme google
|
./breeze --backports prepare-provider-readme google
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -788,9 +825,8 @@ export AIRFLOW_REPO_ROOT=$(pwd)
|
||||||
|
|
||||||
* Build the source package:
|
* Build the source package:
|
||||||
|
|
||||||
```
|
```shell script
|
||||||
./provider_packages/build_source_package.sh
|
./provider_packages/build_source_package.sh
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
|
It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
|
||||||
|
@ -1154,7 +1190,7 @@ backport packages. This is especially helpful when you want to test integrations
|
||||||
additional tools. Below is an example Dockerfile, which installs backport providers for Google and
|
additional tools. Below is an example Dockerfile, which installs backport providers for Google and
|
||||||
an additional third-party tools:
|
an additional third-party tools:
|
||||||
|
|
||||||
```
|
```dockerfile
|
||||||
FROM apache/airflow:1.10.12
|
FROM apache/airflow:1.10.12
|
||||||
|
|
||||||
RUN pip install --user apache-airflow-backport-providers-google==2020.10.5.rc1
|
RUN pip install --user apache-airflow-backport-providers-google==2020.10.5.rc1
|
||||||
|
@ -1174,7 +1210,7 @@ USER ${AIRFLOW_UID}
|
||||||
|
|
||||||
To build an image build and run a shell, run:
|
To build an image build and run a shell, run:
|
||||||
|
|
||||||
```
|
```shell script
|
||||||
docker build . -t my-airflow
|
docker build . -t my-airflow
|
||||||
docker run -ti \
|
docker run -ti \
|
||||||
--rm \
|
--rm \
|
||||||
|
@ -1415,7 +1451,7 @@ Details about maintaining the SEMVER version are going to be discussed and imple
|
||||||
[the related issue](https://github.com/apache/airflow/issues/11425)
|
[the related issue](https://github.com/apache/airflow/issues/11425)
|
||||||
|
|
||||||
|
|
||||||
```
|
```shell script
|
||||||
./breeze prepare-provider-readme [packages]
|
./breeze prepare-provider-readme [packages]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
Загрузка…
Ссылка в новой задаче