5efb1fd089
* Add support for python latest versions * generated constants * Updated runtime tag * generated constants * Updated unit tests * updated runtime tag * updated constants * Removed extra space character * Added new python versions to all os types * Updated tests * Updated tests * convert go fully to go.mod * removed gopkg files, adn fully built all go packages with dependencies * fixed tests to run on packages * cleaned up scripts * removed built exes * make mod files consistent * allow execution of the test packages script * Fixed bug to download tarballs from storage account url supplied from pipeline * Fix bug to download sdks from storage account url supplied from pipeline * Add AddDefaultTestEnvironmentVariables * Replcae SetEnvironmentVariable by AddDefaultTestEnvironmentVariables * Commented out test for python 3.11 * Add support for python latest versions * generated constants * Updated runtime tag * generated constants * Updated unit tests * updated runtime tag * updated constants * Removed extra space character * Added new python versions to all os types * Updated tests * Updated tests * Fixed bug to download tarballs from storage account url supplied from pipeline * Fix bug to download sdks from storage account url supplied from pipeline * fixed tests to run on packages * cleaned up scripts * Add AddDefaultTestEnvironmentVariables * Replcae SetEnvironmentVariable by AddDefaultTestEnvironmentVariables * Commented out test for python 3.11 * Add dotnet 7 to stretch / older images (#1667) * add dotnet 7 to stretch / legacy images * generated constants and supported versions, and updated the final stretch version for dotnet 7 * remove .NET7 condition on bindport2 (#1670) * Pauld/switch go packages to go mod (#1671) * convert go fully to go.mod * removed gopkg files, adn fully built all go packages with dependencies * fixed tests to run on packages * cleaned up scripts * removed built exes * make mod files consistent * allow execution of the test packages script * added readme for updating and testing go dependencies * Add support for python latest versions * generated constants * Updated runtime tag * generated constants * Updated unit tests * updated runtime tag * updated constants * Removed extra space character * Added new python versions to all os types * Updated tests * Updated tests * Fixed bug to download tarballs from storage account url supplied from pipeline * Fix bug to download sdks from storage account url supplied from pipeline * fixed tests to run on packages * cleaned up scripts * Add AddDefaultTestEnvironmentVariables * Replcae SetEnvironmentVariable by AddDefaultTestEnvironmentVariables * Commented out test for python 3.11 * also add execution permissions for the restore packages script * reordered testing shell script builder methods * Updated error message * Fixed astroid version conflict * Fixed version conflict with astroid 2.8.0 * Fixed astroid 2.8.0 version conflict * Addressed PR comments and resolved dependency conflict * Resolved pyflakes version conflict * Enabled test case to use default test env variables * Resolved dependency conflicts * Updated runtime tag Co-authored-by: Paul Dorsch <pauldorsch@microsoft.com> Co-authored-by: Paul Dorsch <107068277+pauld-msft@users.noreply.github.com> Co-authored-by: waliMSFT <105239794+waliMSFT@users.noreply.github.com> |
||
---|---|---|
.. | ||
detector | ||
tools | ||
FinalPublicKey.snk | ||
Readme.md | ||
TestAssembliesKey.snk | ||
__condaConstants.sh | ||
__dotNetCoreRunTimeVersions.sh | ||
__dotNetCoreSdkVersions.sh | ||
__extVarNames.sh | ||
__finalStretchVersions.sh | ||
__functions.sh | ||
__goVersions.sh | ||
__golangVersions.sh | ||
__hugoConstants.sh | ||
__javaVersions.sh | ||
__nodeVersions.sh | ||
__phpVersions.sh | ||
__pythonVersions.sh | ||
__rubyVersions.sh | ||
__sdkStorageConstants.sh | ||
__stagingRuntimeConstants.sh | ||
__variables.sh | ||
buildBuildImageBases.sh | ||
buildBuildImages.sh | ||
buildBuildpacksImages.sh | ||
buildPlatformBinaries.sh | ||
buildPythonSdkByVersion.sh | ||
buildRunTimeImageBases.sh | ||
buildRunTimeImages.sh | ||
buildSln.sh | ||
buildTestBuildImages.sh | ||
calculateContainerSizes.sh | ||
constants.yaml | ||
createSymlinksForDotnet.sh | ||
generateConstants.sh | ||
release.txt | ||
testBuildImages.sh | ||
testBuildScriptGenerator.sh | ||
testDetector.sh | ||
testIntegration.sh | ||
testRunTimeImages.sh | ||
testStartupScriptGenerators.sh |
Readme.md
Why so many scripts?
The scripts in this folder are broken down into different layers to enable a good development as a well as CI experience.
So if a user wants to just build and test build images, they can do so by running its respective script.
These scripts are actually called in VSTS pipelines too (Check the vsts/pipelines/templates/_buildTemplate.yml
file).
Artifact files
Building build and runtime images can conceptually be viewed as building a .NET repo's 'src' folder where after successfully
building the output is written to a artifacts folder. In a similar way, after building build/runtime images, we write out
the names of the images that were built to files '/artifacts/images/build-images.txt' and '/artifacts/images/runtime-images.txt'.
This is the same experience if the images were built locally on a dev machine or a CI agent. The idea is that in case of CI agent,
when a VSTS task needs to push images to a docker registry, it can just consume these artifact files to figure out which images to push.
Untagging images
Taggig a docker image is actually tagging or pinning docker layers. Since a CI agent can go through several builds a day we would be
in the risk of running out of disk space. So we want to make sure to not pin layers which are not used anymore.
For example,
Let's say the following docker file was built with 2 tags called 'test:latest' and 'test:buildnumber1'. Let's call the individual
layers as L1, L2 and L3.
FROM busybox <-- L1
RUN echo hello > hello.txt <-- L2
RUN echo hi > hi.txt <-- L3
Now let's say this docker file was changed in the following way and a new docker build is done with 2 tags called 'test:latest'
and 'test:buildnumber2'. When the following docker file is built there would be new layers called L22 and L33.
FROM busybox <-- L1
RUN echo hello-updated > hello.txt <-- L22
RUN echo hi > hi.txt <-- L33
At this point the 'latest' and 'buildnumber2' tags are pointing to L33 layer and ideally we do not want to keep the layers L2 and L3
around as they are not going to be used anyway and just consume disk space. But they are still pinned via the tag 'buildnumber1'.
So, to prevent this we try to unpin/untag a build number based tag (not the 'latest' tag because its a moving tag) which will
create dangling layers/images helping us to cleanup (for example, L2 and L3 would no longer have any references and can be cleaned up).
Why tag intermediate stages in the multi-stage build image dockerfile?
As of docker version 18.06.1-ce
, whenever docker builds a multi-stage dockerfile, it only tags the layer of the final FROM related
image. Since intermediate FROM statements exist in a multi-stage process, their layers are left dangled. This is problematic for us
because we, in general, want to clean out dangling images (for example, frequent changes to BuildScriptGenerator tool results in
dangling images which we want to clean) to avoid running into disk space problems. So, to prevent these dangling intermediate FROM
statement related images from being cleaned up (i.e rebuilding the whole stage again from scratch), we tag them. These images are
however never pushed to the server and only exist on the machine where the image was built.