Azure
/
aks-engine-azurestack
зеркало из https://github.com/Azure/aks-engine-azurestack.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

feat: migrate from Pod Security Policy to Pod Security admission (#94)

This commit is contained in:
Javier Darsie 2023-02-28 12:11:09 -08:00 коммит произвёл GitHub
Родитель 586934d3b6
Коммит 31d208cfe2
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
30 изменённых файлов: 991 добавлений и 190 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

36
docs/topics/clusterdefinitions.md
Просмотреть файл

@ -597,25 +597,23 @@ See [here](https://kubernetes.io/docs/reference/generated/kube-apiserver/) for a
Below is a list of apiserver options that AKS Engine will configure by default:
| apiserver option | default value |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "--anonymous-auth" | "false" |
| "--admission-control" | "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota" (Kubernetes versions prior to 1.9.0) |
| "--enable-admission-plugins"`*` | "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,ExtendedResourceToleration" (Kubernetes versions 1.9.0 and later) |
| "--authorization-mode" | "Node", "RBAC" (_the latter if enabledRbac is true_) |
| "--audit-log-maxage" | "30" |
| "--audit-log-maxbackup" | "10" |
| "--audit-log-maxsize" | "100" |
| "--feature-gates" | No default (can be a comma-separated list) |
| "--oidc-username-claim" | "oid" (_if has AADProfile_) |
| "--oidc-groups-claim" | "groups" (_if has AADProfile_) |
| "--oidc-client-id" | _calculated value that represents OID client ID_ (_if has AADProfile_) |
| "--oidc-issuer-url" | _calculated value that represents OID issuer URL_ (_if has AADProfile_) |
| "--service-account-issuer" | "https://kubernetes.default.svc.cluster.local" (Kubernetes v1.20.0 and greater only) |
| "--service-account-signing-key-file" | "/etc/kubernetes/certs/apiserver.key" (Kubernetes v1.20.0 and greater only) |
| "--v" | "2" |
`*` In Kubernetes versions 1.10.0 and later the `--admission-control` flag is deprecated and `--enable-admission-plugins` is used instead.
| apiserver option | default value |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| "--admission-control-config-file" | "/etc/kubernetes/apiserver-admission-control.yaml" (default content overridable through _CustomFiles_) |
| "--anonymous-auth" | "false" |
| "--enable-admission-plugins" | "ExtendedResourceToleration" ("PodSecurityPolicy" if the "pod-security-policy" addon is enabled) |
| "--authorization-mode" | "Node", "RBAC" (_the latter if enabledRbac is true_) |
| "--audit-log-maxage" | "30" |
| "--audit-log-maxbackup" | "10" |
| "--audit-log-maxsize" | "100" |
| "--feature-gates" | No default (can be a comma-separated list) |
| "--oidc-username-claim" | "oid" (_if has AADProfile_) |
| "--oidc-groups-claim" | "groups" (_if has AADProfile_) |
| "--oidc-client-id" | _calculated value that represents OID client ID_ (_if has AADProfile_) |
| "--oidc-issuer-url" | _calculated value that represents OID issuer URL_ (_if has AADProfile_) |
| "--service-account-issuer" | "https://kubernetes.default.svc.cluster.local" (Kubernetes v1.20.0 and greater only) |
| "--service-account-signing-key-file" | "/etc/kubernetes/certs/apiserver.key" (Kubernetes v1.20.0 and greater only) |
| "--v" | "2" |
Below is a list of apiserver options that are _not_ currently user-configurable, either because a higher order configuration vector is available that enforces apiserver configuration, or because a static configuration is required to build a functional cluster:

140
docs/topics/pod-security.md Normal file
Просмотреть файл

@ -0,0 +1,140 @@
# Pod Security Admission
Kubernetes v1.25 [removes the `PodSecurityPolicy` (PSP) admission controller][PSPDeprecation]
in favor of the newer [PodSecurity][PSA] admission controller (PSA).
The PodSecurity admission is enabled by default since Kubernetes v1.23.
Up until Kubernetes v1.25, AKS Engine creates a set of [PodSecurityPolicy resources][PSPParts]
via the `"pod-security-policy"` addon (enabled by default).
The `"pod-security-policy"` addon will be forcefully disabled on Kubernetes v1.25+ clusters.
Hence, cluster administrators interested in enforcing the [Pod Security Standards][PSS]
through the `PodSecurity` admission controller should use a combination of:
- [Namespace Labels][PSALabels]
- [Admission Controller configuration file][PSAConfig]
- See how to upload a configuration file [below](#admission-controller-configuration-file).
## Migrating from PSP
Migrating from the PodSecurityPolicy admission is a **pre-requisite** to upgrade to Kubernetes v1.25+.
The [step-by-step guide][Migration] published by the Kubernetes project is a good resource
on how to migrate from the PodSecurityPolicy admission
if the target cluster includes customizations beyond AKS Engine's defaults.
If the target AKS Engine-based cluster only includes the configuration applied
by the `"pod-security-policy"` addon, then there are two migration alternatives:
### Option 1: Migrate using `aks-engine-azurestack upgrade`
To migrate using a cluster upgrade operation, the API Model should be updated in the following way:
- Disable the `"pod-security-policy"` addon
- Remove `PodSecurityPolicy` from the plugins list in `--enable-admission-plugins`
```json
{
"kubernetesConfig": {
"addons": [
{
"name": "pod-security-policy",
"enabled": false
}
],
"apiServerConfig": {
"--enable-admission-plugins": "...,ExtendedResourceToleration",
}
}
}
```
To speed up the upgrade process, use the `--control-plane-only`.
To validate pod security before upgrading to Kubernetes v1.25,
perform a "forced" upgrade to the cluster's current version (needs flag `--force`).
```bash
aks-engine-azurestack upgrade \
--upgrade-version {CurrentKubernetesVersion} \
--force \
--control-plane-only \
...
```
Once upgrade is over, delete the resources created by the `"pod-security-policy"` addon:
```bash
kubectl delete clusterrolebinding default:restricted default:privileged
kubectl delete clusterrole psp:restricted psp:privileged
kubectl delete psp restricted privileged
```
### Option 2: Manual Migration
An alternative, faster, migration process is to follow this **ordered** sequence of steps:
1. For each master node:
1. Delete the `"pod-security-policy"` addon yaml from the `kube-addon-manager` directory
1. Remove the `PodSecurityPolicy` admission controller from the `kube-apiserver` manifest
1. Update API Model as indicated in the [previous section](#migrate-using-aks-engine-azurestack-upgrade)
1. Delete resources created by the `"pod-security-policy"` addon
```bash
# Install node-shell if preferred over SSH => https://github.com/kvaps/kubectl-node-shell
# kubectl krew index add kvaps https://github.com/kvaps/krew-index
# kubectl krew install kvaps/node-shell
# 1.1. Delete the PSP addon yaml
kubectl node-shell {MasterNodeName} \
-- sh -c 'rm -f /etc/kubernetes/addons/pod-security-policy.yaml'
# 1.2. Remove PodSecurityPolicy admission controller
kubectl node-shell {MasterNodeName} \
-- sh -c 'sed -i s/,PodSecurityPolicy//1 /etc/kubernetes/manifests/kube-apiserver.yaml'
# 3. Delete resources created by the PSP addon
kubectl delete clusterrolebinding default:restricted default:privileged
kubectl delete clusterrole psp:restricted psp:privileged
kubectl delete psp restricted privileged
```
## Admission Controller configuration file
The behavior of the PSA controller can be customized through the `AdmissionConfiguration` file.
The default AKS Engine configuration enforces the `privileged` stardard
(see [apiserver-admission-control.yaml][PSADefaultConfig]]).
### Custom Admission Controller configuration
To use a custom `AdmissionConfiguration` file, combine [CustomFiles](/examples/customfiles/README.md)
and the Kubernetes API server flag `--admission-control-config-file`
(default location `/etc/kubernetes/apiserver-admission-control.yaml`).
```json
{
"orchestratorProfile": {
"kubernetesConfig": {
"apiServerConfig": {
"--admission-control-config-file": "/etc/kubernetes/apiserver-admission-control.yaml",
}
},
},
"masterProfile": {
"customFiles": [
{
"source" : "/local/path/to/my/apiserver-admission-control.yaml",
"dest" : "/etc/kubernetes/apiserver-admission-control.yaml"
}
]
}
}
```
[PSPDeprecation]: https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/
[PSA]: https://kubernetes.io/docs/concepts/security/pod-security-admission/
[PSPParts]: /parts/k8s/addons/pod-security-policy.yaml
[PSS]: https://kubernetes.io/docs/concepts/security/pod-security-standards/
[PSALabels]: https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-namespace-labels/
[PSAConfig]: https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/
[Migration]: https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/
[PSADefaultConfig]: /parts/k8s/cloud-init/artifacts/apiserver-admission-control.yaml

21
parts/k8s/cloud-init/artifacts/apiserver-admission-control.yaml Normal file
Просмотреть файл

@ -0,0 +1,21 @@
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
configuration:
apiVersion: pod-security.admission.config.k8s.io/v1{{- if not (IsKubernetesVersionGe "1.25.0")}}beta1{{end}}
kind: PodSecurityConfiguration
defaults:
{{- /* allow everything by default, back-compatible */}}
enforce: privileged
enforce-version: latest
{{- /* cli warning if pod does not enforce the baseline stardard */}}
warn: baseline
warn-version: latest
{{- /* audit log entry if pod does not enforce the restricted stardard */}}
audit: restricted
audit-version: latest
exemptions:
usernames: []
runtimeClasses: []
namespaces: [kube-system]

2
parts/k8s/cloud-init/artifacts/cse_config.sh
Просмотреть файл

@ -439,7 +439,7 @@ ensureAddons() {
{{- if IsAzurePolicyAddonEnabled}}
retrycmd 120 5 30 $KUBECTL get namespace gatekeeper-system || exit_cse {{GetCSEErrorCode "ERR_ADDONS_START_FAIL"}} $GET_KUBELET_LOGS
{{- end}}
{{- if not HasCustomPodSecurityPolicy}}
{{- if and (not HasCustomPodSecurityPolicy) IsPodSecurityPolicyAddonEnabled}}
retrycmd 120 5 30 $KUBECTL get podsecuritypolicy privileged restricted || exit_cse {{GetCSEErrorCode "ERR_ADDONS_START_FAIL"}} $GET_KUBELET_LOGS
{{- end}}
replaceAddonsInit

9
parts/k8s/cloud-init/masternodecustomdata.yml
Просмотреть файл

@ -398,6 +398,15 @@ write_files:
{{CloudInitData "kmsKeyvaultKeyScript"}}
{{end}}
{{- if NeedsDefaultAPIServerAdmissionConfiguration}}
- path: {{GetAPIServerAdmissionConfigurationFilepath}}
permissions: "0644"
encoding: gzip
owner: root
content: !!binary |
{{CloudInitData "apiServerAdmissionConfiguration"}}
{{end}}
MASTER_MANIFESTS_CONFIG_PLACEHOLDER
MASTER_CUSTOM_FILES_PLACEHOLDER

11
pkg/api/addons.go
Просмотреть файл

@ -733,9 +733,10 @@ func (cs *ContainerService) setAddonsConfig(isUpgrade bool) {
defaultKubeProxyAddonsConfig.Config["metrics-bind-address"] = "::1"
}
pspAddonEnabled := !common.ShouldDisablePodSecurityPolicyAddon(o.OrchestratorVersion)
defaultPodSecurityPolicyAddonsConfig := KubernetesAddon{
Name: common.PodSecurityPolicyAddonName,
Enabled: to.BoolPtr(true),
Enabled: to.BoolPtr(pspAddonEnabled),
}
defaultAuditPolicyAddonsConfig := KubernetesAddon{
@ -1043,8 +1044,12 @@ func (cs *ContainerService) setAddonsConfig(isUpgrade bool) {
o.KubernetesConfig.Addons = append(o.KubernetesConfig.Addons[:i], o.KubernetesConfig.Addons[i+1:]...)
}
// Enable pod-security-policy addon during upgrade scenarios, unless explicitly disabled
if isUpgrade && !o.KubernetesConfig.IsAddonDisabled(common.PodSecurityPolicyAddonName) {
// Enable pod-security-policy addon during upgrade scenarios, unless explicitly disabled or v1.25+
if common.ShouldDisablePodSecurityPolicyAddon(o.OrchestratorVersion) {
if i := getAddonsIndexByName(o.KubernetesConfig.Addons, common.PodSecurityPolicyAddonName); i > -1 {
o.KubernetesConfig.Addons[i].Enabled = to.BoolPtr(false)
}
} else if isUpgrade && !o.KubernetesConfig.IsAddonDisabled(common.PodSecurityPolicyAddonName) {
if i := getAddonsIndexByName(o.KubernetesConfig.Addons, common.PodSecurityPolicyAddonName); i > -1 {
o.KubernetesConfig.Addons[i].Enabled = to.BoolPtr(true)
}

102
pkg/api/addons_test.go
Просмотреть файл

@ -3983,6 +3983,106 @@ func TestSetAddonsConfig(t *testing.T) {
isUpgrade: true,
expectedAddons: omitFromAddons([]string{common.PodSecurityPolicyAddonName}, getDefaultAddons("1.15.4", "", common.KubernetesImageBaseTypeMCR)),
},
{
name: "pod-security-policy enabled by default for v1.25- deployment",
cs: &ContainerService{
Properties: &Properties{
OrchestratorProfile: &OrchestratorProfile{
OrchestratorVersion: "1.24.0",
KubernetesConfig: &KubernetesConfig{
KubernetesImageBaseType: common.KubernetesImageBaseTypeMCR,
DNSServiceIP: DefaultKubernetesDNSServiceIP,
KubeletConfig: map[string]string{
"--cluster-domain": "cluster.local",
},
ClusterSubnet: DefaultKubernetesSubnet,
ProxyMode: KubeProxyModeIPTables,
NetworkPlugin: NetworkPluginAzure,
},
},
},
},
isUpgrade: false,
expectedAddons: getDefaultAddons("1.24.0", "", common.KubernetesImageBaseTypeMCR),
},
{
name: "pod-security-policy disabled by default for v1.25+ deployment",
cs: &ContainerService{
Properties: &Properties{
OrchestratorProfile: &OrchestratorProfile{
OrchestratorVersion: common.PodSecurityPolicyRemovedVersion,
KubernetesConfig: &KubernetesConfig{
KubernetesImageBaseType: common.KubernetesImageBaseTypeMCR,
DNSServiceIP: DefaultKubernetesDNSServiceIP,
KubeletConfig: map[string]string{
"--cluster-domain": "cluster.local",
},
ClusterSubnet: DefaultKubernetesSubnet,
ProxyMode: KubeProxyModeIPTables,
NetworkPlugin: NetworkPluginAzure,
},
},
},
},
isUpgrade: false,
expectedAddons: omitFromAddons([]string{common.PodSecurityPolicyAddonName}, getDefaultAddons(common.PodSecurityPolicyRemovedVersion, "", common.KubernetesImageBaseTypeMCR)),
},
{
name: "pod-security-policy forcefully disabled for v1.25+ deployment",
cs: &ContainerService{
Properties: &Properties{
OrchestratorProfile: &OrchestratorProfile{
OrchestratorVersion: common.PodSecurityPolicyRemovedVersion,
KubernetesConfig: &KubernetesConfig{
KubernetesImageBaseType: common.KubernetesImageBaseTypeMCR,
DNSServiceIP: DefaultKubernetesDNSServiceIP,
KubeletConfig: map[string]string{
"--cluster-domain": "cluster.local",
},
ClusterSubnet: DefaultKubernetesSubnet,
ProxyMode: KubeProxyModeIPTables,
NetworkPlugin: NetworkPluginAzure,
Addons: []KubernetesAddon{
{
Name: common.PodSecurityPolicyAddonName,
Enabled: to.BoolPtr(true),
},
},
},
},
},
},
isUpgrade: false,
expectedAddons: omitFromAddons([]string{common.PodSecurityPolicyAddonName}, getDefaultAddons(common.PodSecurityPolicyRemovedVersion, "", common.KubernetesImageBaseTypeMCR)),
},
{
name: "pod-security-policy forcefully disabled for v1.25+ upgrade",
cs: &ContainerService{
Properties: &Properties{
OrchestratorProfile: &OrchestratorProfile{
OrchestratorVersion: common.PodSecurityPolicyRemovedVersion,
KubernetesConfig: &KubernetesConfig{
KubernetesImageBaseType: common.KubernetesImageBaseTypeMCR,
DNSServiceIP: DefaultKubernetesDNSServiceIP,
KubeletConfig: map[string]string{
"--cluster-domain": "cluster.local",
},
ClusterSubnet: DefaultKubernetesSubnet,
ProxyMode: KubeProxyModeIPTables,
NetworkPlugin: NetworkPluginAzure,
Addons: []KubernetesAddon{
{
Name: common.PodSecurityPolicyAddonName,
Enabled: to.BoolPtr(true),
},
},
},
},
},
},
isUpgrade: true,
expectedAddons: omitFromAddons([]string{common.PodSecurityPolicyAddonName}, getDefaultAddons(common.PodSecurityPolicyRemovedVersion, "", common.KubernetesImageBaseTypeMCR)),
},
{
name: "audit-policy disabled",
cs: &ContainerService{
@ -5111,7 +5211,7 @@ func getDefaultAddons(version, kubernetesImageBase, kubernetesImageBaseType stri
},
{
Name: common.PodSecurityPolicyAddonName,
Enabled: to.BoolPtr(true),
Enabled: to.BoolPtr(!common.ShouldDisablePodSecurityPolicyAddon(version)),
},
}

2
pkg/api/common/const.go
Просмотреть файл

@ -64,6 +64,8 @@ const (
KubernetesDefaultReleaseAzureStack string = "1.23"
// KubernetesDefaultReleaseWindowsAzureStack is the default Kubernetes release for Windows on Azure Stack
KubernetesDefaultReleaseWindowsAzureStack string = "1.23"
// PodSecurityPolicyRemovedVersion is the first Kubernetes version that does not includes the PSP admission plugin
PodSecurityPolicyRemovedVersion = "1.25.0"
)
const LegacyControlPlaneVMPrefix string = "k8s-master"

23
pkg/api/common/flags.go Normal file
Просмотреть файл

@ -0,0 +1,23 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
package common
import "strings"
// RemoveFromCommaSeparatedList excludes from the input comma-separated list the strings toRemove
func RemoveFromCommaSeparatedList(input string, toRemove ...string) string {
removeMap := map[string]bool{}
for _, remove := range toRemove {
removeKey := strings.ToLower(strings.TrimSpace(remove))
removeMap[removeKey] = true
}
ret := []string{}
for _, value := range strings.Split(input, ",") {
key := strings.TrimSpace(value)
if !removeMap[strings.ToLower(key)] {
ret = append(ret, key)
}
}
return strings.Join(ret, ",")
}

71
pkg/api/common/flags_test.go Normal file
Просмотреть файл

@ -0,0 +1,71 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
package common
import (
"testing"
)
func TestRemoveFromCommaSeparatedList(t *testing.T) {
cases := []struct {
name string
input string
toRemove []string
expected string
}{
{
"Single value to remove",
"aks,engine,azure,stack,hub",
[]string{"azure"},
"aks,engine,stack,hub",
},
{
"Multiple values to remove",
"aks,engine,azure,stack,hub",
[]string{"engine", "stack"},
"aks,azure,hub",
},
{
"Remove first value",
"aks,engine,azure,stack,hub",
[]string{"aks"},
"engine,azure,stack,hub",
},
{
"Remove last value",
"aks,engine,azure,stack,hub",
[]string{"hub"},
"aks,engine,azure,stack",
},
{
"Value to remove not in list",
"aks,engine,azure,stack,hub",
[]string{"foo", "bar"},
"aks,engine,azure,stack,hub",
},
{
"Input list has spaces",
"aks, engine,azure , stack ,hub",
[]string{"azure"},
"aks,engine,stack,hub",
},
{
"Input list has unexpected casing",
"Aks,Engine,AZURE,Stack,Hub",
[]string{"azure"},
"Aks,Engine,Stack,Hub",
},
}
for _, c := range cases {
c := c
t.Run(c.name, func(t *testing.T) {
t.Parallel()
actual := RemoveFromCommaSeparatedList(c.input, c.toRemove...)
if actual != c.expected {
t.Fatalf("expected removeFromCommaSeparatedList to return '%s', but instead got '%s", c.expected, actual)
}
})
}
}

5
pkg/api/common/helper.go
Просмотреть файл

@ -624,3 +624,8 @@ var dockerAllConfigString = `{
}
}
}`
// ShouldDisablePodSecurityPolicyAddon returns true if the PodSecurityPolicyAddon should be forcefully disabled
func ShouldDisablePodSecurityPolicyAddon(version string) bool {
return IsKubernetesVersionGe(version, PodSecurityPolicyRemovedVersion)
}

64
pkg/api/defaults-apiserver.go
Просмотреть файл

@ -46,22 +46,29 @@ func (cs *ContainerService) setAPIServerConfig() {
}
}
// Default apiserver config
defaultAPIServerConfig := map[string]string{
"--anonymous-auth": "false",
"--audit-log-maxage": "30",
"--audit-log-maxbackup": "10",
"--audit-log-maxsize": "100",
"--profiling": DefaultKubernetesAPIServerEnableProfiling,
"--tls-cipher-suites": TLSStrongCipherSuitesAPIServer,
"--v": DefaultKubernetesAPIServerVerbosity,
}
// Data Encryption at REST configuration conditions
if to.Bool(o.KubernetesConfig.EnableDataEncryptionAtRest) || to.Bool(o.KubernetesConfig.EnableEncryptionWithExternalKms) {
staticAPIServerConfig["--encryption-provider-config"] = "/etc/kubernetes/encryption-config.yaml"
}
// Enable cloudprovider if we're not using cloud controller manager
if !to.Bool(o.KubernetesConfig.UseCloudControllerManager) {
staticAPIServerConfig["--cloud-provider"] = "azure"
staticAPIServerConfig["--cloud-config"] = "/etc/kubernetes/azure.json"
}
// Default apiserver config
defaultAPIServerConfig := map[string]string{
"--admission-control-config-file": "/etc/kubernetes/apiserver-admission-control.yaml",
"--anonymous-auth": "false",
"--audit-log-maxage": "30",
"--audit-log-maxbackup": "10",
"--audit-log-maxsize": "100",
"--profiling": DefaultKubernetesAPIServerEnableProfiling,
"--tls-cipher-suites": TLSStrongCipherSuitesAPIServer,
"--v": DefaultKubernetesAPIServerVerbosity,
}
// Aggregated API configuration
if o.KubernetesConfig.EnableAggregatedAPIs {
defaultAPIServerConfig["--requestheader-client-ca-file"] = "/etc/kubernetes/certs/proxy-ca.crt"
@ -73,12 +80,6 @@ func (cs *ContainerService) setAPIServerConfig() {
defaultAPIServerConfig["--requestheader-username-headers"] = "X-Remote-User"
}
// Enable cloudprovider if we're not using cloud controller manager
if !to.Bool(o.KubernetesConfig.UseCloudControllerManager) {
staticAPIServerConfig["--cloud-provider"] = "azure"
staticAPIServerConfig["--cloud-config"] = "/etc/kubernetes/azure.json"
}
// AAD configuration
if cs.Properties.HasAadProfile() {
defaultAPIServerConfig["--oidc-username-claim"] = "oid"
@ -165,19 +166,14 @@ func (cs *ContainerService) setAPIServerConfig() {
o.KubernetesConfig.APIServerConfig["--service-account-issuer"] = "https://kubernetes.default.svc.cluster.local"
}
invalidFeatureGates := []string{}
// Remove --feature-gate VolumeSnapshotDataSource starting with 1.22
// Reference: https://github.com/kubernetes/kubernetes/pull/101531
if common.IsKubernetesVersionGe(o.OrchestratorVersion, "1.22.0-alpha.1") {
invalidFeatureGates = append(invalidFeatureGates, "VolumeSnapshotDataSource")
}
removeInvalidFeatureGates(o.KubernetesConfig.APIServerConfig, invalidFeatureGates)
cs.overrideAPIServerConfig()
}
func getDefaultAdmissionControls(cs *ContainerService) (string, string) {
o := cs.Properties.OrchestratorProfile
admissionControlKey := "--enable-admission-plugins"
admissionControlValues := "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,ValidatingAdmissionWebhook,ResourceQuota,ExtendedResourceToleration"
// Only include admission controllers that are not enabled by default
admissionControlValues := "ExtendedResourceToleration"
// Pod Security Policy configuration
if o.KubernetesConfig.IsAddonEnabled(common.PodSecurityPolicyAddonName) {
@ -186,3 +182,21 @@ func getDefaultAdmissionControls(cs *ContainerService) (string, string) {
return admissionControlKey, admissionControlValues
}
// overrideAPIServerConfig fixes the kube-apiserver configuration,
// mostly by cleaning up removed features (flags, gates or admission controllers)
func (cs *ContainerService) overrideAPIServerConfig() {
o := cs.Properties.OrchestratorProfile
// Remove --feature-gate VolumeSnapshotDataSource starting with 1.22
// Reference: https://github.com/kubernetes/kubernetes/pull/101531
if common.IsKubernetesVersionGe(o.OrchestratorVersion, "1.22.0-alpha.1") {
removeInvalidFeatureGates(o.KubernetesConfig.APIServerConfig, []string{"VolumeSnapshotDataSource"})
}
if common.ShouldDisablePodSecurityPolicyAddon(o.OrchestratorVersion) {
curPlugins := o.KubernetesConfig.APIServerConfig["--enable-admission-plugins"]
newPlugins := common.RemoveFromCommaSeparatedList(curPlugins, "PodSecurityPolicy")
o.KubernetesConfig.APIServerConfig["--enable-admission-plugins"] = newPlugins
}
}

133
pkg/api/defaults-apiserver_test.go
Просмотреть файл

@ -354,34 +354,121 @@ func TestAPIServerConfigEnableSecureKubelet(t *testing.T) {
}
func TestAPIServerConfigDefaultAdmissionControls(t *testing.T) {
version := "1.15.4"
enableAdmissionPluginsKey := "--enable-admission-plugins"
admissonControlKey := "--admission-control"
cs := CreateMockContainerService("testcluster", version, 3, 2, false)
cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig = map[string]string{}
cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig[admissonControlKey] = "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,AlwaysPullImages,ExtendedResourceToleration"
cs.Properties.OrchestratorProfile.KubernetesConfig.Addons = []KubernetesAddon{
cases := []struct {
name string
k8sVersion string
pspEnabled bool
prevAdmissionPlugins string
expected string
}{
{
Name: common.PodSecurityPolicyAddonName,
Enabled: to.BoolPtr(true),
"Admission plugins for v1.25- & PSP enabled",
"1.23.4",
true,
"",
"ExtendedResourceToleration,PodSecurityPolicy",
},
{
"Admission plugins for v1.25- & PSP disabled",
"1.23.4",
false,
"",
"ExtendedResourceToleration",
},
{
"Admission plugins for v1.25+ & PSP enabled",
common.PodSecurityPolicyRemovedVersion,
true,
"",
"ExtendedResourceToleration",
},
{
"Admission plugins for v1.25+ & PSP disabled",
common.PodSecurityPolicyRemovedVersion,
false,
"",
"ExtendedResourceToleration",
},
// Note: this is a misconfiguration, not a valid test case
// {
// "Admission plugins for upgrade to v1.25- & PSP enabled",
// "1.23.4",
// true,
// "UserConfiguredAdmission",
// "UserConfiguredAdmission,PodSecurityPolicy",
// },
{
"Admission plugins for upgrade to v1.25- & PSP disabled",
"1.23.4",
false,
"UserConfiguredAdmission",
"UserConfiguredAdmission",
},
{
"Admission plugins for upgrade to v1.25+ & PSP enabled",
common.PodSecurityPolicyRemovedVersion,
true,
"UserConfiguredAdmission,PodSecurityPolicy",
"UserConfiguredAdmission",
},
{
"Admission plugins for upgrade to v1.25+ & PSP disabled",
common.PodSecurityPolicyRemovedVersion,
false,
"UserConfiguredAdmission",
"UserConfiguredAdmission",
},
}
cs.setAPIServerConfig()
a := cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig
if _, found := a[enableAdmissionPluginsKey]; !found {
t.Fatalf("Admission control key '%s' not set in API server config for version %s", enableAdmissionPluginsKey, version)
}
enableAdmissionPluginsKey := "--enable-admission-plugins"
admissonControlKey := "--admission-control"
admissonControlConfigFileKey := "--admission-control-config-file"
// --admission-control was deprecated in v1.10
if _, found := a[admissonControlKey]; found {
t.Fatalf("Deprecated admission control key '%s' set in API server config for version %s", admissonControlKey, version)
}
// PodSecurityPolicy should be enabled in admission control
admissionControlVal := a[enableAdmissionPluginsKey]
if !strings.Contains(admissionControlVal, ",PodSecurityPolicy") {
t.Fatalf("Admission control value '%s' expected to contain PodSecurityPolicy", admissionControlVal)
for _, c := range cases {
c := c
t.Run(c.name, func(t *testing.T) {
t.Parallel()
cs := CreateMockContainerService("testcluster", c.k8sVersion, 3, 2, false)
cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig = map[string]string{}
cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig[admissonControlKey] = "This,Flag,Should,Be,Removed"
if c.prevAdmissionPlugins != "" {
cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig[enableAdmissionPluginsKey] = c.prevAdmissionPlugins
}
cs.Properties.OrchestratorProfile.KubernetesConfig.Addons = []KubernetesAddon{
{
Name: common.PodSecurityPolicyAddonName,
Enabled: to.BoolPtr(c.pspEnabled),
},
}
cs.setAPIServerConfig()
apiServerConfig := cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig
// Check the --admission-control is not included, it was deprecated in v1.10
if _, found := apiServerConfig[admissonControlKey]; found {
t.Fatalf("Deprecated admission control flag '%s' set in API server config for version %s", admissonControlKey, c.k8sVersion)
}
// Check the --enable-admission-plugins flag is set
if _, found := apiServerConfig[enableAdmissionPluginsKey]; !found {
t.Fatalf("Admission plugin flag '%s' not set in API server config for version %s", enableAdmissionPluginsKey, c.k8sVersion)
}
// PodSecurityPolicy validation
admissionPlugins := apiServerConfig[enableAdmissionPluginsKey]
if common.ShouldDisablePodSecurityPolicyAddon(c.k8sVersion) {
if strings.Contains(admissionPlugins, "PodSecurityPolicy") {
t.Fatal("--enable-admission-plugins should not contain 'PodSecurityPolicy' after v1.25+")
}
// Flag --admission-control-config-file should be set
if _, found := apiServerConfig[admissonControlConfigFileKey]; !found {
t.Fatalf("Admission plugin config file flag '%s' not set in API server config for version %s", enableAdmissionPluginsKey, c.k8sVersion)
}
} else if c.pspEnabled && !strings.Contains(admissionPlugins, "PodSecurityPolicy") {
t.Fatal("--enable-admission-plugins should contain 'PodSecurityPolicy' if the 'pod-security-policy' addon is enabled")
} else if !c.pspEnabled && strings.Contains(admissionPlugins, "PodSecurityPolicy") {
t.Fatal("--enable-admission-plugins should not contain 'PodSecurityPolicy' if the 'pod-security-policy' addon is disabled")
}
if !strings.EqualFold(admissionPlugins, c.expected) {
t.Fatalf("expected --enable-admission-plugins value is '%s', got instead '%s'", c.expected, admissionPlugins)
}
})
}
}

6
pkg/api/vlabs/validate.go
Просмотреть файл

@ -798,6 +798,12 @@ func (a *Properties) validateAddons(isUpdate bool) error {
if !common.IsKubernetesVersionGe(a.OrchestratorProfile.OrchestratorVersion, "1.16.0") {
return errors.Errorf("%s add-on can only be used in 1.16+", addon.Name)
}
case common.PodSecurityPolicyAddonName:
if common.ShouldDisablePodSecurityPolicyAddon(a.OrchestratorProfile.OrchestratorVersion) {
log.Warn("The PodSecurityPolicy admission was removed in Kubernetes v1.25+. " +
"The pod security standards will be enforced by the built-in PodSecurity admission controller instead. " +
"See https://github.com/Azure/aks-engine-azurestack/blob/master/docs/topics/pod-security.md")
}
case common.AzureArcOnboardingAddonName:
if err := addon.validateArcAddonConfig(); err != nil {
return err

10
pkg/api/vlabs/validate_test.go
Просмотреть файл

@ -997,10 +997,20 @@ func ExampleProperties_validateAddons() {
fmt.Printf("error in validateAddons: %s", err)
}
cs.Properties.OrchestratorProfile.OrchestratorVersion = common.PodSecurityPolicyRemovedVersion
cs.Properties.OrchestratorProfile.KubernetesConfig = &KubernetesConfig{}
cs.Properties.OrchestratorProfile.KubernetesConfig.Addons = []KubernetesAddon{
{Name: common.PodSecurityPolicyAddonName, Enabled: to.BoolPtr(true)},
}
if err := cs.Properties.validateAddons(true); err != nil {
fmt.Printf("error in validateAddons: %s", err)
}
// Output:
// level=warning msg="The rescheduler addon has been deprecated and disabled, it will be removed during this update"
// level=warning msg="The kube-dashboard addon is deprecated, we recommend you install the dashboard yourself, see https://github.com/kubernetes/dashboard"
// level=warning msg="The Azure CNI networkmonitor addon has been deprecated, it will be marked as disabled"
// level=warning msg="The PodSecurityPolicy admission was removed in Kubernetes v1.25+. The pod security standards will be enforced by the built-in PodSecurity admission controller instead. See https://github.com/Azure/aks-engine-azurestack/blob/master/docs/topics/pod-security.md"
}
func ExampleProperties_validateLinuxProfile() {

21
pkg/armhelpers/mockclients.go
Просмотреть файл

@ -27,6 +27,7 @@ import (
log "github.com/sirupsen/logrus"
appsv1 "k8s.io/api/apps/v1"
v1 "k8s.io/api/core/v1"
policyv1beta1 "k8s.io/api/policy/v1beta1"
rbacv1 "k8s.io/api/rbac/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
@ -86,6 +87,7 @@ type MockKubernetesClient struct {
FailListPods bool
FailListNodes bool
FailListServiceAccounts bool
FailListPodSecurityPolicy bool
FailGetNode bool
UpdateNodeFunc func(*v1.Node) (*v1.Node, error)
GetNodeFunc func(name string) (*v1.Node, error)
@ -102,6 +104,7 @@ type MockKubernetesClient struct {
ShouldSupportEviction bool
PodsList *v1.PodList
ServiceAccountList *v1.ServiceAccountList
PodSecurityPolicyList *policyv1beta1.PodSecurityPolicyList
FailGetDeploymentCount int
FailUpdateDeploymentCount int
}
@ -359,6 +362,24 @@ func (mkc *MockKubernetesClient) ListServiceAccounts(namespace string) (*v1.Serv
return saList, nil
}
// ListPodSecurityPolices returns the list of Pod Security Policies
func (mkc *MockKubernetesClient) ListPodSecurityPolices(opts metav1.ListOptions) (*policyv1beta1.PodSecurityPolicyList, error) {
if mkc.FailListPodSecurityPolicy {
return nil, errors.New("ListPodSecurityPolices failed")
}
if mkc.PodSecurityPolicyList != nil {
return mkc.PodSecurityPolicyList, nil
}
psp1 := &policyv1beta1.PodSecurityPolicy{}
psp1.Name = "privileged"
psp2 := &policyv1beta1.PodSecurityPolicy{}
psp2.Name = "restricted"
pspList := &policyv1beta1.PodSecurityPolicyList{}
pspList.Items = append(pspList.Items, *psp1)
pspList.Items = append(pspList.Items, *psp2)
return pspList, nil
}
// GetNode returns details about node with passed in name
func (mkc *MockKubernetesClient) GetNode(name string) (*v1.Node, error) {
if mkc.GetNodeFunc != nil {

29
pkg/engine/armvariables.go
Просмотреть файл

@ -142,20 +142,21 @@ func getK8sMasterVars(cs *api.ContainerService) (map[string]interface{}, error)
}
cloudInitFiles := map[string]interface{}{
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
}
if enableEncryptionWithExternalKms {

186
pkg/engine/armvariables_test.go
Просмотреть файл

@ -147,20 +147,21 @@ func TestK8sVars(t *testing.T) {
"primaryAvailabilitySetName": "",
"primaryScaleSetName": cs.Properties.GetPrimaryScaleSetName(),
"cloudInitFiles": map[string]interface{}{
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
},
"provisionScriptParametersCommon": "[concat('" + cs.GetProvisionScriptParametersCommon(api.ProvisionScriptParametersInput{Location: common.WrapAsARMVariable("location"), ResourceGroup: common.WrapAsARMVariable("resourceGroup"), TenantID: common.WrapAsARMVariable("tenantID"), SubscriptionID: common.WrapAsARMVariable("subscriptionId"), ClientID: common.WrapAsARMVariable("servicePrincipalClientId"), ClientSecret: common.WrapAsARMVariable("singleQuote") + common.WrapAsARMVariable("servicePrincipalClientSecret") + common.WrapAsARMVariable("singleQuote"), APIServerCertificate: common.WrapAsParameter("apiServerCertificate"), KubeletPrivateKey: common.WrapAsParameter("clientPrivateKey"), ClusterKeyVaultName: common.WrapAsARMVariable("clusterKeyVaultName")}) + "')]",
"provisionScriptParametersMaster": "[concat('COSMOS_URI= MASTER_VM_NAME=',variables('masterVMNames')[variables('masterOffset')],' ETCD_PEER_URL=',variables('masterEtcdPeerURLs')[variables('masterOffset')],' ETCD_CLIENT_URL=',variables('masterEtcdClientURLs')[variables('masterOffset')],' MASTER_NODE=true NO_OUTBOUND=false AUDITD_ENABLED=false CLUSTER_AUTOSCALER_ADDON=false APISERVER_PRIVATE_KEY=',parameters('apiServerPrivateKey'),' CA_CERTIFICATE=',parameters('caCertificate'),' CA_PRIVATE_KEY=',parameters('caPrivateKey'),' MASTER_FQDN=',variables('masterFqdnPrefix'),' KUBECONFIG_CERTIFICATE=',parameters('kubeConfigCertificate'),' KUBECONFIG_KEY=',parameters('kubeConfigPrivateKey'),' ETCD_SERVER_CERTIFICATE=',parameters('etcdServerCertificate'),' ETCD_CLIENT_CERTIFICATE=',parameters('etcdClientCertificate'),' ETCD_SERVER_PRIVATE_KEY=',parameters('etcdServerPrivateKey'),' ETCD_CLIENT_PRIVATE_KEY=',parameters('etcdClientPrivateKey'),' ETCD_PEER_CERTIFICATES=',string(variables('etcdPeerCertificates')),' ETCD_PEER_PRIVATE_KEYS=',string(variables('etcdPeerPrivateKeys')),' ENABLE_AGGREGATED_APIS=',string(parameters('enableAggregatedAPIs')),' KUBECONFIG_SERVER=',variables('kubeconfigServer'))]",
@ -223,22 +224,23 @@ func TestK8sVars(t *testing.T) {
t.Fatal(err)
}
expectedMap["cloudInitFiles"] = map[string]interface{}{
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"untaintNodesScript": getBase64EncodedGzippedCustomScript(untaintNodesScript, cs),
"untaintNodesSystemdService": getBase64EncodedGzippedCustomScript(untaintNodesSystemdService, cs),
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"untaintNodesScript": getBase64EncodedGzippedCustomScript(untaintNodesScript, cs),
"untaintNodesSystemdService": getBase64EncodedGzippedCustomScript(untaintNodesSystemdService, cs),
}
diff = cmp.Diff(varMap, expectedMap)
@ -279,6 +281,7 @@ func TestK8sVars(t *testing.T) {
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"provisionCIS": getBase64EncodedGzippedCustomScript(kubernetesCISScript, cs),
@ -425,6 +428,7 @@ func TestK8sVars(t *testing.T) {
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"provisionCIS": getBase64EncodedGzippedCustomScript(kubernetesCISScript, cs),
@ -608,20 +612,21 @@ func TestK8sVars(t *testing.T) {
"primaryAvailabilitySetName": "",
"primaryScaleSetName": cs.Properties.GetPrimaryScaleSetName(),
"cloudInitFiles": map[string]interface{}{
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
},
"provisionConfigsCustomCloud": getBase64EncodedGzippedCustomScript(kubernetesCSECustomCloud, cs),
"provisionScriptParametersCommon": "[concat('" + cs.GetProvisionScriptParametersCommon(api.ProvisionScriptParametersInput{Location: common.WrapAsARMVariable("location"), ResourceGroup: common.WrapAsARMVariable("resourceGroup"), TenantID: common.WrapAsARMVariable("tenantID"), SubscriptionID: common.WrapAsARMVariable("subscriptionId"), ClientID: common.WrapAsARMVariable("servicePrincipalClientId"), ClientSecret: common.WrapAsARMVariable("singleQuote") + common.WrapAsARMVariable("servicePrincipalClientSecret") + common.WrapAsARMVariable("singleQuote"), APIServerCertificate: common.WrapAsParameter("apiServerCertificate"), KubeletPrivateKey: common.WrapAsParameter("clientPrivateKey"), ClusterKeyVaultName: common.WrapAsARMVariable("clusterKeyVaultName")}) + "')]",
@ -729,21 +734,22 @@ func TestK8sVars(t *testing.T) {
}
expectedMap["cloudInitFiles"] = map[string]interface{}{
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"systemdBPFMount": getBase64EncodedGzippedCustomScript(systemdBPFMount, cs),
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"systemdBPFMount": getBase64EncodedGzippedCustomScript(systemdBPFMount, cs),
}
expectedMap["provisionScriptParametersCommon"] = "[concat('" + cs.GetProvisionScriptParametersCommon(api.ProvisionScriptParametersInput{Location: common.WrapAsARMVariable("location"), ResourceGroup: common.WrapAsARMVariable("resourceGroup"), TenantID: common.WrapAsARMVariable("tenantID"), SubscriptionID: common.WrapAsARMVariable("subscriptionId"), ClientID: common.WrapAsARMVariable("servicePrincipalClientId"), ClientSecret: common.WrapAsARMVariable("singleQuote") + common.WrapAsARMVariable("servicePrincipalClientSecret") + common.WrapAsARMVariable("singleQuote"), APIServerCertificate: common.WrapAsParameter("apiServerCertificate"), KubeletPrivateKey: common.WrapAsParameter("clientPrivateKey"), ClusterKeyVaultName: common.WrapAsARMVariable("clusterKeyVaultName")}) + "')]"
diff = cmp.Diff(varMap, expectedMap)
@ -879,20 +885,21 @@ func TestK8sVarsMastersOnly(t *testing.T) {
"primaryAvailabilitySetName": "",
"primaryScaleSetName": "",
"cloudInitFiles": map[string]interface{}{
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
},
"provisionScriptParametersCommon": "[concat('" + cs.GetProvisionScriptParametersCommon(api.ProvisionScriptParametersInput{Location: common.WrapAsARMVariable("location"), ResourceGroup: common.WrapAsARMVariable("resourceGroup"), TenantID: common.WrapAsARMVariable("tenantID"), SubscriptionID: common.WrapAsARMVariable("subscriptionId"), ClientID: common.WrapAsARMVariable("servicePrincipalClientId"), ClientSecret: common.WrapAsARMVariable("singleQuote") + common.WrapAsARMVariable("servicePrincipalClientSecret") + common.WrapAsARMVariable("singleQuote"), APIServerCertificate: common.WrapAsParameter("apiServerCertificate"), KubeletPrivateKey: common.WrapAsParameter("clientPrivateKey"), ClusterKeyVaultName: common.WrapAsARMVariable("clusterKeyVaultName")}) + "')]",
"provisionScriptParametersMaster": "[concat('COSMOS_URI= MASTER_VM_NAME=',variables('masterVMNames')[variables('masterOffset')],' ETCD_PEER_URL=',variables('masterEtcdPeerURLs')[variables('masterOffset')],' ETCD_CLIENT_URL=',variables('masterEtcdClientURLs')[variables('masterOffset')],' MASTER_NODE=true NO_OUTBOUND=false AUDITD_ENABLED=false CLUSTER_AUTOSCALER_ADDON=false APISERVER_PRIVATE_KEY=',parameters('apiServerPrivateKey'),' CA_CERTIFICATE=',parameters('caCertificate'),' CA_PRIVATE_KEY=',parameters('caPrivateKey'),' MASTER_FQDN=',variables('masterFqdnPrefix'),' KUBECONFIG_CERTIFICATE=',parameters('kubeConfigCertificate'),' KUBECONFIG_KEY=',parameters('kubeConfigPrivateKey'),' ETCD_SERVER_CERTIFICATE=',parameters('etcdServerCertificate'),' ETCD_CLIENT_CERTIFICATE=',parameters('etcdClientCertificate'),' ETCD_SERVER_PRIVATE_KEY=',parameters('etcdServerPrivateKey'),' ETCD_CLIENT_PRIVATE_KEY=',parameters('etcdClientPrivateKey'),' ETCD_PEER_CERTIFICATES=',string(variables('etcdPeerCertificates')),' ETCD_PEER_PRIVATE_KEYS=',string(variables('etcdPeerPrivateKeys')),' ENABLE_AGGREGATED_APIS=',string(parameters('enableAggregatedAPIs')),' KUBECONFIG_SERVER=',variables('kubeconfigServer'))]",
@ -941,22 +948,23 @@ func TestK8sVarsMastersOnly(t *testing.T) {
cs.Properties.OrchestratorProfile.KubernetesConfig.EnableEncryptionWithExternalKms = to.BoolPtr(true)
expectedMap["clusterKeyVaultName"] = string("[take(concat('kv', tolower(uniqueString(concat(variables('masterFqdnPrefix'),variables('location'),parameters('nameSuffix'))))), 22)]")
expectedMap["cloudInitFiles"] = map[string]interface{}{
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"kmsKeyvaultKeySystemdService": getBase64EncodedGzippedCustomScript(kmsKeyvaultKeySystemdService, cs),
"kmsKeyvaultKeyScript": getBase64EncodedGzippedCustomScript(kmsKeyvaultKeyScript, cs),
"provisionScript": getBase64EncodedGzippedCustomScript(kubernetesCSEMainScript, cs),
"provisionSource": getBase64EncodedGzippedCustomScript(kubernetesCSEHelpersScript, cs),
"provisionInstalls": getBase64EncodedGzippedCustomScript(kubernetesCSEInstall, cs),
"provisionConfigs": getBase64EncodedGzippedCustomScript(kubernetesCSEConfig, cs),
"customSearchDomainsScript": getBase64EncodedGzippedCustomScript(kubernetesCustomSearchDomainsScript, cs),
"etcdSystemdService": getBase64EncodedGzippedCustomScript(etcdSystemdService, cs),
"dhcpv6ConfigurationScript": getBase64EncodedGzippedCustomScript(dhcpv6ConfigurationScript, cs),
"dhcpv6SystemdService": getBase64EncodedGzippedCustomScript(dhcpv6SystemdService, cs),
"kubeletSystemdService": getBase64EncodedGzippedCustomScript(kubeletSystemdService, cs),
"etcdMonitorSystemdService": getBase64EncodedGzippedCustomScript(etcdMonitorSystemdService, cs),
"healthMonitorScript": getBase64EncodedGzippedCustomScript(kubernetesHealthMonitorScript, cs),
"kubeletMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesKubeletMonitorSystemdService, cs),
"apiserverMonitorSystemdService": getBase64EncodedGzippedCustomScript(apiserverMonitorSystemdService, cs),
"dockerMonitorSystemdService": getBase64EncodedGzippedCustomScript(kubernetesDockerMonitorSystemdService, cs),
"kmsKeyvaultKeySystemdService": getBase64EncodedGzippedCustomScript(kmsKeyvaultKeySystemdService, cs),
"kmsKeyvaultKeyScript": getBase64EncodedGzippedCustomScript(kmsKeyvaultKeyScript, cs),
"apiServerAdmissionConfiguration": getBase64EncodedGzippedCustomScript(apiServerAdmissionConfiguration, cs),
}
varMap, err = GetKubernetesVariables(cs)

2
pkg/engine/const.go
Просмотреть файл

@ -91,6 +91,7 @@ const (
kubernetesHealthMonitorScript = "k8s/cloud-init/artifacts/health-monitor.sh"
// kubernetesKubeletMonitorSystemdTimer = "k8s/cloud-init/artifacts/kubelet-monitor.timer" // TODO enable
kubernetesKubeletMonitorSystemdService = "k8s/cloud-init/artifacts/kubelet-monitor.service"
apiServerAdmissionConfiguration = "k8s/cloud-init/artifacts/apiserver-admission-control.yaml"
apiserverMonitorSystemdService = "k8s/cloud-init/artifacts/apiserver-monitor.service"
kubernetesDockerMonitorSystemdService = "k8s/cloud-init/artifacts/docker-monitor.service"
etcdMonitorSystemdService = "k8s/cloud-init/artifacts/etcd-monitor.service"
@ -116,6 +117,7 @@ const (
// cloud-init destination file references
const (
apiServerAdmissionConfigurationFilepath = "/etc/kubernetes/apiserver-admission-control.yaml"
customCloudConfigCSEScriptFilepath = "/opt/azure/containers/provision_configs_custom_cloud.sh"
customCloudAzureCNIConfigCSEScriptFilepath = "/opt/azure/containers/provision_azurestack_cni.sh"
cseHelpersScriptFilepath = "/opt/azure/containers/provision_source.sh"

9
pkg/engine/customfiles.go
Просмотреть файл

@ -74,3 +74,12 @@ func getBase64CustomFile(source io.Reader) string {
cfStr = strings.Replace(cfStr, "\r\n", "\n", -1)
return getBase64EncodedGzippedCustomScriptFromStr(cfStr)
}
func containsCustomFile(customFiles []api.CustomFile, filepath string) bool {
for _, cu := range customFiles {
if cu.Dest == filepath {
return true
}
}
return false
}

54
pkg/engine/customfiles_test.go
Просмотреть файл

@ -129,3 +129,57 @@ func TestCustomFilesIntoReaders(t *testing.T) {
t.Errorf("expected length of CustomFileReader slice to be %d, but got %d instead", 2, len(cfr))
}
}
func TestContainsCustomFile(t *testing.T) {
cases := []struct {
name string
customFiles []api.CustomFile
filepath string
expected bool
}{
{
"filepath not in customFiles",
[]api.CustomFile{
{
Source: "/src/path/to/file.yaml",
Dest: "/dst/path/to/file.yaml",
},
},
"/file/not/in/custom/files.yaml",
false,
},
{
"filepath in customFiles",
[]api.CustomFile{
{
Source: "/src/path/to/file.yaml",
Dest: "/dst/path/to/file.yaml",
},
},
"/dst/path/to/file.yaml",
true,
},
{
"filepath not in customFiles, different casing",
[]api.CustomFile{
{
Source: "/src/path/to/file.yaml",
Dest: "/dst/path/to/file.yaml",
},
},
"/dst/PATH/to/file.yaml",
false,
},
}
for _, c := range cases {
c := c
t.Run(c.name, func(t *testing.T) {
t.Parallel()
actual := containsCustomFile(c.customFiles, c.filepath)
if actual != c.expected {
t.Fatalf("expected hasCustomFile to return '%t', but instead got '%t", c.expected, actual)
}
})
}
}

7
pkg/engine/template_generator.go
Просмотреть файл

@ -742,6 +742,13 @@ version = 2
"GetKMSKeyvaultKeyCSEScriptFilepath": func() string {
return kmsKeyvaultKeyCSEScriptFilepath
},
"NeedsDefaultAPIServerAdmissionConfiguration": func() bool {
configFilePath := cs.Properties.OrchestratorProfile.KubernetesConfig.APIServerConfig["--admission-control-config-file"]
return !containsCustomFile(masterCustomFiles(cs.Properties), configFilePath)
},
"GetAPIServerAdmissionConfigurationFilepath": func() string {
return apiServerAdmissionConfigurationFilepath
},
"HasPrivateAzureRegistryServer": func() bool {
return cs.Properties.OrchestratorProfile.KubernetesConfig.PrivateAzureRegistryServer != ""
},

80
pkg/engine/templates_generated.go
Просмотреть файл

@ -34,6 +34,7 @@
// ../../parts/k8s/addons/smb-flexvolume.yaml
// ../../parts/k8s/addons/tiller.yaml
// ../../parts/k8s/armparameters.t
// ../../parts/k8s/cloud-init/artifacts/apiserver-admission-control.yaml
// ../../parts/k8s/cloud-init/artifacts/apiserver-monitor.service
// ../../parts/k8s/cloud-init/artifacts/apt-preferences
// ../../parts/k8s/cloud-init/artifacts/auditd-rules
@ -15915,6 +15916,44 @@ func k8sArmparametersT() (*asset, error) {
return a, nil
}
var _k8sCloudInitArtifactsApiserverAdmissionControlYaml = []byte(`apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
configuration:
apiVersion: pod-security.admission.config.k8s.io/v1{{- if not (IsKubernetesVersionGe "1.25.0")}}beta1{{end}}
kind: PodSecurityConfiguration
defaults:
{{- /* allow everything by default, back-compatible */}}
enforce: privileged
enforce-version: latest
{{- /* cli warning if pod does not enforce the baseline stardard */}}
warn: baseline
warn-version: latest
{{- /* audit log entry if pod does not enforce the restricted stardard */}}
audit: restricted
audit-version: latest
exemptions:
usernames: []
runtimeClasses: []
namespaces: [kube-system]
`)
func k8sCloudInitArtifactsApiserverAdmissionControlYamlBytes() ([]byte, error) {
return _k8sCloudInitArtifactsApiserverAdmissionControlYaml, nil
}
func k8sCloudInitArtifactsApiserverAdmissionControlYaml() (*asset, error) {
bytes, err := k8sCloudInitArtifactsApiserverAdmissionControlYamlBytes()
if err != nil {
return nil, err
}
info := bindataFileInfo{name: "k8s/cloud-init/artifacts/apiserver-admission-control.yaml", size: 0, mode: os.FileMode(0), modTime: time.Unix(0, 0)}
a := &asset{bytes: bytes, info: info}
return a, nil
}
var _k8sCloudInitArtifactsApiserverMonitorService = []byte(`[Unit]
Description=a script that checks apiserver health and restarts if needed
After=kubelet.service
@ -16649,7 +16688,7 @@ ensureAddons() {
{{- if IsAzurePolicyAddonEnabled}}
retrycmd 120 5 30 $KUBECTL get namespace gatekeeper-system || exit_cse {{GetCSEErrorCode "ERR_ADDONS_START_FAIL"}} $GET_KUBELET_LOGS
{{- end}}
{{- if not HasCustomPodSecurityPolicy}}
{{- if and (not HasCustomPodSecurityPolicy) IsPodSecurityPolicyAddonEnabled}}
retrycmd 120 5 30 $KUBECTL get podsecuritypolicy privileged restricted || exit_cse {{GetCSEErrorCode "ERR_ADDONS_START_FAIL"}} $GET_KUBELET_LOGS
{{- end}}
replaceAddonsInit
@ -20132,6 +20171,15 @@ write_files:
{{CloudInitData "kmsKeyvaultKeyScript"}}
{{end}}
{{- if NeedsDefaultAPIServerAdmissionConfiguration}}
- path: {{GetAPIServerAdmissionConfigurationFilepath}}
permissions: "0644"
encoding: gzip
owner: root
content: !!binary |
{{CloudInitData "apiServerAdmissionConfiguration"}}
{{end}}
MASTER_MANIFESTS_CONFIG_PLACEHOLDER
MASTER_CUSTOM_FILES_PLACEHOLDER
@ -24892,6 +24940,7 @@ var _bindata = map[string]func() (*asset, error){
"k8s/addons/smb-flexvolume.yaml": k8sAddonsSmbFlexvolumeYaml,
"k8s/addons/tiller.yaml": k8sAddonsTillerYaml,
"k8s/armparameters.t": k8sArmparametersT,
"k8s/cloud-init/artifacts/apiserver-admission-control.yaml": k8sCloudInitArtifactsApiserverAdmissionControlYaml,
"k8s/cloud-init/artifacts/apiserver-monitor.service": k8sCloudInitArtifactsApiserverMonitorService,
"k8s/cloud-init/artifacts/apt-preferences": k8sCloudInitArtifactsAptPreferences,
"k8s/cloud-init/artifacts/auditd-rules": k8sCloudInitArtifactsAuditdRules,
@ -25048,20 +25097,21 @@ var _bintree = &bintree{nil, map[string]*bintree{
"armparameters.t": {k8sArmparametersT, map[string]*bintree{}},
"cloud-init": {nil, map[string]*bintree{
"artifacts": {nil, map[string]*bintree{
"apiserver-monitor.service": {k8sCloudInitArtifactsApiserverMonitorService, map[string]*bintree{}},
"apt-preferences": {k8sCloudInitArtifactsAptPreferences, map[string]*bintree{}},
"auditd-rules": {k8sCloudInitArtifactsAuditdRules, map[string]*bintree{}},
"cis.sh": {k8sCloudInitArtifactsCisSh, map[string]*bintree{}},
"cse_config.sh": {k8sCloudInitArtifactsCse_configSh, map[string]*bintree{}},
"cse_customcloud.sh": {k8sCloudInitArtifactsCse_customcloudSh, map[string]*bintree{}},
"cse_customcloud_cni.sh": {k8sCloudInitArtifactsCse_customcloud_cniSh, map[string]*bintree{}},
"cse_helpers.sh": {k8sCloudInitArtifactsCse_helpersSh, map[string]*bintree{}},
"cse_install.sh": {k8sCloudInitArtifactsCse_installSh, map[string]*bintree{}},
"cse_main.sh": {k8sCloudInitArtifactsCse_mainSh, map[string]*bintree{}},
"cse_stig_ubuntu2004.sh": {k8sCloudInitArtifactsCse_stig_ubuntu2004Sh, map[string]*bintree{}},
"default-grub": {k8sCloudInitArtifactsDefaultGrub, map[string]*bintree{}},
"dhcpv6.service": {k8sCloudInitArtifactsDhcpv6Service, map[string]*bintree{}},
"docker-monitor.service": {k8sCloudInitArtifactsDockerMonitorService, map[string]*bintree{}},
"apiserver-admission-control.yaml": {k8sCloudInitArtifactsApiserverAdmissionControlYaml, map[string]*bintree{}},
"apiserver-monitor.service": {k8sCloudInitArtifactsApiserverMonitorService, map[string]*bintree{}},
"apt-preferences": {k8sCloudInitArtifactsAptPreferences, map[string]*bintree{}},
"auditd-rules": {k8sCloudInitArtifactsAuditdRules, map[string]*bintree{}},
"cis.sh": {k8sCloudInitArtifactsCisSh, map[string]*bintree{}},
"cse_config.sh": {k8sCloudInitArtifactsCse_configSh, map[string]*bintree{}},
"cse_customcloud.sh": {k8sCloudInitArtifactsCse_customcloudSh, map[string]*bintree{}},
"cse_customcloud_cni.sh": {k8sCloudInitArtifactsCse_customcloud_cniSh, map[string]*bintree{}},
"cse_helpers.sh": {k8sCloudInitArtifactsCse_helpersSh, map[string]*bintree{}},
"cse_install.sh": {k8sCloudInitArtifactsCse_installSh, map[string]*bintree{}},
"cse_main.sh": {k8sCloudInitArtifactsCse_mainSh, map[string]*bintree{}},
"cse_stig_ubuntu2004.sh": {k8sCloudInitArtifactsCse_stig_ubuntu2004Sh, map[string]*bintree{}},
"default-grub": {k8sCloudInitArtifactsDefaultGrub, map[string]*bintree{}},
"dhcpv6.service": {k8sCloudInitArtifactsDhcpv6Service, map[string]*bintree{}},
"docker-monitor.service": {k8sCloudInitArtifactsDockerMonitorService, map[string]*bintree{}},
"docker_clear_mount_propagation_flags.conf": {k8sCloudInitArtifactsDocker_clear_mount_propagation_flagsConf, map[string]*bintree{}},
"enable-dhcpv6.sh": {k8sCloudInitArtifactsEnableDhcpv6Sh, map[string]*bintree{}},
"etc-issue": {k8sCloudInitArtifactsEtcIssue, map[string]*bintree{}},

6
pkg/kubernetes/client.go
Просмотреть файл

@ -11,6 +11,7 @@ import (
appsv1 "k8s.io/api/apps/v1"
v1 "k8s.io/api/core/v1"
policy "k8s.io/api/policy/v1beta1"
policyv1 "k8s.io/api/policy/v1beta1"
rbacv1 "k8s.io/api/rbac/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@ -84,6 +85,11 @@ func (c *ClientSetClient) ListServiceAccountsByOptions(namespace string, opts me
return c.clientset.CoreV1().ServiceAccounts(namespace).List(context.TODO(), opts)
}
// ListPodSecurityPolices returns the list of Pod Security Policies
func (c *ClientSetClient) ListPodSecurityPolices(opts metav1.ListOptions) (*policyv1.PodSecurityPolicyList, error) {
return c.clientset.PolicyV1beta1().PodSecurityPolicies().List(context.TODO(), opts)
}
// ListDeployments returns a list of deployments in the provided namespace.
func (c *ClientSetClient) ListDeployments(namespace string, opts metav1.ListOptions) (*appsv1.DeploymentList, error) {
return c.clientset.AppsV1().Deployments(namespace).List(context.TODO(), opts)

3
pkg/kubernetes/interfaces.go
Просмотреть файл

@ -7,6 +7,7 @@ import (
log "github.com/sirupsen/logrus"
appsv1 "k8s.io/api/apps/v1"
v1 "k8s.io/api/core/v1"
policyv1 "k8s.io/api/policy/v1beta1"
rbacv1 "k8s.io/api/rbac/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
@ -26,6 +27,8 @@ type Client interface {
ListNodesByOptions(opts metav1.ListOptions) (*v1.NodeList, error)
// ListServiceAccounts returns a list of Service Accounts in a namespace
ListServiceAccounts(namespace string) (*v1.ServiceAccountList, error)
// ListPodSecurityPolices returns the list of Pod Security Policies
ListPodSecurityPolices(opts metav1.ListOptions) (*policyv1.PodSecurityPolicyList, error)
// GetDaemonSet returns details about DaemonSet with passed in name.
GetDaemonSet(namespace, name string) (*appsv1.DaemonSet, error)
// GetDeployment returns a given deployment in a namespace.

16
pkg/kubernetes/mock_kubernetes/client_mock.go
Просмотреть файл

@ -12,6 +12,7 @@ import (
logrus "github.com/sirupsen/logrus"
v1 "k8s.io/api/apps/v1"
v10 "k8s.io/api/core/v1"
v1beta1 "k8s.io/api/policy/v1beta1"
v11 "k8s.io/api/rbac/v1"
v12 "k8s.io/apimachinery/pkg/apis/meta/v1"
reflect "reflect"
@ -115,6 +116,21 @@ func (mr *MockClientMockRecorder) ListServiceAccounts(namespace interface{}) *go
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ListServiceAccounts", reflect.TypeOf((*MockClient)(nil).ListServiceAccounts), namespace)
}
// ListPodSecurityPolices mocks base method
func (m *MockClient) ListPodSecurityPolices(opts v12.ListOptions) (*v1beta1.PodSecurityPolicyList, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "ListPodSecurityPolices", opts)
ret0, _ := ret[0].(*v1beta1.PodSecurityPolicyList)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// ListPodSecurityPolices indicates an expected call of ListPodSecurityPolices
func (mr *MockClientMockRecorder) ListPodSecurityPolices(opts interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ListPodSecurityPolices", reflect.TypeOf((*MockClient)(nil).ListPodSecurityPolices), opts)
}
// GetDaemonSet mocks base method
func (m *MockClient) GetDaemonSet(namespace, name string) (*v1.DaemonSet, error) {
m.ctrl.T.Helper()

1
pkg/operations/kubernetesupgrade/upgradecluster.go
Просмотреть файл

@ -195,6 +195,7 @@ func (uc *UpgradeCluster) getUpgradeWorkflow(kubeConfig string, aksEngineVersion
u := &Upgrader{}
u.Init(uc.Translator, uc.Logger, uc.ClusterTopology, uc.Client, kubeConfig, uc.StepTimeout, uc.CordonDrainTimeout, aksEngineVersion, uc.ControlPlaneOnly)
u.CurrentVersion = uc.CurrentVersion
u.Force = uc.Force
return u
}

1
pkg/operations/kubernetesupgrade/upgradecluster_test.go
Просмотреть файл

@ -749,6 +749,7 @@ var _ = Describe("Upgrade Kubernetes cluster tests", func() {
uc := UpgradeCluster{
Translator: &i18n.Translator{},
Logger: log.NewEntry(log.New()),
Force: true,
}
mockClient := armhelpers.MockAKSEngineClient{

52
pkg/operations/kubernetesupgrade/upgrader.go
Просмотреть файл

@ -26,6 +26,7 @@ import (
"github.com/sirupsen/logrus"
appsv1 "k8s.io/api/apps/v1"
v1 "k8s.io/api/core/v1"
policyv1beta1 "k8s.io/api/policy/v1beta1"
rbacv1 "k8s.io/api/rbac/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
@ -42,6 +43,7 @@ type Upgrader struct {
AKSEngineVersion string
CurrentVersion string
ControlPlaneOnly bool
Force bool
}
type vmStatus int
@ -77,6 +79,14 @@ func (ku *Upgrader) Init(translator *i18n.Translator, logger *logrus.Entry, clus
// RunUpgrade runs the upgrade pipeline
func (ku *Upgrader) RunUpgrade() error {
if err := ku.validatePodSecurityPolices(); err != nil {
if ku.Force {
ku.logger.Warning("Error validating PodSecurityPolices")
} else {
ku.logger.Warning("Error validating PodSecurityPolices. Consider using --force if you really want to proceed")
return errors.Wrap(err, "error validating PodSecurityPolices")
}
}
controlPlaneUpgradeTimeout := perNodeUpgradeTimeout
if ku.ClusterTopology.DataModel.Properties.MasterProfile.Count > 0 {
controlPlaneUpgradeTimeout = perNodeUpgradeTimeout * time.Duration(ku.ClusterTopology.DataModel.Properties.MasterProfile.Count)
@ -938,3 +948,45 @@ func getAvailableIndex(vms map[int]*vmInfo) int {
return maxIndex + 1
}
// validatePodSecurityPolices tries to guess if the user needs to manually migrate from PSP to PSA
// before upgrading to Kubernetes v1.25
func (ku *Upgrader) validatePodSecurityPolices() error {
if common.IsKubernetesVersionGe(ku.CurrentVersion, common.PodSecurityPolicyRemovedVersion) {
return nil
}
ku.logger.Debug("Checking no user-created PodSecurityPolicies still present.")
client, err := ku.getKubernetesClient(getResourceTimeout)
if err != nil {
return errors.Wrap(err, "error getting Kubernetes client")
}
policies, err := client.ListPodSecurityPolices(metav1.ListOptions{})
if err != nil {
return errors.Wrap(err, "error listing PodSecurityPolices")
}
next := ku.DataModel.Properties.OrchestratorProfile.OrchestratorVersion
return validateUserCreatedPodSecurityPolices(ku.CurrentVersion, next, policies.Items)
}
// validatePodSecurityPolices tries to guess if the user needs to manually migrate from PSP to PSA
func validateUserCreatedPodSecurityPolices(currentVersion, upgradeVersion string, policies []policyv1beta1.PodSecurityPolicy) error {
if common.IsKubernetesVersionGe(currentVersion, common.PodSecurityPolicyRemovedVersion) {
return nil
}
if !common.IsKubernetesVersionGe(upgradeVersion, common.PodSecurityPolicyRemovedVersion) {
return nil
}
if len(policies) > 2 {
return errors.New("user-created PodSecurityPolices found in the cluster (try 'kubectl get psp'), " +
"migrate from PodSecurityPolices before upgrading to Kubernetes v1.25+, " +
"see https://github.com/Azure/aks-engine-azurestack/blob/master/docs/topics/pod-security.md")
}
for _, policy := range policies {
if policy.Name != "privileged" && policy.Name != "restricted" {
return errors.New("user-created PodSecurityPolices found in the cluster (try 'kubectl get psp'), " +
"migrate from PodSecurityPolices before upgrading to Kubernetes v1.25+, " +
"see https://github.com/Azure/aks-engine-azurestack/blob/master/docs/topics/pod-security.md")
}
}
return nil
}

79
pkg/operations/kubernetesupgrade/upgrader_test.go Normal file
Просмотреть файл

@ -0,0 +1,79 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.
package kubernetesupgrade
import (
"testing"
policyv1beta1 "k8s.io/api/policy/v1beta1"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestValidateUserCreatedPodSecurityPolices(t *testing.T) {
cases := []struct {
name string
curr string
next string
policies []policyv1beta1.PodSecurityPolicy
errorExpected bool
}{
{
"Upgrade from v1.23 to v1.24, no not validate",
"1.23.0",
"1.24.0",
[]policyv1beta1.PodSecurityPolicy{},
false,
},
{
"Upgrade from v1.25, no not validate",
"1.25.0",
"1.26.0",
[]policyv1beta1.PodSecurityPolicy{},
false,
},
{
"Upgrade to v1.25, only default policies present",
"1.24.0",
"1.25.0",
[]policyv1beta1.PodSecurityPolicy{
{ObjectMeta: v1.ObjectMeta{Name: "privileged"}},
{ObjectMeta: v1.ObjectMeta{Name: "restricted"}},
},
false,
},
{
"Upgrade to v1.25, default policies present plus extra",
"1.24.0",
"1.25.0",
[]policyv1beta1.PodSecurityPolicy{
{ObjectMeta: v1.ObjectMeta{Name: "privileged"}},
{ObjectMeta: v1.ObjectMeta{Name: "restricted"}},
{ObjectMeta: v1.ObjectMeta{Name: "extra"}},
},
true,
},
{
"Upgrade to v1.25, non default policies present",
"1.24.0",
"1.25.0",
[]policyv1beta1.PodSecurityPolicy{
{ObjectMeta: v1.ObjectMeta{Name: "privileged"}},
{ObjectMeta: v1.ObjectMeta{Name: "extra"}},
},
true,
},
}
for _, c := range cases {
c := c
t.Run(c.name, func(t *testing.T) {
t.Parallel()
err := validateUserCreatedPodSecurityPolices(c.curr, c.next, c.policies)
if err == nil && c.errorExpected {
t.Fatal("expected validateUserCreatedPodSecurityPolices to return an error but it did not")
} else if err != nil && !c.errorExpected {
t.Fatalf("validateUserCreatedPodSecurityPolices not expected to return an error but it returned '%s'", err)
}
})
}
}