Added "cluster-manifest" folder from AKS Baseline RI for deployment through Flux fo shared services

README.md

@@ -43,8 +43,8 @@ We also provide example of metrics of interest from these Shared-Services that c
 
 In this section we demonstrate two implementation options:
 
- * A GitOps solution using the AKS [Flux](https://fluxcd.io/) extension.
- * A CI/CD pipeline built using GitHub Actions. Note that this feature is not yet implemented, see issue https://github.com/Azure/aks-baseline-automation/issues/28
+ * A GitOps solution using the AKS [Flux](https://fluxcd.io/) extension. Note that this feature is not yet implemented, see issue https://github.com/Azure/aks-baseline-automation/issues/30.
+ * A CI/CD pipeline built using GitHub Actions. Note that this feature is not yet implemented, see issue https://github.com/Azure/aks-baseline-automation/issues/28.
 
 
 The GitOps solution features:

docs/IaC-prerequisites.md

@@ -76,7 +76,9 @@ If you opt-in to setup a shell on your machine, there are required access and to
    sudo apt-get update && sudo apt-get install terraform
    ```
 
-4. Clone/download this repo locally, or even better fork this repository.
+4. Register the Azure features used by this Reference implementation. For the list of these features look at the pre-checks performed in the IaC workflows (for example [IaC-bicep-AKS.yml](../.github/workflows/IaC-bicep-AKS.yml) 
+   
+5. Clone/download this repo locally, or even better fork this repository.
 
    > :twisted_rightwards_arrows: If you have forked this reference implementation repo, you'll be able to customize some of the files and commands for a more personalized experience; also ensure references to repos mentioned are updated to use your own (e.g. the following `GITHUB_REPO`).
 

shared-services/README.md

@@ -1,4 +1,4 @@
 # Shared Services
 This folder contains helm charts and other artifacts used to deploy common services used across multiple clusters and multiple applications.
 
-Example of shared services could be third-party services such as [Prisma defender](https://docs.paloaltonetworks.com/prisma/prisma-cloud) and [Splunk](https://github.com/splunk/splunk-connect-for-kubernetes) or open source services such as [NGINX](https://www.nginx.com/resources/glossary/kubernetes-ingress-controller), [KEDA](https://keda.sh), [External-dns](https://github.com/kubernetes-sigs/external-dns#:~:text=ExternalDNS%20supports%20multiple%20DNS%20providers%20which%20have%20been,and%20we%20have%20limited%20resources%20to%20test%20changes.), [Cert-manager](https://cert-manager.io/docs/) or [Istio](https://istio.io/).
+Example of shared services could be third-party services such as [Traefik](https://doc.traefik.io/traefik/v1.7/user-guide/kubernetes/?msclkid=2309fcb3b1bc11ec92c03b099f5d4e1c), [Prisma defender](https://docs.paloaltonetworks.com/prisma/prisma-cloud) and [Splunk](https://github.com/splunk/splunk-connect-for-kubernetes) or open source services such as [NGINX](https://www.nginx.com/resources/glossary/kubernetes-ingress-controller), [KEDA](https://keda.sh), [External-dns](https://github.com/kubernetes-sigs/external-dns#:~:text=ExternalDNS%20supports%20multiple%20DNS%20providers%20which%20have%20been,and%20we%20have%20limited%20resources%20to%20test%20changes.), [Cert-manager](https://cert-manager.io/docs/) or [Istio](https://istio.io/).

shared-services/cluster-manifests/README.md

@@ -0,0 +1,61 @@
+# Cluster Baseline Configuration Files (GitOps)
+
+> Note: This is part of the Azure Kubernetes Service (AKS) Baseline cluster reference implementation. For more information check out the [readme file in the root](../README.md).
+
+This is the root of the GitOps configuration directory. These Kubernetes object files are expected to be deployed via our in-cluster Flux operator. They are our AKS cluster's baseline configurations. Generally speaking, they are workload agnostic and tend to all cluster-wide configuration concerns.
+
+## Contents
+
+* Default Namespaces
+* Kubernetes RBAC Role Assignments (cluster and namespace) to Azure AD Groups. _Optional_
+* [Kured](#kured)
+* Ingress Network Policy
+* Azure Monitor Prometheus Scraping
+* Azure AD Pod Identity
+
+### Kured
+
+Kured is included as a solution to handle occasional required reboots from daily OS patching. This open-source software component is only needed if you require a managed rebooting solution between weekly [node image upgrades](https://docs.microsoft.com/azure/aks/node-image-upgrade). Building a process around deploying node image upgrades [every week](https://github.com/Azure/AKS/releases) satisfies most organizational weekly patching cadence requirements. Combined with most security patches on Linux not requiring reboots often, this leaves your cluster in a well supported state. If weekly node image upgrades satisfies your business requirements, then remove Kured from this solution by deleting [`kured.yaml`](./cluster-baseline-settings/kured.yaml). If however weekly patching using node image upgrades is not sufficient and you need to respond to daily security updates that mandate a reboot ASAP, then using a solution like Kured will help you achieve that objective. **Kured is not supported by Microsoft Support.**
+
+## Private bootstrapping repository
+
+Typically, your bootstrapping repository wouldn't be a public facing repository like this one, but instead a private GitHub or Azure DevOps repo. The Flux operator deployed with the cluster supports private git repositories as your bootstrapping source. In addition to requiring network line of sight to the repository from your cluster's nodes, you'll also need to ensure that you've provided the necessary credentials. This can come, typically, in the form of certificate based SSH or personal access tokens (PAT), both ideally scoped as read-only to the repo with no additional permissions.
+
+If you are using bicep modify the [`cluster.parameters.json`](../../IaC/bicep/rg-spoke/cluster.parameters.json) file as follows.
+
+### Git over SSH
+
+Use the following pattern in your `fluxConfigurations` extension resource.
+
+```json
+{
+    "type": "providers/fluxConfigurations",
+    "properties": {
+        "gitRepository": {
+            "url": "git@github.com:yourorg/yourrepo.git"
+        },
+        "configurationProtectedSettings": {
+            "sshPrivateKey": "<Base64 encoded PEM private key>"
+        }
+    }
+}
+```
+
+### Git over HTTPS with personal access tokens
+
+You'll use the following pattern in your `fluxConfigurations` extension resource.
+
+```json
+{
+    "type": "providers/fluxConfigurations",
+    "properties": {
+        "gitRepository": {
+            "url": "https://github.com/yourorg/yourrepo.git",
+            "httpsUser": "<Username>"
+        },
+        "configurationProtectedSettings": {
+            "httpsKey": "<Base64 encoded, UTF-8 encoded Personal Access Token>"
+        }
+    }
+}
+```

shared-services/cluster-manifests/a0008/ingress-network-policy.yaml

@@ -0,0 +1,16 @@
+kind: NetworkPolicy
+apiVersion: networking.k8s.io/v1
+metadata:
+  name: allow-only-ingress-to-workload
+  namespace: a0008
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: aspnetapp
+  ingress:
+  - from:
+    - namespaceSelector: {}
+      podSelector:
+        matchLabels:
+          app.kubernetes.io/name: traefik-ingress-ilb
+          app.kubernetes.io/instance: traefik-ingress-ilb

shared-services/cluster-manifests/a0008/ns-a0008.yaml

@@ -0,0 +1,4 @@
+kind: Namespace
+apiVersion: v1
+metadata:
+  name: a0008

shared-services/cluster-manifests/a0008/rbac.yaml

@@ -0,0 +1,42 @@
+# This file is exclusively to be used when you are using Azure AD direct assignment for Kubertnetes RBAC 
+# and NOT when you are using Azure RBAC as your Kubernetes RBAC backing store.
+#
+#  Map k8s user facing namespace roles to AAD groups: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: RoleBinding
+#metadata:
+#  name: namespace-admin
+#  namespace: a0008
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: admin
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-namespace-role-binding>
+#---
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: RoleBinding
+#metadata:
+#  name: namespace-editor
+#  namespace: a0008
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: edit
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-namespace-role-binding>
+#---
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: RoleBinding
+#metadata:
+#  name: namespace-viewer
+#  namespace: a0008
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: view
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-namespace-role-binding>

shared-services/cluster-manifests/cluster-baseline-settings/aad-pod-identity.yaml

@@ -0,0 +1,655 @@
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    api-approved.kubernetes.io: unapproved
+    controller-gen.kubebuilder.io/version: v0.5.0
+  name: azureassignedidentities.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  names:
+    kind: AzureAssignedIdentity
+    listKind: AzureAssignedIdentityList
+    plural: azureassignedidentities
+    singular: azureassignedidentity
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: AzureAssignedIdentity contains the identity <-> pod mapping which is matched.
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: AzureAssignedIdentitySpec contains the relationship between an AzureIdentity and an AzureIdentityBinding.
+            properties:
+              azureBindingRef:
+                description: AzureBindingRef is an embedded resource referencing the AzureIdentityBinding used by the AzureAssignedIdentity, which requires x-kubernetes-embedded-resource fields to be true
+                properties:
+                  apiVersion:
+                    description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+                    type: string
+                  kind:
+                    description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                    type: string
+                  metadata:
+                    type: object
+                  spec:
+                    description: AzureIdentityBindingSpec matches the pod with the Identity. Used to indicate the potential matches to look for between the pod/deployment and the identities present.
+                    properties:
+                      azureIdentity:
+                        type: string
+                      metadata:
+                        type: object
+                      selector:
+                        type: string
+                      weight:
+                        description: Weight is used to figure out which of the matching identities would be selected.
+                        type: integer
+                    type: object
+                  status:
+                    description: AzureIdentityBindingStatus contains the status of an AzureIdentityBinding.
+                    properties:
+                      availableReplicas:
+                        format: int32
+                        type: integer
+                      metadata:
+                        type: object
+                    type: object
+                type: object
+                x-kubernetes-embedded-resource: true
+              azureIdentityRef:
+                description: AzureIdentityRef is an embedded resource referencing the AzureIdentity used by the AzureAssignedIdentity, which requires x-kubernetes-embedded-resource fields to be true
+                properties:
+                  apiVersion:
+                    description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+                    type: string
+                  kind:
+                    description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                    type: string
+                  metadata:
+                    type: object
+                  spec:
+                    description: AzureIdentitySpec describes the credential specifications of an identity on Azure.
+                    properties:
+                      adEndpoint:
+                        type: string
+                      adResourceID:
+                        description: For service principal. Option param for specifying the  AD details.
+                        type: string
+                      auxiliaryTenantIDs:
+                        description: Service principal auxiliary tenant ids
+                        items:
+                          type: string
+                        nullable: true
+                        type: array
+                      clientID:
+                        description: Both User Assigned MSI and SP can use this field.
+                        type: string
+                      clientPassword:
+                        description: Used for service principal
+                        properties:
+                          name:
+                            description: Name is unique within a namespace to reference a secret resource.
+                            type: string
+                          namespace:
+                            description: Namespace defines the space within which the secret name must be unique.
+                            type: string
+                        type: object
+                      metadata:
+                        type: object
+                      replicas:
+                        format: int32
+                        nullable: true
+                        type: integer
+                      resourceID:
+                        description: User assigned MSI resource id.
+                        type: string
+                      tenantID:
+                        description: Service principal primary tenant id.
+                        type: string
+                      type:
+                        description: UserAssignedMSI or Service Principal
+                        type: integer
+                    type: object
+                  status:
+                    description: AzureIdentityStatus contains the replica status of the resource.
+                    properties:
+                      availableReplicas:
+                        format: int32
+                        type: integer
+                      metadata:
+                        type: object
+                    type: object
+                type: object
+                x-kubernetes-embedded-resource: true
+              metadata:
+                type: object
+              nodename:
+                type: string
+              pod:
+                type: string
+              podNamespace:
+                type: string
+              replicas:
+                format: int32
+                nullable: true
+                type: integer
+            type: object
+          status:
+            description: AzureAssignedIdentityStatus contains the replica status of the resource.
+            properties:
+              availableReplicas:
+                format: int32
+                type: integer
+              metadata:
+                type: object
+              status:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    api-approved.kubernetes.io: unapproved
+    controller-gen.kubebuilder.io/version: v0.5.0
+  name: azureidentities.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  names:
+    kind: AzureIdentity
+    listKind: AzureIdentityList
+    plural: azureidentities
+    singular: azureidentity
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.type
+      name: Type
+      type: string
+    - jsonPath: .spec.clientID
+      name: ClientID
+      type: string
+    - description: CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1
+    schema:
+      openAPIV3Schema:
+        description: AzureIdentity is the specification of the identity data structure.
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: AzureIdentitySpec describes the credential specifications of an identity on Azure.
+            properties:
+              adEndpoint:
+                type: string
+              adResourceID:
+                description: For service principal. Option param for specifying the  AD details.
+                type: string
+              auxiliaryTenantIDs:
+                description: Service principal auxiliary tenant ids
+                items:
+                  type: string
+                nullable: true
+                type: array
+              clientID:
+                description: Both User Assigned MSI and SP can use this field.
+                type: string
+              clientPassword:
+                description: Used for service principal
+                properties:
+                  name:
+                    description: Name is unique within a namespace to reference a secret resource.
+                    type: string
+                  namespace:
+                    description: Namespace defines the space within which the secret name must be unique.
+                    type: string
+                type: object
+              metadata:
+                type: object
+              replicas:
+                format: int32
+                nullable: true
+                type: integer
+              resourceID:
+                description: User assigned MSI resource id.
+                type: string
+              tenantID:
+                description: Service principal primary tenant id.
+                type: string
+              type:
+                description: UserAssignedMSI or Service Principal
+                type: integer
+            type: object
+          status:
+            description: AzureIdentityStatus contains the replica status of the resource.
+            properties:
+              availableReplicas:
+                format: int32
+                type: integer
+              metadata:
+                type: object
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    api-approved.kubernetes.io: unapproved
+    controller-gen.kubebuilder.io/version: v0.5.0
+  name: azureidentitybindings.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  names:
+    kind: AzureIdentityBinding
+    listKind: AzureIdentityBindingList
+    plural: azureidentitybindings
+    singular: azureidentitybinding
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.azureIdentity
+      name: AzureIdentity
+      type: string
+    - jsonPath: .spec.selector
+      name: Selector
+      type: string
+    - description: CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1
+    schema:
+      openAPIV3Schema:
+        description: AzureIdentityBinding brings together the spec of matching pods and the identity which they can use.
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: AzureIdentityBindingSpec matches the pod with the Identity. Used to indicate the potential matches to look for between the pod/deployment and the identities present.
+            properties:
+              azureIdentity:
+                type: string
+              metadata:
+                type: object
+              selector:
+                type: string
+              weight:
+                description: Weight is used to figure out which of the matching identities would be selected.
+                type: integer
+            type: object
+          status:
+            description: AzureIdentityBindingStatus contains the status of an AzureIdentityBinding.
+            properties:
+              availableReplicas:
+                format: int32
+                type: integer
+              metadata:
+                type: object
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    api-approved.kubernetes.io: unapproved
+    controller-gen.kubebuilder.io/version: v0.5.0
+  name: azurepodidentityexceptions.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  names:
+    kind: AzurePodIdentityException
+    listKind: AzurePodIdentityExceptionList
+    plural: azurepodidentityexceptions
+    singular: azurepodidentityexception
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: AzurePodIdentityException contains the pod selectors for all pods that don't require NMI to process and request token on their behalf.
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: AzurePodIdentityExceptionSpec matches pods with the selector defined. If request originates from a pod that matches the selector, nmi will proxy the request and send response back without any validation.
+            properties:
+              metadata:
+                type: object
+              podLabels:
+                additionalProperties:
+                  type: string
+                type: object
+            type: object
+          status:
+            description: AzurePodIdentityExceptionStatus contains the status of an AzurePodIdentityException.
+            properties:
+              metadata:
+                type: object
+              status:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: aad-pod-id-mic-service-account
+  namespace: cluster-baseline-settings
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: aad-pod-id-nmi-service-account
+  namespace: cluster-baseline-settings
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: aad-pod-id-mic-role
+rules:
+- apiGroups: ["apiextensions.k8s.io"]
+  resources: ["customresourcedefinitions"]
+  verbs: ["*"]
+- apiGroups: [""]
+  resources: ["pods", "nodes"]
+  verbs: [ "list", "watch" ]
+- apiGroups: [""]
+  resources: ["events"]
+  verbs: ["create", "patch"]
+- apiGroups: [""]
+  resources: ["configmaps"]
+  verbs: ["get", "create", "update"]
+- apiGroups: [""]
+  resources: ["endpoints"]
+  verbs: ["create", "get","update"]
+- apiGroups: ["aadpodidentity.k8s.io"]
+  resources: ["azureidentitybindings", "azureidentities"]
+  verbs: ["get", "list", "watch", "post", "update"]
+- apiGroups: ["aadpodidentity.k8s.io"]
+  resources: ["azurepodidentityexceptions"]
+  verbs: ["list", "update"]
+- apiGroups: ["aadpodidentity.k8s.io"]
+  resources: ["azureassignedidentities"]
+  verbs: ["*"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: aad-pod-id-nmi-role
+rules:
+- apiGroups: ["apiextensions.k8s.io"]
+  resources: ["customresourcedefinitions"]
+  verbs: ["get", "list"]
+- apiGroups: [""]
+  resources: ["pods"]
+  verbs: ["get", "list", "watch"]
+- apiGroups: [""]
+  resources: ["secrets"]
+  verbs: ["get"]
+- apiGroups: ["aadpodidentity.k8s.io"]
+  resources: ["azureidentitybindings", "azureidentities", "azurepodidentityexceptions"]
+  verbs: ["get", "list", "watch"]
+- apiGroups: ["aadpodidentity.k8s.io"]
+  resources: ["azureassignedidentities"]
+  verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: aad-pod-id-mic-binding
+  labels:
+    k8s-app: aad-pod-id-mic-binding
+subjects:
+- kind: ServiceAccount
+  name: aad-pod-id-mic-service-account
+  namespace: cluster-baseline-settings
+roleRef:
+  kind: ClusterRole
+  name: aad-pod-id-mic-role
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: aad-pod-id-nmi-binding
+  labels:
+    k8s-app: aad-pod-id-nmi-binding
+subjects:
+- kind: ServiceAccount
+  name: aad-pod-id-nmi-service-account
+  namespace: cluster-baseline-settings
+roleRef:
+  kind: ClusterRole
+  name: aad-pod-id-nmi-role
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    component: nmi
+    tier: node
+    k8s-app: aad-pod-id
+  name: nmi
+  namespace: cluster-baseline-settings
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  selector:
+    matchLabels:
+      component: nmi
+      tier: node
+  template:
+    metadata:
+      labels:
+        component: nmi
+        tier: node
+    spec:
+      serviceAccountName: aad-pod-id-nmi-service-account
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
+      volumes:
+      - hostPath:
+          path: /run/xtables.lock
+          type: FileOrCreate
+        name: iptableslock
+      - name: kubelet-config
+        hostPath:
+          path: /etc/default/kubelet
+          type: FileOrCreate
+      containers:
+      - name: nmi
+        image: "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.8.5"
+        args:
+          - "--node=$(NODE_NAME)"
+          - "--http-probe-port=8085"
+          - "--enableScaleFeatures=true"
+          - "--metadata-header-required=true"
+        env:
+          - name: HOST_IP
+            valueFrom:
+              fieldRef:
+                fieldPath: status.podIP
+          - name: NODE_NAME
+            valueFrom:
+              fieldRef:
+                fieldPath: spec.nodeName
+        resources:
+          limits:
+            cpu: 200m
+            memory: 512Mi
+          requests:
+            cpu: 100m
+            memory: 256Mi
+        securityContext:
+          runAsUser: 0
+          capabilities:
+            drop:
+            - ALL
+            add:
+            - DAC_READ_SEARCH
+            - NET_ADMIN
+            - NET_RAW
+        volumeMounts:
+        - mountPath: /run/xtables.lock
+          name: iptableslock
+        - name: kubelet-config
+          mountPath: /etc/default/kubelet
+          readOnly: true
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8085
+          initialDelaySeconds: 10
+          periodSeconds: 5
+      nodeSelector:
+        kubernetes.io/os: linux
+        agentpool: npuser01
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    component: mic
+    k8s-app: aad-pod-id
+  name: mic
+  namespace: cluster-baseline-settings
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      component: mic
+      app: mic
+  template:
+    metadata:
+      labels:
+        component: mic
+        app: mic
+    spec:
+      serviceAccountName: aad-pod-id-mic-service-account
+      containers:
+      - name: mic
+        image: "mcr.microsoft.com/oss/azure/aad-pod-identity/mic:v1.8.5"
+        args:
+          - "--cloudconfig=/etc/kubernetes/azure.json"
+          - "--logtostderr"
+        securityContext:
+          runAsUser: 0
+        env:
+        - name: MIC_POD_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        resources:
+          limits:
+            cpu: 200m
+            memory: 1024Mi
+          requests:
+            cpu: 100m
+            memory: 256Mi
+        volumeMounts:
+        - name: k8s-azure-file
+          mountPath: /etc/kubernetes/azure.json
+          readOnly: true
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8080
+          initialDelaySeconds: 10
+          periodSeconds: 5
+      volumes:
+      - name: k8s-azure-file
+        hostPath:
+          path: /etc/kubernetes/azure.json
+      nodeSelector:
+        kubernetes.io/os: linux
+        agentpool: npuser01
+---
+apiVersion: "aadpodidentity.k8s.io/v1"
+kind: AzurePodIdentityException
+metadata:
+  name: mic-exception
+  namespace: cluster-baseline-settings
+spec:
+  podLabels:
+    app: mic
+    component: mic
+---
+apiVersion: "aadpodidentity.k8s.io/v1"
+kind: AzurePodIdentityException
+metadata:
+  name: aks-addon-exception
+  namespace: kube-system
+spec:
+  podLabels:
+    kubernetes.azure.com/managedby: aks
+---
+apiVersion: aadpodidentity.k8s.io/v1
+kind: AzurePodIdentityException
+metadata:
+  name: flux-extension-exception
+  namespace: flux-system
+spec:
+  podLabels:
+    app.kubernetes.io/component: fluxconfig-agent

shared-services/cluster-manifests/cluster-baseline-settings/kured.yaml

@@ -0,0 +1,161 @@
+# Source: https://github.com/weaveworks/kured/releases/download/1.9.0/kured-1.9.0-dockerhub.yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: kured
+rules:
+# Allow kured to read spec.unschedulable
+# Allow kubectl to drain/uncordon
+#
+# NB: These permissions are tightly coupled to the bundled version of kubectl; the ones below
+# match https://github.com/kubernetes/kubernetes/blob/v1.19.4/staging/src/k8s.io/kubectl/pkg/cmd/drain/drain.go
+#
+- apiGroups: [""]
+  resources: ["nodes"]
+  verbs:     ["get", "patch"]
+- apiGroups: [""]
+  resources: ["pods"]
+  verbs:     ["list", "delete", "get"]
+- apiGroups: ["apps"]
+  resources: ["daemonsets"]
+  verbs:     ["get"]
+- apiGroups: [""]
+  resources: ["pods/eviction"]
+  verbs:     ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: kured
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: kured
+subjects:
+- kind: ServiceAccount
+  name: kured
+  namespace: cluster-baseline-settings
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  namespace: cluster-baseline-settings
+  name: kured
+rules:
+# Allow kured to lock/unlock itself
+- apiGroups:     ["apps"]
+  resources:     ["daemonsets"]
+  resourceNames: ["kured"]
+  verbs:         ["update"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  namespace: cluster-baseline-settings
+  name: kured
+subjects:
+- kind: ServiceAccount
+  namespace: cluster-baseline-settings
+  name: kured
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: kured
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kured
+  namespace: cluster-baseline-settings
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kured                           # Must match `--ds-name`
+  namespace: cluster-baseline-settings  # Must match `--ds-namespace`
+spec:
+  selector:
+    matchLabels:
+      name: kured
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        name: kured
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/path: "/metrics"
+        prometheus.io/port: "8080"
+    spec:
+      serviceAccountName: kured
+      tolerations:
+        - key: node-role.kubernetes.io/master
+          effect: NoSchedule
+        - effect: NoSchedule
+          key: CriticalAddonsOnly
+          operator: Equal
+          value: "true"
+      hostPID: true # Facilitate entering the host mount namespace via init
+      restartPolicy: Always
+      nodeSelector:
+        kubernetes.io/arch: amd64
+        kubernetes.io/os: linux
+      containers:
+        - name: kured
+          # PRODUCTION READINESS CHANGE REQUIRED
+          # This image should be sourced from a non-public container registry, such as the
+          # one deployed along side of this reference implementation.
+          # az acr import --source docker.io/weaveworks/kured:1.9.0 -n <your-acr-instance-name>
+          # and then set this to
+          # image: <your-acr-instance-name>.azurecr.io/weaveworks/kured:1.9.0
+          image: docker.io/weaveworks/kured:1.9.0
+          imagePullPolicy: IfNotPresent
+          resources:
+            limits:
+              cpu: 500m
+              memory: 48Mi
+            requests:
+              cpu: 200m
+              memory: 16Mi
+          securityContext:
+            privileged: true # Give permission to nsenter /proc/1/ns/mnt
+          env:
+            - name: KURED_NODE_ID
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+          command:
+            - /usr/bin/kured
+            - --ds-namespace=cluster-baseline-settings
+#            - --force-reboot=false
+#            - --drain-grace-period=-1
+#            - --skip-wait-for-delete-timeout=0
+#            - --drain-timeout=0
+#            - --period=1h
+#            - --ds-name=kured
+#            - --lock-annotation=weave.works/kured-node-lock
+#            - --lock-ttl=0
+#            - --prometheus-url=http://prometheus.monitoring.svc.cluster.local
+#            - --alert-filter-regexp=^RebootRequired$
+#            - --alert-firing-only=false
+#            - --reboot-sentinel=/var/run/reboot-required
+#            - --prefer-no-schedule-taint=""
+#            - --reboot-sentinel-command=""
+#            - --slack-hook-url=https://hooks.slack.com/...
+#            - --slack-username=prod
+#            - --slack-channel=alerting
+#            - --notify-url="" # See also shoutrrr url format
+#            - --message-template-drain=Draining node %s
+#            - --message-template-drain=Rebooting node %s
+#            - --blocking-pod-selector=runtime=long,cost=expensive
+#            - --blocking-pod-selector=name=temperamental
+#            - --blocking-pod-selector=...
+#            - --reboot-days=sun,mon,tue,wed,thu,fri,sat
+#            - --reboot-delay=90s
+#            - --start-time=0:00
+#            - --end-time=23:59:59
+#            - --time-zone=UTC
+#            - --annotate-nodes=false
+#            - --lock-release-delay=30m
+#            - --log-format=text

shared-services/cluster-manifests/cluster-baseline-settings/ns-cluster-baseline-settings.yaml

@@ -0,0 +1,4 @@
+kind: Namespace
+apiVersion: v1
+metadata:
+  name: cluster-baseline-settings

shared-services/cluster-manifests/cluster-baseline-settings/rbac.yaml

@@ -0,0 +1,42 @@
+# This file is exclusively to be used when you are using Azure AD direct assignment for Kubertnetes RBAC 
+# and NOT when you are using Azure RBAC as your Kubernetes RBAC backing store.
+#
+#  Map k8s user facing namespace roles to AAD groups: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: RoleBinding
+#metadata:
+#  name: namespace-admin
+#  namespace: cluster-baseline-settings
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: admin
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-namespace-role-binding>
+#---
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: RoleBinding
+#metadata:
+#  name: a0008-edit
+#  namespace: cluster-baseline-settings
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: namespace-editer
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-namespace-role-binding>
+#---
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: RoleBinding
+#metadata:
+#  name: namespace-viewer
+#  namespace: cluster-baseline-settings
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: view
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-namespace-role-binding>

shared-services/cluster-manifests/cluster-rbac.yaml

@@ -0,0 +1,27 @@
+# This file is exclusively to be used when you are using Azure AD direct assignment for Kubertnetes RBAC 
+# and NOT when you are using Azure RBAC as your Kubernetes RBAC backing store.
+#
+#  Mapping k8s cluster user facing roles to AAD groups: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: ClusterRoleBinding
+#metadata:
+#  name: cluster-admins
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: cluster-admin
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-cluster-role-binding>
+#---
+#apiVersion: rbac.authorization.k8s.io/v1
+#kind: ClusterRoleBinding
+#metadata:
+#  name: cluster-viewers
+#roleRef:
+#  apiGroup: rbac.authorization.k8s.io
+#  kind: ClusterRole
+#  name: view
+#subjects:
+#  - kind: Group
+#    name: <replace-with-an-aad-group-object-id-for-this-cluster-role-binding>

shared-services/cluster-manifests/kube-system/container-azm-ms-agentconfig.yaml

@@ -0,0 +1,45 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: container-azm-ms-agentconfig
+  namespace: kube-system
+data:
+  # https://raw.githubusercontent.com/microsoft/Docker-Provider/ci_prod/kubernetes/container-azm-ms-agentconfig.yaml
+  schema-version: v1
+  config-version: ver1
+  log-data-collection-settings: |-
+    [log_collection_settings]
+       [log_collection_settings.stdout]
+          enabled = true
+          exclude_namespaces = ["kube-system"]
+       [log_collection_settings.stderr]
+          enabled = true
+          exclude_namespaces = ["kube-system"]
+       [log_collection_settings.env_var]
+          enabled = true
+       [log_collection_settings.enrich_container_logs]
+          enabled = false
+       [log_collection_settings.collect_all_kube_events]
+          enabled = false
+  prometheus-data-collection-settings: |-
+    [prometheus_data_collection_settings.cluster]
+        interval = "1m"
+        monitor_kubernetes_pods = true
+        monitor_kubernetes_pods_namespaces = ["a0008","cluster-baseline-settings"]
+    [prometheus_data_collection_settings.node]
+        interval = "1m"
+        urls = ["http://$NODE_IP:9103/metrics"]
+  metric_collection_settings: |-
+    [metric_collection_settings.collect_kube_system_pv_metrics]
+        enabled = true
+  alertable-metrics-configuration-settings: |-
+    [alertable_metrics_configuration_settings.container_resource_utilization_thresholds]
+        container_cpu_threshold_percentage = 90.0
+        container_memory_rss_threshold_percentage = 90.0
+        container_memory_working_set_threshold_percentage = 90.0
+    [alertable_metrics_configuration_settings.pv_utilization_thresholds]
+        pv_usage_threshold_percentage = 75.0
+  integrations: |-
+    [integrations.azure_network_policy_manager]
+        collect_basic_metrics = true
+        collect_advanced_metrics = false