sg-aks-workshop/validate-scenarios

Validate Scenarios

Now that we have implemented everything, let's go back and revisit our requirements and make sure they have been met.

Requirements

This is a quick recap of the requirements:

1. Leverage Existing Identity Mgmt Solution

2. Implement Security Least Privilege Principle

3. Log Everything for Audit Reporting purposes

4. Ensure Security Controls are being met (No Drifting)

5. Monitoring and Alerting Events

   • Alert when SSH into Container
   • Alert AKS Cluster has IP whitelisting set

6. Integrate with Existing SIEM

7. Deploy into Existing VNET with Ingress and Egress Restrictions

8. Resources can only be created in specific regions due to data sovereignty

9. Container Registry Whitelisting

10. Ability to Chargeback to Line of Business

11. Secrets Mgmt

12. Container Image Mgmt

13. Restrict Creation of Public IPs

14. Implement & Deploy Image Processing Application

15. Easily rollout new versions of Application

Requirements to Technology Matrix

Requirement	Technology Used
1. Leverage Existing Identity Mgmt Solution	Azure AD (AAD)
2. Implement Security Least Privilege Principle	AAD + ARM RBAC + K8s RBAC per Namespace
3. Log Everything for Audit Reporting purposes	Azure Monitor for Containers, Azure Storage
4. Ensure Security Controls are being met (No Drifting)	Flux, Git Repo
5a. Alert when SSH into Container	Falco + Azure Monitor for Containers
5b. Alert AKS Cluster has IP whitelisting set	Azure Security Center + Azure Monitor for Containers
6. Integrate with Existing SIEM	Azure Security Center or Azure Sentinel
7. Deploy into Existing VNET with Ingress and Egress Restrictions	Azure VNET, Azure Firewall
8. Resources can only be created in specific regions due to data sovereignty	Azure Policy
9. Container Registry Whitelisting	Open Policy Agent + OSS Version of Gatekeeper
10. Ability to Chargeback to Line of Business	KubeCost
11. Secrets Mgmt	Azure AD Pod Identity + Azure Key Vault
12. Container Image Mgmt	Anchore
13. Restrict Creation of Public IPs	Azure Policy
14. Implement & Deploy Image Processing Application	Azure Monitor for Containers, Application Insights
15. Easily rollout new versions of Application	Kubernetes Built-in

The rest of these sections shows how we can validate the requirements above.

1. Validate - Leverage Existing Identity Mgmt Solution

• Log into AKS Cluster with Azure AD Credentials

Pull down the cluster configuration file and try to execute a command, you should get prompted to authenticate to Azure AD.

# Grab K8s Config
az aks get-credentials -g $RG -n $PREFIX-aks

# Execute a Command
kubectl get nodes

2. Validate - Implement Security Least Privilege Principle

• Validate Cluster Reader cannot Create Resources

Authenticate to AKS using the cluster reader credentials and then try and execute a create command.

# Try to create a pod in the default namespace
kubectl run --generator=run-pod/v1 -it --rm centos2 --image=centos

3. Validate - Log Everything for Audit Reporting purposes

• Run log analytics query against AzureActivity table

Go into Azure Monitor for Containers -> Logs and get a summary of activity logs by Resource Provider.

AzureActivity
| summarize AggregatedValue = count() by ResourceProvider

4. Validate - Ensure Security Controls are being met (No Drifting)

• Look at Flux Logs for GitOps style Drift Configuration. As you can see from the screenshot it detects whether something has changed or not.

• Delete a resource and see it get re-created. Delete the Production NS and then wait upwards of 5 mins to see that Flux re-creates the resource.

kubectl delete ns production
kubectl logs -l name=flux -n flux -f

5. Validate - Monitoring and Alerting Events

• View Azure Security Center Compliance Dashboard

• Exec into a Container and get an Alert

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: centos
spec:
  containers:
  - name: centoss
    image: centos
    ports:
    - containerPort: 80
    command:
    - sleep
    - "3600"
EOF
# Check that Pod is Created and then Exec In
kubectl get po -o wide
kubectl exec -it centos -- /bin/bash
curl www.ubuntu.com
exit

Wait a few minutes for the log data to get processed and then go into the Azure Monitor Logs Workspace and execute the following query to see the log entry.

let startTimestamp = ago(1d);
let ContainerIDs = KubePodInventory
| where TimeGenerated > startTimestamp
| where ClusterId =~ "/subscriptions/${SUBID}/resourceGroups/${RG}/providers/Microsoft.ContainerService/managedClusters/${CLUSTER_NAME}"
| where Name contains "sysdig-falco"
| distinct ContainerID;
ContainerLog
| where ContainerID in (ContainerIDs)
| where LogEntry contains "Notice A shell was spawned"
| project LogEntrySource, LogEntry, TimeGenerated, Computer, Image, Name, ContainerID
| order by TimeGenerated desc
| limit 200

• AKS Cluster missing IP Whitelisting

Similiar to the above we will run the following Azure Monitor queries against the Azure Security Center data to see which Policies are being met from a security perspective.

AzureActivity
| where CategoryValue == 'Policy' and Level != 'Informational'
| where ResourceProvider == "Microsoft.ContainerService"
| extend p=todynamic(Properties)
| extend policies=todynamic(tostring(p.policies))
| mvexpand policy = policies
| summarize resource_count=count() by tostring(policy.policyDefinitionName),tostring(policy.policyDefinitionReferenceId)

// Check for Authorized IP Policy
let policyDefId = '0e246bcf-5f6f-4f87-bc6f-775d4712c7ea';
AzureActivity
| where CategoryValue == 'Policy' and Level != 'Informational'
| where ResourceProvider == "Microsoft.ContainerService"
| extend p=todynamic(Properties)
| extend policies=todynamic(tostring(p.policies))
| mvexpand policy = policies
| where policy.policyDefinitionName in (policyDefId)
| distinct ResourceId

6. Validate - Integrate with Existing SIEM

• View Azure Security Center Security Solutions

Go to the Azure Portal, click on Security Center, then on the Security Solutions blade.

7. Validate - Deploy into Existing VNET with Ingress and Egress Restrictions

• Validate Traffic In & Out of Cluster (North/South)

kubectl exec -it centos -- /bin/bash
curl http://superman.com

• Validate Traffic Restriction between Namespaces (East/West)

kubectl exec -it centos -- /bin/bash
curl http://imageclassifierweb.dev.svc.cluster.local
exit

8. Validate - Resources can only be created in specific regions due to data sovereignty

• Try to create resource outside of allowed region locations

az storage account create --sku Standard_LRS --kind StorageV2 --location westus -g notinallowedregions-rg -n niarsa

9. Validate - Container Registry Whitelisting

• Try to pull from a non-whitelisted Container Registry

# Test out Allowed Registry Policy Against production Namespace
kubectl run --generator=run-pod/v1 -it --rm centosprod --image=centos -n production

10. Validate - Ability to Chargeback to Line of Business

• View Chargeback Dashboard

# Do a port-forward to see Kubecost Dashboard
kubectl port-forward deployment/kubecost-cost-analyzer -n kubecost 9090
# Open in Browser
open "http://localhost:9090"

11. Validate - Secrets Mgmt

• Check that there is no sensitive data stored in the container image or in a configuration file in plain text.

• The first place to start is looking at the application manifest file and we can see from looking at it that it is not storing credentials, it simply points to an Azure Key Vault Name.

      containers:
      - name: imageclassifierweb
        image: kevingbb/imageclassifierweb:v3
        imagePullPolicy: Always
        env:
        - name: KeyVault__Vault
          valueFrom:
            secretKeyRef:
              name: image-akv-secret
              key: KeyVault__Vault

• The next step is to look inside the container to see if there is any configuration information.

# Exec into Web Container for Example
kubectl get pods -n dev

# Grab Pod Name (from above) and use to exec into Pod
kubectl exec -it imageclassifierweb-d7fdcd8bf-dphbm -n dev -c imageclassifierweb /bin/sh
# OR
kubectl exec -it $(k get po -l=app=imageclassifierweb -n dev -o jsonpath="{.items[0].metadata.name}") -n dev -c imageclassifierweb /bin/sh

# Once inside of Pod Look Around
ls -al

# Exit Out
exit

• Check for Sensitive values in Key Vault

# Look at Secret Value in Key Vault
az keyvault secret show --name "AppSecret" --vault-name "${PREFIX}akv"
az keyvault secret show --name "AppInsightsInstrumentationKey" --vault-name "${PREFIX}akv"

• So how does the secret get into the application then? Great question, it relies on Azure AD Pod Identity, or what we like to call Managed Pod Identity. Click here for more details.

12. Validate - Container Image Mgmt

• Check that container images in ACR are passing image scanning policy check.

# Get ACR Variable Name
echo ACR_NAME=${PREFIX}acr.azurecr.io
# Exec into Container
kubectl exec -it $(kubectl get po -l app=anchore-anchore-engine -l component=analyzer -n anchore -o jsonpath='{.items[0].metadata.name}') -n anchore bash
# Setup Variables
ANCHORE_CLI_USER=admin
ANCHORE_CLI_PASS=foobar
ANCHORE_CLI_URL=http://anchore-anchore-engine-api.anchore.svc.cluster.local:8228/v1/

# Wait for all images to be "analyzed"
anchore-cli image list

# Check for Active Subscriptions
anchore-cli subscription list

# Get Policies
anchore-cli policy list

# Evaluate against Policy (Pass or Fail) Using ACR_NAME From Above
ACR_NAME=...
anchore-cli evaluate check $ACR_NAME/imageclassifierweb:v1
anchore-cli evaluate check $ACR_NAME/imageclassifierworker:v1

# Exit out of Pod
exit

13. Validate - Restrict Creation of Public IPs

This is similar to #8 in that Azure Policy can be used to restrict the creation of Public IPs exception in certain Resource Groups. This was not implemented in the workshop so that Public IPs were possible to be able to test and see endpoints.

14. Validate - Implement & Deploy Image Processing Application

• Does the Application Run, Visit Public IP

15. Validate - Easily roll out new versions of Application

• Ensure the app successfully rolls out a new version of the application and does not cause any downtime.

# Check Deployment History
kubectl rollout history deploy imageclassifierweb -n dev

# Make a change to the Manifest File & Apply a new version of the application
# Hint: Change registry to ACR
kubectl apply -f appv3msi.yaml

# Watch the Rollout
kubectl rollout status deploy imageclassifierweb -n dev

# Check Deployment History Again
kubectl rollout history deploy imageclassifierweb -n dev

• Test the app to make sure it continues to work.

Next Steps

Thought Leadership

Key Links

• Collect and Analyze Azure Activity Logs
• How to setup Azure Monitor for Container Alerts
• How to query logs from Azure Monitor for Containers
• How to update Azure Monitor for Containers to Enable Metrics