frameworkcontroller/example/run

Run FrameworkController

We provide various approaches to run FrameworkController:

• Run By Kubernetes StatefulSet
• Run By Docker Container
• Run By OS Process

Notes:

• For a single k8s cluster, one instance of FrameworkController orchestrates all Frameworks in all namespaces.
• For a single k8s cluster, ensure at most one instance of FrameworkController is running at any point in time.
• For the full FrameworkController configuration, see Config Usage and Config Example.

Run By Kubernetes StatefulSet

• This approach is better for production, since StatefulSet by itself provides self-healing and can ensure at most one instance of FrameworkController is running at any point in time.
• Using official image to demonstrate this example.

Prerequisite

If the k8s cluster enforces Authorization, you need to first create a ServiceAccount with granted permission for FrameworkController. For example, if the cluster enforces RBAC:

kubectl create serviceaccount frameworkcontroller --namespace default
kubectl create clusterrolebinding frameworkcontroller \
  --clusterrole=cluster-admin \
  --user=system:serviceaccount:default:frameworkcontroller

Run

Run FrameworkController with above ServiceAccount and the k8s inClusterConfig:

Run with default config

kubectl create -f frameworkcontroller-with-default-config.yaml

frameworkcontroller-with-default-config.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: frameworkcontroller
  namespace: default
spec:
  serviceName: frameworkcontroller
  selector:
    matchLabels:
      app: frameworkcontroller
  replicas: 1
  template:
    metadata:
      labels:
        app: frameworkcontroller
    spec:
      # Using the ServiceAccount with granted permission
      # if the k8s cluster enforces authorization.
      serviceAccountName: frameworkcontroller
      containers:
      - name: frameworkcontroller
        image: frameworkcontroller/frameworkcontroller
        # Using k8s inClusterConfig, so usually, no need to specify
        # KUBE_APISERVER_ADDRESS or KUBECONFIG
        #env:
        #- name: KUBE_APISERVER_ADDRESS
        #  value: {http[s]://host:port}
        #- name: KUBECONFIG
        #  value: {Pod Local KubeConfig File Path}

Run with customized config

kubectl create -f frameworkcontroller-customized-config.yaml
kubectl create -f frameworkcontroller-with-customized-config.yaml

frameworkcontroller-customized-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: frameworkcontroller-config
  namespace: default
data:
  frameworkcontroller.yaml: |
    kubeClientQps: 200
    kubeClientBurst: 300
    workerNumber: 500
    largeFrameworkCompression: true
    frameworkCompletedRetainSec: 2592000
    #podFailureSpec:
    #- code: 221
    #  phrase: ContainerTensorflowOOMKilled
    #  type:
    #    attributes: [Permanent]
    #  podPatterns:
    #  - containers:
    #    - messageRegex: '(?msi)tensorflow.*ResourceExhaustedError.*OOM.*'
    #      codeRange: {min: 1}
    #      nameRegex: '(?ms).*'
    #- {More customized podFailureSpec, better to also include these in the default config}    

frameworkcontroller-with-customized-config.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: frameworkcontroller
  namespace: default
spec:
  serviceName: frameworkcontroller
  selector:
    matchLabels:
      app: frameworkcontroller
  replicas: 1
  template:
    metadata:
      labels:
        app: frameworkcontroller
    spec:
      # Using the ServiceAccount with granted permission
      # if the k8s cluster enforces authorization.
      serviceAccountName: frameworkcontroller
      containers:
      - name: frameworkcontroller
        image: frameworkcontroller/frameworkcontroller
        # Using k8s inClusterConfig, so usually, no need to specify
        # KUBE_APISERVER_ADDRESS or KUBECONFIG
        #env:
        #- name: KUBE_APISERVER_ADDRESS
        #  value: {http[s]://host:port}
        #- name: KUBECONFIG
        #  value: {Pod Local KubeConfig File Path}
        command: [
          "bash", "-c",
          "cp /frameworkcontroller-config/frameworkcontroller.yaml . &&
          ./start.sh"]
        volumeMounts:
        - name: frameworkcontroller-config
          mountPath: /frameworkcontroller-config
      volumes:
      - name: frameworkcontroller-config
        configMap:
          name: frameworkcontroller-config

Run By Docker Container

• This approach may be better for development sometimes.
• Using official image to demonstrate this example.

Run

If you have an insecure ApiServer address (can be got from Insecure ApiServer or kubectl proxy) which does not enforce authentication, you only need to provide the address:

docker run -e KUBE_APISERVER_ADDRESS={http[s]://host:port} \
  frameworkcontroller/frameworkcontroller

Otherwise, you need to provide your KubeConfig File which inlines or refers the ApiServer Credential Files with granted permission:

docker run -e KUBECONFIG=/mnt/.kube/config \
  -v {Host Local KubeConfig File Path}:/mnt/.kube/config \
  -v {Host Local ApiServer Credential File Path}:{Container Local ApiServer Credential File Path} \
  frameworkcontroller/frameworkcontroller

For example, if the k8s cluster is created by Minikube:

docker run -e KUBECONFIG=/mnt/.kube/config \
  -v ${HOME}/.kube/config:/mnt/.kube/config \
  -v ${HOME}/.minikube:${HOME}/.minikube \
  frameworkcontroller/frameworkcontroller

Run By OS Process

• This approach may be better for development sometimes.
• Using local built binary distribution to demonstrate this example.

Prerequisite

Ensure you have installed Golang 1.12.6 or above and the ${GOPATH} is valid.

Then build the FrameworkController binary distribution:

export PROJECT_DIR=${GOPATH}/src/github.com/microsoft/frameworkcontroller
rm -rf ${PROJECT_DIR}
mkdir -p ${PROJECT_DIR}
git clone https://github.com/microsoft/frameworkcontroller.git ${PROJECT_DIR}
cd ${PROJECT_DIR}
./build/frameworkcontroller/go-build.sh

Run

If you have an insecure ApiServer address (can be got from Insecure ApiServer or kubectl proxy) which does not enforce authentication, you only need to provide the address:

KUBE_APISERVER_ADDRESS={http[s]://host:port} \
  ./dist/frameworkcontroller/start.sh

Otherwise, you need to provide your KubeConfig File which inlines or refers the ApiServer Credential Files with granted permission:

KUBECONFIG={Process Local KubeConfig File Path} \
  ./dist/frameworkcontroller/start.sh

For example:

KUBECONFIG=${HOME}/.kube/config \
  ./dist/frameworkcontroller/start.sh

And in above example, ${HOME}/.kube/config is the default value of KUBECONFIG, so you can skip it:

./dist/frameworkcontroller/start.sh

Next

1. Submit Framework