Skip to main content

Use custom service account with DDCR

This topic describes how to:

  1. Install the Harness Delegate with limited permissions.
  2. Use the infrastructure to run chaos experiments in single or multiple namespaces in a Kubernetes cluster.

Before you begin, review the following:​

Why service accounts matter​

A service account is required in the Delegate YAML when Delegate is installed in the target cluster to execute chaos experiments, because Delegate has to assume a role to execute chaos experiments.

In case the Delegate is deployed in a centralized infrastructure (and connected to the target cluster), service account is not required in the Delegate YAML because the Kubernetes connectors already have the role permissions, and Delegate only orchestrates the tasks.

Dedicated Delegate installed on Target Cluster​

By default, Delegate uses the cluster admin role. To limit the permissions for the Delegate, follow the steps below.

  1. Create a dedicated namespace for Harness delegate during installation. For example harness-delegate-ng.

  2. Remove the cluster role binding from the Delegate manifest, as shown in the diagram.

  3. Create a new service account for the Delegate (to which you can assign a role further) in the dedicated namespace where the Delegate is installed. Here, chaos-delegate refers to the name of the service account in the Delegate.

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: chaos-delegate
      namespace: harness-delegate-ng

  4. Attach the service account you created earlier to the Delegate YAML, as shown in the diagram.

  5. Apply the below RBAC to configure the permissions to create chaos runners (that is, transient pods).

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      namespace: harness-delegate-ng
      name: chaos-runner-role
    rules:
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - ""
      resources:
      - pods
      - pods/log
      - pods/exec
      - secrets
      - services
      - configmaps
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - batch
      resources:
      - jobs
      - cronjobs
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: chaos-runner-rolebinding
      namespace: harness-delegate-ng
    subjects:
    - kind: ServiceAccount
      name: chaos-delegate
      namespace: harness-delegate-ng
    roleRef:
      kind: Role
      name: chaos-runner-role
      apiGroup: rbac.authorization.k8s.io

  6. Create a cluster role that will be used later to onboard application namespaces.

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: chaos-clusterrole
    rules:
    # Discovery permissions
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - watch
      - list
      - get
    - apiGroups:
      - ""
      resources:
      - pods
      - replicationcontrollers
      - services
      - statefulsets
      - nodes
      - namespaces     #(nodes and namespace permissions are required to autocreate network experiments)
      verbs:
      - watch
      - list
      - get
    - apiGroups:
      - batch
      resources:
      - jobs
      - cronjobs
      verbs:
      - watch
      - list
      - get
    # Chaos permissions
    - apiGroups:
      - ""
      resources:
      - pods
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - networking.k8s.io
      resources:
      - networkpolicies
      verbs:
      - create
      - delete
      - get
      - list
    - apiGroups:    #(Prerequisite: Metrics server should be installed on the cluster. This is required if your are running a custom script in command (CMD) source probe to get CPU and memory metrics for pods and nodes.)
      - metrics.k8s.io
      resources:
      - pods
      - nodes
      verbs:
      - get
      - list
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - list
      - get
      - update
    - apiGroups:
      - ""
      resources:
      - replicationcontrollers
      - services
      verbs:
      - get
      - list
    - apiGroups:
      - apps.openshift.io
      resources:
      - deploymentconfigs
      verbs:
      - list
      - get
    - apiGroups:
      - argoproj.io
      resources:
      - rollouts
      verbs:
      - list
      - get

  7. If you wish to provide access to all namespaces, create a cluster role binding.

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: chaos-rolebinding
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: chaos-clusterrole
    subjects:
    - kind: ServiceAccount
      name: chaos-delegate
      namespace: harness-delegate-ng

  • Now, you can onboard a namespace by creating a role binding in the application namespace (For example, Onboarding app1, app2 and so on)

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: chaos-rolebinding
      namespace: app1 #for app2, provide namespace as app2
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: chaos-clusterrole
    subjects:
    - kind: ServiceAccount
      name: chaos-delegate
      namespace: harness-delegate-ng

  1. Create a Kubernetes connector using Delegate permissions.

    connector delegate setup

  2. Finally create the Kubernetes infrastructure using the Kubernetes connectors created in the step 8.

  3. Edit the infrastructure you created in step 9 to provide the details of dedicated namespace that was created earlier. This is the namespaace where the chaos runner will be launched along with the Service Account to ensure that experiments run with relevant permissions.

Permissions to manage Delegate resources​

While using Harness Delegate, transient pods are created. Harness recommends you provide dedicated namespace for the transient resources. Delegate manages the resources, that require specific permissions in the dedicated namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: chaos-delegate
  namespace: harness-delegate-ng
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: harness-delegate-ng
  name: chaos-delegate
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  - replicasets
  - daemonsets
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
  - deletecollection
- apiGroups:
  - ""
  resources:
  - pods
  - secrets
  - services
  - configmaps
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
  - deletecollection
- apiGroups:
  - batch
  resources:
  - jobs
  - cronjobs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
  - deletecollection
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: chaos-delegate
  namespace: harness-delegate-ng
subjects:
- kind: ServiceAccount
  name: chaos-delegate
  namespace: harness-delegate-ng
roleRef:
  kind: Role
  name: chaos-delegate
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: chaos-delegate
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - events
  verbs:
  - watch
  - list
  - get
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - patch
  resourceNames:
  - harness-delegate-ng
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: chaos-delegate
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: chaos-delegate
subjects:
- kind: ServiceAccount
  name: chaos-delegate
  namespace: harness-delegate-ng

Enable Chaos in Namespaces​

Case 1: One Infrastructure, Multiple Namespaces​

One of the use cases of enabling chaos in selected namespaces is to use one infrastructure to execute chaos experiments in these selected (multiple) namespaces.

Execute the below one-time steps.

  1. Create a Kubernetes cluster connector: A connector describes how the Delegate communicates with the target cluster.

Two cases arise when a Delegate communicates with target cluster:

a. When the target application and the Delegate are running in the same cluster.

b. When the target application and the Delegate are running on different clusters.

Target Application and Delegate Running in Same Cluster​

After creating a Kubernetes cluster, follow the steps below:

  1. Add permissions to manage the discovery and transient chaos pods in the Delegate namespace (harness-delegate-ng in the example below). Harness recommends you keep the Delegate and chaos infrastructure in the Delegate namespace.
  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    namespace: harness-delegate-ng
    name: chaos-delegate
  rules:
  - apiGroups:
    - "*"
    resources:
    - "*"
    verbs:
    - "*"
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: chaos-delegate
    namespace: harness-delegate-ng
  subjects:
  - kind: ServiceAccount
    name: chaos-delegate
    namespace: harness-delegate-ng
  roleRef:
    kind: Role
    name: chaos-delegate
    apiGroup: rbac.authorization.k8s.io
  1. Create a Kubernetes cluster connector using the Delegate permissions you added in the previous step.

k8s connector

Target Application and Delegate Running on Different Clusters​

The diagram below describes how the Harness environment and your (user) environment communicate with the help of Harness Delegate to execute chaos experiments.

  1. Create a service account in the target cluster with the permissions mentioned below. These permissions allow the Harness chaos transient pods to be present in a dedicated namespace. Harness recommends you keep the Delegate and chaos infrastructure in the Delegate namespace. Also ensure the following are fulfilled:

  • Delegate should be installed in your centralised infrastructure.

  • Network connectivity between centralised infrastructure and target cluster infrastructure (where chaos runs)

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: chaos-sa
      namespace: harness-delegate-chaos # our recommendation is to create a dedicated namespace in the target cluster for delegate
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: chaos-sa-secret
      namespace: harness-delegate-chaos
      annotations:
        kubernetes.io/service-account.name: chaos-sa
    type: kubernetes.io/service-account-token
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      namespace: harness-delegate-chaos
      name: chaosrunner-pod-role
    rules:
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - ""
      resources:
      - pods
      - pods/log
      - pods/exec
      - secrets
      - services
      - configmaps
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - batch
      resources:
      - jobs
      - cronjobs
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: chaosrunner-pod-rolebinding
      namespace: harness-delegate-chaos
    subjects:
    - kind: ServiceAccount
      name: chaos-sa
      namespace: harness-delegate-chaos
    roleRef:
      kind: Role
      name: chaosrunner-pod-role
      apiGroup: rbac.authorization.k8s.io

  1. Create a Kubernetes cluster connector using service account based authentication. Obtain the master URL by executing the command kubectl cluster-info and get the service account token and secret from the Kubernetes secret.

    k8s connector

  2. Create a cluster role and provide cluster-wide access or cluster role binding for selected namespaces using role binding, depending on your usage. You can use this configuration to onboard application namespaces. To discover the resources and run chaos, use the permissions (described below) in each namespace.

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: chaos-clusterrole
    rules:
    # Discovery permissions
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - watch
      - list
      - get
    - apiGroups:
      - ""
      resources:
      - pods
      - replicationcontrollers
      - services
      - statefulsets
      - nodes
      - namespaces     #(nodes and namespace permissions are required to autocreate network experiments)
      verbs:
      - watch
      - list
      - get
    - apiGroups:
      - batch
      resources:
      - jobs
      - cronjobs
      verbs:
      - watch
      - list
      - get
    # Chaos permissions
    - apiGroups:
      - ""
      resources:
      - pods
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - networking.k8s.io
      resources:
      - networkpolicies
      verbs:
      - create
      - delete
      - get
      - list
    - apiGroups:    #(Prerequisite: Metrics server should be installed on the cluster. This is required if your are running a custom script in command (CMD) source probe to get CPU and memory metrics for pods and nodes.)
      - metrics.k8s.io
      resources:
      - pods
      - nodes
      verbs:
      - get
      - list
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - list
      - get
      - update
    - apiGroups:
      - ""
      resources:
      - replicationcontrollers
      - services
      verbs:
      - get
      - list
    - apiGroups:
      - apps.openshift.io
      resources:
      - deploymentconfigs
      verbs:
      - list
      - get
    - apiGroups:
      - argoproj.io
      resources:
      - rollouts
      verbs:
      - list
      - get

  3. You can onboard a namespace by creating a role binding in the application namespace (Onboarding app1 namespace).

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: agentless-model-rolebinding-app1
      namespace: app1
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: agentless-model-clusterrole
    subjects:
    - kind: ServiceAccount
      name: targetclustersa
      namespace: harness-delegate-n
note

  • To onboard a new application namespace, create another role binding in the application namespace (Onboarding app2 namespace, replace the namepsace field in the above YAML spec with app2.)

  • To give access to all namespaces, use the following YAML configuration:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: CluesterRoleBinding
    metadata:
      name: agentless-model-rolebinding
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: agentless-model-rolebinding
    subjects:
    - kind: ServiceAccount
      name: chaos-delegate
      namespace: harness-delegate-ng

  • For the above setting to work, ensure you update the namespace and the service account in the Harness portal service discovery agent setting and chaos infrastructure settings.

  1. Create Harness infrastructure definition using the same Kubernetes cluster connector that was created in Step 2.

    infra setup 1

    infra setup 2

  2. Edit the infrastructure you created in step 5 to provide the details of the dedicated namespace that was created. This is the namespace where the chaos runner is launched along with the Service Account to ensure that experiments are executed with relevant permissions.

    service account used

Case 2: One Infrastructure, One Namespace​

To use one infrastructure to execute multiple experiments in one namespace, follow the steps below.

  1. Create a service account and role to execute chaos experiments, and ensure they are in the application namespace.

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: chaos-delegate
      namespace: app3
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: chaos-delegate
      namespace: app3
      annotations:
        kubernetes.io/service-account.name: chaos-delegate
    type: kubernetes.io/service-account-token
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: chaos-delegate
      namespace: app3
    rules:
    # Discovery management permissions
    - apiGroups:
      - ""
      resources:
      - pods
      verbs:
      - create
      - delete
      - get
      - list
    - apiGroups:
      - ""
      resources:
      - pods/log
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - apps
      resources:
      - deployments
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
    # Discovery process permissions
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - watch
      - list
      - get
    - apiGroups:
      - ""
      resources:
      - pods
      - replicationcontrollers
      - services
      - statefulsets
      verbs:
      - watch
      - list
      - get
    - apiGroups:
      - batch
      resources:
      - jobs
      - cronjobs
      verbs:
      - watch
      - list
      - get
    # Chaos management permissions
    - apiGroups:
      - ""
      resources:
      - pods
      - secrets
      - configmaps
      - services
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - batch
      resources:
      - jobs
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - ""
      resources:
      - pods/log
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - ""
      resources:
      - pods/exec
      verbs:
      - get
      - list
      - create
    - apiGroups:
      - apps
      resources:
      - deployments
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - deletecollection
    # Chaos execution permissions
    - apiGroups:
      - ""
      resources:
      - pods
      verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
      - deletecollection
    - apiGroups:
      - networking.k8s.io
      resources:
      - networkpolicies
      verbs:
      - create
      - delete
      - get
      - list
    - apiGroups:
      - apps
      resources:
      - deployments
      - replicasets
      - daemonsets
      - statefulsets
      verbs:
      - list
      - get
      - update
    - apiGroups:
      - ""
      resources:
      - replicationcontrollers
      - services
      verbs:
      - get
      - list
    - apiGroups:
      - apps.openshift.io
      resources:
      - deploymentconfigs
      verbs:
      - list
      - get
    - apiGroups:
      - argoproj.io
      resources:
      - rollouts
      verbs:
      - list
      - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: chaos-delegate
      namespace: app3
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: chaos-delegate
    subjects:
    - kind: ServiceAccount
      name: chaos-delegate
      namespace: app3

  2. Create a Kubernetes cluster connector with the service account credentials. Obtain the master URL by executing the command kubectl cluster-info and get the service account token and secret from the Kubernetes secret.

    same cluster

  3. Create Harness infrastructure definition using the same Kubernetes cluster connector.

  4. Ensure to use the application namespace as chaos infrastructure namespace and update the service account name accordingly.