RBAC

The Spark-Kubernetes RBAC documentation describes what is needed for spark-submit jobs to run successfully: minimally a role/cluster-role to allow the driver pod to create and manage executor pods.

However, to add security, each spark-submit job launched by the spark-k8s operator will be assigned its own service account.

When the spark-k8s operator is installed via Helm, a cluster role named spark-k8s-clusterrole is created with pre-defined permissions.

When a new Spark application is submitted, the operator creates a new service account with the same name as the application and binds this account to the cluster role spark-k8s-clusterrole created by Helm.