Roles and role groups

Software - especially the distributed kind - often consists of multiple different processes that work together to achieve their goal. Think of a client communicating with a server. Stackable calls these roles. The roles or processes are often replicated multiple times. These replicas can further be subdivided into role groups. Roles and role groups are defined in the resource manifest.

Motivation

The use of roles and role groups is required to control how processes are configured and where they run.

Configuration

Different processes have different tasks that they need to fulfill, which in turn have different configuration settings that only apply to that task. For example coordination, storage, logging and processing tasks require different amounts of threads, memory and storage capacity. These different settings can be put in the per role configuration spec.

Scheduling

While configuration is concerned with modifying how the process works on the inside, scheduling is concerned with the context where it should run inside the cluster. A processing task should be scheduled on a node with faster CPUs or nodes with GPUs attached. A caching process is best scheduled on a node with SSD storage, for example.

This is done by using label selectors. Nodes can be labeled and then processes can be assigned to nodes based on these labels.

Role groups

Role groups are motivated by the need to subdivide the replicas of a process into groups as well, either to configure or to schedule multiple groups differently. For example in a cluster spanning multiple physical locations, you may want to make sure half of your processes run in one location and half in the other.

How it works

A distributed software consists of one or more roles. Every role must be configured with at least one role group. Configuration can be specified at role level or at role group level, where the role group level overrides the role level. Label selectors for scheduling and the number of replicas are specified on the role group, not role. The role group is the lowest level, so it makes sense to have to specify this there.

Example

HDFS uses three distinct processes that work together to fulfill its duty: NameNodes, DataNodes and JournalNodes. With roles you can specify different configuration settings for each of these, as well as schedule them on specific machines.

apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
  name: simple-hdfs
spec:
  journalNodes:
    roleGroups:
      default:
        replicas: 3  (1)
  nameNodes:
    roleGroups:
      default:
        replicas: 3
  dataNodes:
    config:
      resources:
        storage:
          data:
            capacity: 1Gi  (2)
    roleGroups:
      default:
        replicas: 2
      hiCapacity:  (3)
        config:
          resources:
            storage:
              data: 2Gi
        replicas: 1
  ...

1	The JournalNode role with only a single default role group. For the role group 3 replicas are specified, specifying a replica count is optional, the default is 1.
2	A common config setting for the DataNodes role. This configuration setting applies only to pods for this role.
3	The DataNode role has two role groups, the default group and the hiCapacity group. In it the config setting for the group is overridden with a higher value of 2Gi. This role group has only one replica.