ADR022: Spark history server

Monitoring Spark applications usually involves two types of components: a history server and a metrics collection server. In this document we will concern ourselves with the history server. Using metrics for application monitoring requires a completely different set of protocols and technologies and is outside the scope of this document.

The Spark history server is part of the Apache Spark distribution and it provides a web interface where application developers can visualize events that took place during the lifetime of a job as well as the outcome of various job phases.

In this document, we propose a way to add the Spark history server to the Kubernetes resources that are managed by the Spark-on-Kubernetes operator. We also describe how Spark application definitions can leverage the history server and how the resource definitions make use of standard Stackable descriptors for S3 storage.

Even though using a history server is optional when running Spark applications on the Stackable platform, it is highly recommended to make use of it in production environments.

A history server is a separate entity, completely decoupled from any `SparkApplication`s. The Spark-on-k8s operator makes use of it only if available and if the Spark application developer has requested it. See the chapter below, on how `SparkApplication`s can request logging events to an existing history server.

The most important part of the history server definition is the location of the event files. This is specified in the logFileDirectory entry of the resource specification as shown below.

Here the event files are stored in the S3 bucket spark-eventlogs with the eventlogs/ prefix. The complete definition of the specified S3 connection is resolved by the operator from the eventlogs-s3-connection resource (not shown here).

All Spark history properties can be defined in the properties section. The operator will make them available to the server at runtime. The operator makes no effort to validate these properties so extreme care must be taken making use of this section. In particular, properties concerning the event file storage can conflict with the definition provided in the logFileDirectory section and it’s not recommended to set them here at all.

Here is a complete example of a history server definition:

Spark applications don’t have to use a history server but it’s highly recommended to do so in production environments. The section eventLogs of a SparkApplication instructs the operator to configure the application such that it can be monitored using the provided (and existing) history server.

The example below shows the relevant configuration snippet of an application definition:

In the example above, the application will log its events to the bucket specifed in logFileDirectory. In addition, the application processes data from a S3 bucket configured within the s3connection section of the specification.

The operator will read the s3Connection and set up the fs.s3a.aws.credentials.provider and co (endpoint, accesskey, secretkey, tls, path-style access - basically all attributes of S3Connection) settings. Afterwards if logFileDirectory attribute is set: Set fs.s3a.bucket.<logFileDirectory-bucket-name(here spark-eventlogs)>.aws.credentials.provider and co to overwrite endpoint and credentials for the logging bucket. Set spark.eventLog.enabled property to true and will construct the spark.eventLog.dir from s3a://<logFileDirectory-bucket-name(here spark-eventlogs)>/<logFileDirectory-prefix(here eventlogs/)>.