Installation

On this page you will install the Stackable Airflow Operator and its dependencies - Postgresql and Redis - as well as the commons and secret operator which are required by all Stackable Operators.

Dependencies

Postgresql and Redis are required by Airflow: Postgresql to store metadata about DAG runs, and Redis to schedule and/or queue DAG jobs. They are components that may well already be available for customers, in which case we treat them here as pre-requisites for an airflow cluster and hence as part of the installation process. These components will be installed using Helm. Note that specific versions are declared:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install --wait airflow-postgresql bitnami/postgresql --version 12.1.5 \
    --set auth.username=airflow \
    --set auth.password=airflow \
    --set auth.database=airflow
helm install --wait airflow-redis bitnami/redis \
    --set auth.password=redis \
    --version 17.3.7 \
    --set replica.replicaCount=1

Stackable Operators

There are 2 ways to run Stackable Operators

  1. Using stackablectl

  2. Using Helm

stackablectl

stackablectl is the command line tool to interact with Stackable operators and our recommended way to install Operators. Follow the installation steps for your platform.

After you have installed stackablectl run the following command to install all Operators necessary for Airflow:

stackablectl operator install \
  commons=23.4.0 \
  secret=23.4.0 \
  airflow=23.4.0

The tool will show

[INFO ] Installing commons operator
[INFO ] Installing secret operator
[INFO ] Installing airflow operator
Consult the Quickstart to learn more about how to use stackablectl. For example, you can use the --cluster kind flag to create a Kubernetes cluster with kind.

Helm

You can also use Helm to install the Operators. Add the Stackable Helm repository:

helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/

Then install the Stackable Operators:

helm install --wait commons-operator stackable-stable/commons-operator --version 23.4.0
helm install --wait secret-operator stackable-stable/secret-operator --version 23.4.0
helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.4.0

Helm will deploy the Operators in a Kubernetes Deployment and apply the CRDs for the Airflow cluster (as well as the CRDs for the required operators). You are now ready to deploy Apache Airflow in Kubernetes.

What’s next

Set up an Airflow cluster and its dependencies and verify that it works by inspecting and running an example DAG.