airflow kubernetes executor pod template

You can also create custom pod_template_file on a per-task basis so that you can recycle the same base values between multiple tasks. apiVersion : v1 kind : Pod metadata : name : placeholder-name spec : containers : - env : - name : AIRFLOW_CORE_EXECUTOR value : LocalExecutor # Hard Coded Airflow Envs - name : AIRFLOW_CORE_FERNET_KEY valueFrom : secretKeyRef : name : RELEASE-NAME-fernet-key key : fernet-key - name : AIRFLOW_DATABASE_SQL_ALCHEMY_CONN valueFrom : secretKeyRef : name : RELEASE-NAME-airflow-metadata key : connection - name : AIRFLOW_CONN_AIRFLOW_DB valueFrom : secretKeyRef : name : RELEASE-NAME-airflow-metadata key : connection image : dummy_image imagePullPolicy : IfNotPresent name : base volumeMounts : - mountPath : '/opt/airflow/logs' name : airflow-logs - mountPath : /opt/airflow/airflow.cfg name : airflow-config readOnly : true subPath : airflow.cfg restartPolicy : Never securit圜ontext : runAsUser : 50000 fsGroup : 50000 serviceAccountName : 'RELEASE-NAME-worker-serviceaccount' volumes : - emptyDir : ' ) except ValueError as e : if i > 4 : raise e sidecar_task = test_sharedvolume_mount ()

Also, configuration information specific to the Kubernetes Executor, such as the worker namespace and image information, needs to be specified in the Airflow Configuration file.Īdditionally, the Kubernetes Executor enables specification of additional features on a per-task basis using the Executor config. One example of an Airflow deployment running on a distributed set of five nodes in a Kubernetes cluster is shown below.Ĭonsistent with the regular Airflow architecture, the Workers need access to the DAG files to execute the tasks within those DAGs and interact with the Metadata repository. The worker pod then runs the task, reports the result, and terminates. Apache Airflow - A platform to programmatically author, schedule, and monitor workflows - airflow/chart/files/pod-template-file.kubernetes-helm-yaml at main. When a DAG submits a task, the KubernetesExecutor requests a worker pod from the Kubernetes API. KubernetesExecutor requires a non-sqlite database in the backend. Not necessarily need to be running on Kubernetes, but does need access to a Kubernetes cluster. KubernetesExecutor runs as a process in the Airflow Scheduler. The Kubernetes executor runs each task instance in its own pod on a Kubernetes cluster. But What About Cases Where the Scheduler Pod Crashes?.Debugging Airflow DAGs on the command line.

Airflow kubernetes executor pod template