Airflow allows configuring and tuning the parallelism/concurrency of tasks at various levels (like DAGs) and via different approaches (like Pools). But at the top level, the father of all knobs provided that controls the overall number of task instances that the entire cluster can run simultaneously (running state), is the scheduler’s (or more appropriately the […]
The most common desired alerting mechanism when an Airflow task fails is slack notifications. This can be achieved simply with two different aspects or components of Airflow: Airflow Callbacks, and Airflow Slack provider package. Callbacks With Airflow Callback functions feature, one can run a particular Python function when a task fails, succeeds, is up for […]
The component responsible for reading our Python files that contains DAGs and creating internal DAG objects out of our code for execution has been the Scheduler itself for a pretty long time. But with Airflow 2.3.0 a lot of code had been refactored and this entire processing was separated into a component called DagFileProcessorManager. DagFileProcessorManager […]
It is important to know the source file(s) selection process of Airflow that are read to load the DAGs. This process is implemented by the DagFileProcessorProcess, triggered by the DagFileProcessorManager. Let’s go through each step that is involved in creating the file paths list: First, the path or directory is resolved from where Airflow (or […]
When you start a celery worker in Airflow with the airflow celery worker command, you can specify the –concurrency|-c option to define how many worker processes you need. What does this really mean though? It’s an option that is simply passed on to Celery when you are using Celery to execute your Airflow tasks (CeleryExecutor). […]
Imagine you have a DAG scheduled to run every 5 minutes: Once in a while, a DAG Run may take more than 5 minutes causing the next DAG Run of the same DAG to trigger in parallel, i.e., you’ll end up having two DAG runs of the same DAG running parallelly/concurrently at the same time. […]
When using the KubernetesPodOperator (KPO) in Airflow, we can specify the resource limits and requirements for the pod containers. This can be specified across resources like CPU, memory, GPU and ephemeral storage. Let’s take a look at how the resource requests and limits have to be specified in code. Dict The simplest way is to […]