The component responsible for reading our Python files that contains DAGs and creating internal DAG objects out of our code for execution has been the Scheduler itself for a pretty long time. But with Airflow 2.3.0 a lot of code had been refactored and this entire processing was separated into a component called DagFileProcessorManager. DagFileProcessorManager […]
It is important to know the source file(s) selection process of Airflow that are read to load the DAGs. This process is implemented by the DagFileProcessorProcess, triggered by the DagFileProcessorManager. Let’s go through each step that is involved in creating the file paths list: First, the path or directory is resolved from where Airflow (or […]
When you start a celery worker in Airflow with the airflow celery worker command, you can specify the –concurrency|-c option to define how many worker processes you need. What does this really mean though? It’s an option that is simply passed on to Celery when you are using Celery to execute your Airflow tasks (CeleryExecutor). […]
When setting up a new or existing Firebase Web App on a new domain (or subdomain), you may encounter errors (shown below) where most of them talk about requests being blocked by the referrer which is yoursite.com or sub.yoursite.com. Errors Errors or warnings of the following kind may be seen or observed. Firebase config instantiation […]
Imagine you have a DAG scheduled to run every 5 minutes: Once in a while, a DAG Run may take more than 5 minutes causing the next DAG Run of the same DAG to trigger in parallel, i.e., you’ll end up having two DAG runs of the same DAG running parallelly/concurrently at the same time. […]
Once you’ve caught an exception, you can capture and log a message with the exception info (including full stack traceback), using logging.exception() or logging.error(). With logging.exception(): With logging.error() we must set the exc_info keyword argument to True (default is False): In both cases, the Error occurred message is logged with level ERROR on the root […]
When using the KubernetesPodOperator (KPO) in Airflow, we can specify the resource limits and requirements for the pod containers. This can be specified across resources like CPU, memory, GPU and ephemeral storage. Let’s take a look at how the resource requests and limits have to be specified in code. Dict The simplest way is to […]
At times you will want to catch an exception and extract the stack trace from the exception object to capture it somewhere (for instance log it). In our codebase, we generally aim to handle all kinds of unexpected or incorrect input and output by putting validations and expecting certain kinds of exceptions (try/except). But there […]
If your disk is almost full and you want to track down where the disk space is going, i.e., which files or folders are occupying the most or maximum amount of space then you can start with using the du (disk usage) utility to which you’ll have to pass a couple of options: The different […]
TL;DR: To sort du command’s output in ascending order of the file and/or directory sizes, we can pipe its output to sort -h. To sort in descending order, pipe it to sort -hr. This should work in most Unix-based or Unix-like operating systems (Linux, BSD, macOS): The du command displays the disk usage statistics (estimated […]