Which Airflow Components Process the DAG Files

The component responsible for reading our Python files that contains DAGs and creating internal DAG objects out of our code for execution has been the Scheduler itself for a pretty long time. But with Airflow 2.3.0 a lot of code had been refactored and this entire processing was separated into a component called DagFileProcessorManager. DagFileProcessorManager […]

What Files are Processed by Airflow to Load DAGs

It is important to know the source file(s) selection process of Airflow that are read to load the DAGs. This process is implemented by the DagFileProcessorProcess, triggered by the DagFileProcessorManager. Let’s go through each step that is involved in creating the file paths list: First, the path or directory is resolved from where Airflow (or […]

How to Capture and Log Tracebacks (Stack trace) in Python

Once you’ve caught an exception, you can capture and log a message with the exception info (including full stack traceback), using logging.exception() or logging.error(). With logging.exception(): With logging.error() we must set the exc_info keyword argument to True (default is False): In both cases, the Error occurred message is logged with level ERROR on the root […]