This 5-day instructor-led training provides end-to-end coverage of Apache Airflow, the open-source platform for authoring, scheduling, and monitoring complex data workflows, with a practical focus on multi-cloud deployment using AWS, Azure, and Google Cloud Platform (GCP). Designed for data engineers, DevOps professionals, and cloud architects, this training enables participants to confidently develop, manage, and deploy production-ready data pipelines across hybrid cloud environments.
The program begins with essential Python programming skills, ensuring all participants can confidently author and manage DAGs (Directed Acyclic Graphs), which define workflows in Airflow. After setting a solid Python foundation, participants dive into the Airflow ecosystem, exploring its modular architecture, metadata database, scheduling engine, and user interfaces (CLI and UI).
Through a mix of lectures, real-world examples, and hands-on labs, learners will explore core concepts such as task dependencies, scheduling, backfilling, SLAs, branching, and dynamic DAG generation. The training includes deep coverage of Airflow Operators, including those for Python, Bash, SQL, and cloud-native integrations like AWS EMR, Azure HDInsight, Google Dataproc, and Blob/S3/Cloud Storage. Participants will also learn to use XComs for task communication, Sensors to wait for external events, and Hooks to interact with third-party systems.
The course goes beyond development by exploring Airflow Executors (Sequential, Local, Celery, Kubernetes) and configurations for scalable deployments. Security is addressed through encryption, role-based access, and connection secrets. The final modules focus on CI/CD, DevOps practices, and monitoring and profiling DAGs in real-world environments.
Throughout the training, learners will configure and deploy Airflow in different modes (standalone, Celery, Kubernetes), integrate with cloud services, and gain practical experience via cloud-specific DAGs, making this program uniquely suited for professionals building cross-cloud data workflows.
Duration: 5 Days
Course Code: BDT 509
Learning Objectives:
By the end of this course, participants will be able to:
This course is ideal for:
Module 1: Python Essentials for DAG Authoring
Module 2: Understanding Airflow in the ETL Landscape
Module 3: Airflow Architecture & Installation
Module 4: Configuring Airflow Environments
Module 5: DAG Authoring and Scheduling
Module 6: Using Operators in Practice
PythonOperator, BashOperator, SQL-based Operators
Hands-on Labs:
Module 7: Execution Engines & Task Control
Module 8: Modular & Dynamic Workflows
Module 9: Sensors and Hooks
Module 10: Plugins, Profiling & Airflow UI Extensions
Module 11: DevOps & CI/CD with Airflow
Module 12: Practical Cloud Integration
Training Material Provided:
