I finally got a certification to formally prove my knowledge in Airflow. Several ML and data job openings that I am interested are scouting for skills in creation of data pipelines and particularly in using Airflow.

My prior hands-on experience in creating data pipelines helped me a lot when I started learning Airflow. I am very thankful to astronomer.ai for their Airflow fundamentals preparation course. My heartfelt thanks to Marc Lamberti for his fantastic teaching style. They have invested a lot of time and effort in designing and organising this course. Each video is concise, clear and very effective in explaining Airflow concepts. I very highly recommend taking the Apache Airflow Fundamentals Preparation course.

I recommend using astrocloud docker images for Airflow to avoid losing a lot of time in installing Airflow correctly. This is explained in the course videos. If you are interested in trying out some code yourself, please feel free to check out some code here https://github.com/fastaimldata/astro

Not content with just the course, I also read some Guides provided on the astronomer site. I found some of them, like the one on Airflow Decorators to be very useful and well-written.

Disclaimer: I have prior hands-on experience in creating data pipelines to pre-process and prepare raw data collected from the web using luigi. Luigi was originally developed by Erik Bernhardsson and Elias Freider at Spotify, generously open-sourced and actively maintained. If you are wondering, there is a nice UI for task monitoring and decent documentation. It is sophisticated enough since it let me create and run batch tasks on multiple machines. Unfortunately, I developed these data pipelines for my company. Which means this code is closed-source and I wont be able to release it!

Here is my verifiable certificate link from credly. Who else is going to get started?