I like Apache Airflow because of its flexibility, scalability, and powerful workflow orchestration capabilities. It provides a structured and efficient way to design, schedule, execute, and monitor complex data pipelines and automated workflows. Its DAG (Directed Acyclic Graph) structure makes workflows easy to understand, manage, and troubleshoot by clearly defining the dependencies and sequence of tasks.

Another major advantage of Airflow is its extensive range of integrations with databases, cloud platforms, data warehouses, APIs, and other data engineering tools. This makes it easy to incorporate Airflow into existing data ecosystems without requiring significant changes to the overall architecture. Its scheduling capabilities also allow workflows to run automatically at specific intervals or based on defined dependencies and conditions.

Airflow also offers strong monitoring and observability features, making it easier to track task execution, identify failures, retry unsuccessful tasks, and understand pipeline performance. Combined with its scalability and active open-source community, these features make Apache Airflow a reliable and powerful platform for building and managing automated data workflows efficiently. Review collected by and hosted on G2.com.

One of the challenges with Apache Airflow is that it can be somewhat complex to set up, configure, and manage, especially for beginners who are new to workflow orchestration. Understanding concepts such as DAGs, operators, scheduling, dependencies, and task execution can require some initial learning.

As workflows become larger and more complex, managing dependencies and configurations can also become challenging. Troubleshooting failed tasks may sometimes take time, as identifying the root cause of an issue can require checking logs, task dependencies, configurations, and external services. In addition, maintaining and monitoring a large number of DAGs can increase operational overhead. However, with proper documentation, best practices, and experience, these challenges can be managed effectively. Review collected by and hosted on G2.com.

It is easiy to deploy with docker. Provide secure authentication. A better UI in airlfow3.x. There. is many method, operator, hooks are added. easily to add dependecy. A better workflow monitoring tool. Easily schedule , activate and deactivate jobs. Good performance when used in any cloud platform like aws in serverless mode. easily scaleable. Easy to integrate third part library. Apache airflow is free of cost but you need to pay compute cost if you are using any cloud. A wide support community Review collected by and hosted on G2.com.

When I create a task in expansion mode, it doesn’t display the graphical flow properly. I also noticed there’s too much logging. And when setting it up on AWS MWAA in a private VPC, it requires additional configuration. Review collected by and hosted on G2.com.

We use Apache Airflow as the central orchestrator for our entire machine learning and data engineering lifecycle. Specifically, it manages the schedules and dependencies for pulling raw data from our production databases, orchestrating daily feature engineering jobs, and kicking off automated model retraining pipelines. Once the models are evaluated, Airflow also triggers the deployment scripts to push updated model artifacts to our staging environment. It basically acts as the glue holding our data workflows together, ensuring that everything runs in the exact sequence required.What I appreciate most about Airflow is its "configuration as code" philosophy. Because every workflow is defined as a Python script, it fits perfectly into our existing development practices. We can version control our DAGs in Git, run code reviews on them, and easily write custom operators when the built-in ones don't quite fit our needs. The UI is also incredibly detailed; when a complex pipeline fails at 3:00 AM, the tree view and the ability to dig directly into the logs of a specific failed task save us an immense amount of troubleshooting time. The open-source community is massive, so if you are trying to connect Airflow to a database or a cloud service, chances are a robust provider package already exists. Review collected by and hosted on G2.com.

The biggest headache with Airflow is definitely the operational overhead and the steep learning curve required to keep it running smoothly. Managing the scheduler, web server, and workers—especially when handling resource-intensive machine learning tasks—requires a lot of infrastructure tuning. We struggled initially with the scheduler getting bogged down and tasks getting stuck in a queued state. Additionally, local development can be painful to set up realistically, and the fact that the scheduler constantly parses Python files means you have to be extremely careful about how you write your code to avoid severe performance degradation. It is not a tool you can just turn on and forget about; it requires dedicated DevOps attention. Review collected by and hosted on G2.com.

Our data pipelines, and all machine learning workflows are orchestrated on top of Apache Airflow. We have a variety of ETL jobs running on a schedule and some more complicated multi-step ML training loops. Airflow is in the middle, consuming data from a number of production databases, running our data preprocessing scripts, feeding our modelling training pipelines, and sending the results to our data analytics dashboards and cloud storage. It's like the clock and the nervous system within our whole data architecture design.That's definitely the selling point of Airflow that everyone can't outrace from - configuration as code. Being completely in Python it gives us a lot of flexibility. I can work on real data workflows using common software engineering techniques such as git version control, code reviews, and unit-testing my code. The UI is also great for debugging: I'm sure someone is running a complex pipeline somewhere, at 3:00 AM some of their critical tasks are failing, and using the tree view and direct access to the task logs, I can be almost certain which task failed and why. Oh, and, because they have such a big community, we have an operator or a provider for almost any cloud service or tool we want to connect to. Review collected by and hosted on G2.com.

This is extremely powerful, but by no means a “set it and forget” it tool. The overhead of the infrastructure is definitely high, and the need to fine-tune and keep the scheduler up and running without a single point of failure can be very demanding on a medium sized team. It is also a rather steep learning curve for novices. Creating a DAG correctly is quite complex, because one has to grasp the concepts such as idempotence, execution dates, etc., which are not intuitive for a junior engineer and can lead to misunderstandings and Double Runs. Finally, the local development machine may be somewhat cumbersome and complicated to set up. Review collected by and hosted on G2.com.

What I like most about Airflow is its flexibility and number of features for building workflows using DAGs. It is very useful for managing complex pipelines with dependencies. Integration with different systems is also strong. Once setup is done, it works reliably and is used frequently in day-to-day operations. Review collected by and hosted on G2.com.

Ease of use is one area where it can improve, especially for new users. Initial setup and implementation take effort if you are managing it yourself. The UI is not very user-friendly and sometimes slow. Also, debugging failed workflows can take time. Review collected by and hosted on G2.com.

I use Apache Airflow to schedule and manage data pipelines, and I appreciate how it automates ETL pipelines easily and enables monitoring of tasks. I love its flexibility to build and manage complex workflows effortlessly using code, along with features like Operators and sensors. The UI is very helpful for tracking pipelines and quickly debugging failures. It helps me create, automate, and schedule complex workflows by reducing manual efforts, making pipeline monitoring and error handling much easier. I love most of the features that Airflow provides, and I rate it 9/10 as I love this platform. Review collected by and hosted on G2.com.

For beginners, Apache Airflow can be complex to setup, especially with Docker. At times when a DAG is updated, the UI can feel slow. Review collected by and hosted on G2.com.

What I like most about Apache Airflow is how it makes managing workflows feel organised and predictable. You can clearly define tasks, set dependencies, and see everything laid out in one place, which makes complex pipelines much easier to understand.

I also like how flexible it is. You can build and customise workflows to fit pretty much any data pipeline, and the scheduling just works once it is set up. The UI is another big plus, it makes it easy to monitor runs, debug issues, and quickly see where something has failed without digging through logs for ages. Review collected by and hosted on G2.com.

One thing I don’t like about Apache Airflow is that it can feel quite heavy and complex, especially at the start. Setting it up properly takes time, and there are quite a few moving parts to understand before everything runs smoothly.

Debugging can also be frustrating at times. When something fails, the logs are not always the easiest to follow, so it can take longer than expected to figure out what went wrong. It can also be a bit resource intensive, which is not ideal if you are running smaller projects or just need something lightweight. Review collected by and hosted on G2.com.

For me, the standout feature is definitely the Web UI. As a data engineer, I often find myself troubleshooting, and the Grid view in Airflow makes it remarkably simple to identify exactly where a pipeline has failed. I can quickly access the logs for any specific task and determine what went wrong within seconds. This level of transparency is something that traditional cron jobs or basic scripts simply don't offer. Having a central dashboard for all your workflows truly provides peace of mind. Review collected by and hosted on G2.com.

The main challenge is the ease of implementation for beginners. Setting up the infrastructure (like the webserver, scheduler, and database) requires a good bit of DevOps knowledge, which can be a hurdle for a small team.

Because it’s open-source, you don’t have traditional customer support, so you rely heavily on the community. While the community is active, the documentation can sometimes be a bit overwhelming when you are trying to troubleshoot a very specific configuration issue. It’s a powerful tool, but the ease of use from a setup perspective could definitely be improved. Review collected by and hosted on G2.com.

What I like best about Apache Airflow is its flexibility in building and managing complex workflows using code. The DAG-based approach makes it easy to define dependencies, schedule jobs, and monitor pipeline execution in a centralized way. Review collected by and hosted on G2.com.

Debugging failed workflows and managing dependencies across multiple tags can become challenging over time. Also, initial setup and maintenance can be complex, especially for large-scale deployments if it is not a managed service. Basic local setup is manageable, but production-grade deployment with scaling, monitoring, and high availability requires significant expertise. Review collected by and hosted on G2.com.

I love using Apache Airflow for creating ETL pipelines, especially with its UI-driven DAG visualization that makes understanding the workflow and dependencies so much easier. The graph and stage view are amazing, and identifying errors is straightforward. The task monitoring feature is really useful, letting me see which jobs are running, and the retry and failure handling are essential for smooth operations. Setup was straightforward, particularly on AWS, and it's incredibly helpful when integrated with Acceldata for pipeline observability. Review collected by and hosted on G2.com.

The only thing I noticed, when there are more DAGs, is that the UI can sometimes feel slow when dealing with many tasks. Review collected by and hosted on G2.com.