G2

Vikas s.

Senior Machine Learning Engineer

Information Technology and Services

Mid-Market (51-1000 emp.)

10/29/2026

"Robust DAG orchestration for distributed machine learning pipelines"

5/5

What do you like best about Apache Airflow?

Apache Airflow is the gold standard for orchestrating complex, multi-stage ML pipelines as Directed Acyclic Graphs (DAGs) defined entirely as code. Its programmatic approach allows us to dynamically construct data processing workflows, manage task dependencies, and parameterize model retraining jobs with complete flexibility. Airflow's rich ecosystem of providers and operators—such as the KubernetesPodOperator, Amazon SageMaker, and Google Cloud Vertex AI integrations—enables isolated, containerized task execution that scales effortlessly. The web UI provides clear visibility into task execution history, SLAs, and retry states, making it easy to isolate bottleneck tasks in feature engineering or batch inference pipelines. Review collected by and hosted on G2.com.

What do you dislike about Apache Airflow?

Managing state and passing large data payloads directly between tasks via XCom is inefficient and requires storing heavy intermediate artifacts in external object stores like AWS S3 or GCS. Airflow is strictly a workflow orchestrator rather than a data processing engine, so configuring local execution and testing complex DAGs during local development can feel heavy and resource-intensive. Additionally, tuning the Scheduler and Celery/Kubernetes Executor parameters for high-concurrency environments requires deliberate infrastructure management to avoid worker node resource contention or delayed task scheduling. Review collected by and hosted on G2.com.

What problems is Apache Airflow solving and how is that benefiting you?

Apache Airflow solves the issue of brittle, unmonitored cron jobs and fragmented pipeline dependencies across ML infrastructure. Before adopting Airflow, orchestrating data extraction, feature engineering, model training, and batch predictions relied on custom shell scripts that failed silently or lacked automated retry mechanisms. Airflow centralizes workflow scheduling, provides programmatic dependency enforcement, and automates fault-tolerant retries, significantly improving the reliability of our automated model retraining pipelines and cutting downtime for production ML workflows. Review collected by and hosted on G2.com.