A question for data engineers who are responsible for building and maintaining production ETL pipelines, managing data quality across large datasets, and keeping infrastructure costs in check — is Databricks actually worth the commitment, or do the cluster management complexity and DBU cost unpredictability offset the platform integration benefits?
Feature: Databricks unifies Spark-based processing, collaborative notebooks (Python, SQL, Scala), Delta Lake for ACID-compliant table management, Workflows for job scheduling, and Unity Catalog for governance — in one platform. Delta Lake improves pipeline reliability through schema enforcement and time travel for debugging, the notebook environment makes step-by-step transformation testing immediately visible without deploying separately.
Integration: The platform connects to cloud storage, data warehouses, Git, and third-party tools as part of the ETL workflow. SnapLogic lands data in Bronze and Databricks handling Silver and Gold transformation, with serverless compute and Unity Catalog managing governance without separate tooling.
Some big data processing and distributions software alternatives worth comparing:
The honest read: Databricks earns its place for data engineers managing complex, multi-stage Spark pipelines where the Delta Lake reliability, notebook collaboration, and ML-to-engineering workflow continuity justify the learning curve on cluster and compute configuration.
Would value input from data engineers who have used Databricks in production for more than six months. What was the first cluster configuration mistake that taught you the most about cost management, and at what pipeline complexity did the Delta Lake + Unity Catalog combination start delivering a noticeable reliability improvement over your previous stack?
##### Post Metadata - Posted at: 15 days ago - Author title: Marketing Executive - Net upvotes: 1 ## Comments ### Comment 1 <p>The "worth it" answer probably hinges on something not mentioned yet: whether the same team spans both exploratory ML work and production ETL, or whether those are two separate teams in your org. Databricks' pitch is largely about continuity between notebook exploration and production pipelines. If data engineering and ML rarely touch each other's work, that continuity benefit goes mostly unused, and you're left paying the DBU premium and cluster overhead for a unification story that doesn't match how your org is actually structured.</p> ##### Comment Metadata - Posted at: 15 days ago - Author title: Tech Consultant ## Related Product [Databricks](https://www.g2.com/products/databricks/reviews) ## Related Category [Big Data Processing and Distribution](https://www.g2.com/categories/big-data-processing-and-distribution) ## Related discussions - [How well does Trello scale into a larger team?](https://www.g2.com/discussions/1-how-well-does-trello-scale-into-a-larger-team) - Posted at: over 13 years ago - Comments: 6 - [Can we please add a new section](https://www.g2.com/discussions/2-can-we-please-add-a-new-section) - Posted at: over 13 years ago - Comments: 0 - [Quantifiable benefits from implementing your CRM](https://www.g2.com/discussions/quantifiable-benefits-from-implementing-your-crm) - Posted at: over 13 years ago - Comments: 4