Ilum: A Data Platform Built by Data Engineers, for Data Engineers Ilum is a Data Lakehouse platform that unifies data management, distributed processing, analytics, and AI workflows for AI engineers, data engineers, data scientists, and analysts. It belongs to the Data Platform, Data Lakehouse, and Data Engineering software categories and supports flexible deployment across cloud, on-premise, and hybrid environments. Ilum enables technical teams to build, operate, and scale modern data infrastructure using open standards. It integrates tools for batch processing, stream processing, notebook-based exploration, workflow orchestration, and business intelligence, All In a Single Platform. Ilum supports modern open table formats like Delta Lake, Apache Iceberg, Apache Hudi, and Apache Paimon. It also offers native integration with Apache Spark and Trino for compute, with Apache Flink support currently in development. Key features include: - SQL Editor: Query Delta, Iceberg, Hudi, or Spark SQL with autocomplete, result previews, and metadata inspection. - Data Lineage & Catalog: Visualize data flow using OpenLineage and explore datasets through a searchable Data Catalog. - Notebook Integration: Use built-in Jupyter notebooks pre-wired to Spark, metadata, and your data environment for exploration or modeling. - Spark Job Management: Submit, monitor, and debug Spark jobs with integrated logs, metrics, scheduling, and a built-in Spark History Server. - Trino Support: Run federated queries across multiple data sources using Trino directly from within Ilum. - Declarative Pipelines: Define repeatable ETL and analytics pipelines, with dependency tracking and recovery logic. - Automatic ERD Diagrams: Instantly generate ER diagrams from schemas to aid in data understanding and onboarding. - ML Experimentation & Tracking: Includes MLflow for managing experiments, tracking parameters, metrics, and artifacts, fully integrated with notebooks and data pipelines to streamline model development workflows. - AI Integration & Deployment: Supports both classical ML and modern AI use cases, including GenAI workflows, vector search, and embedding-based applications. Models can be registered, versioned, and deployed for inference within declarative pipelines. - Built-in AI Agent Interface: Ilum integrates, providing a GPT-style interface to interact with your data, trigger pipelines, generate SQL, or explore metadata using natural language, bringing GenAI capabilities directly into your data platform. - BI Dashboards: Native support for Apache Superset, with JDBC integration for Tableau, Power BI, and other BI tools. Additional highlights: - Multi-Cluster Management: Connect multiple Spark or Kubernetes clusters to scale and isolate workloads. - Fine-Grained Access Control: LDAP, OAuth2, and Hydra integration for secure, role-based access. - Hybrid Ready: Designed to replace Databricks or Cloudera in environments where cloud adoption is partial, regulated, or not possible.