# Data Engineering

> The pipelines and platforms that move, transform, and govern data at scale: processing engines like Apache Spark, Kafka, and dbt, ingestion tools like Airbyte and Fivetran, lakehouse formats like Iceberg and Delta Lake, data catalogs, and synthetic data generators like Gretel AI. They feed the training corpora, fine-tuning sets, and RAG knowledge bases everything else depends on. AI-specific labeling lives in AI Data & Human Feedback.

24 tools in 5 subcategories. Web page: https://ailandscape.org/category/data-engineering · Each tool also has a markdown version at /tool/{slug}.md

## Data Processing

- [Apache Flink](https://ailandscape.org/tool/apache-flink): Stateful stream processing framework
- [Apache Hadoop](https://ailandscape.org/tool/apache-hadoop): Distributed storage and processing framework
- [Apache Kafka](https://ailandscape.org/tool/apache-kafka): Distributed event streaming platform
- [Apache Pulsar](https://ailandscape.org/tool/apache-pulsar): Cloud-native distributed messaging and streaming
- [Apache Spark](https://ailandscape.org/tool/apache-spark): Unified analytics engine for large-scale data
- [dbt](https://ailandscape.org/tool/dbt): Data transformation tool for analytics engineers
- [Redpanda](https://ailandscape.org/tool/redpanda): Kafka-compatible streaming platform without ZooKeeper

## Ingestion & Integration

- [Airbyte](https://ailandscape.org/tool/airbyte): Open-source data integration platform
- [dltHub](https://ailandscape.org/tool/dlthub): Open-source data load tool for Python-first pipelines
- [Fivetran](https://ailandscape.org/tool/fivetran): Automated data movement platform
- [Meltano](https://ailandscape.org/tool/meltano): Open-source ELT platform for Singer taps and targets
- [Stitch](https://ailandscape.org/tool/stitch): Simple, extensible ETL platform

## Data Lakehouse

- [Apache Hudi](https://ailandscape.org/tool/apache-hudi): Transactional data lake platform
- [Apache Iceberg](https://ailandscape.org/tool/apache-iceberg): Open table format for large analytic datasets
- [Delta Lake](https://ailandscape.org/tool/delta-lake): Open-source storage framework for data lakehouses

## Data Catalog

- [Alation](https://ailandscape.org/tool/alation): Enterprise data intelligence platform
- [Amundsen](https://ailandscape.org/tool/amundsen): Data discovery and metadata engine by Lyft
- [Apache Atlas](https://ailandscape.org/tool/apache-atlas): Data governance and metadata management
- [DataHub](https://ailandscape.org/tool/datahub): Metadata platform for data ecosystem
- [OpenMetadata](https://ailandscape.org/tool/openmetadata): Open standard for metadata and data discovery

## Synthetic Data

- [Gretel AI](https://ailandscape.org/tool/gretel-ai): Synthetic data platform for privacy-preserving ML
- [Mostly AI](https://ailandscape.org/tool/mostly-ai): Enterprise synthetic data generation platform
- [SDV](https://ailandscape.org/tool/sdv): Synthetic Data Vault for tabular and relational data
- [YData](https://ailandscape.org/tool/ydata): Synthetic data generation for ML and testing

---

Part of [AI Landscape](https://ailandscape.org), an open map of the AI ecosystem. Index for AI assistants: https://ailandscape.org/llms.txt