AI Data Engineering
The retrieval corpus is the product. Pipelines, vector stores, and the lineage behind them.
Who this is for
Data engineers feeding AI systems, who own freshness, quality, and provenance.
What you should be able to do
Run an ingestion pipeline with lineage, quality gates, and a reproducible embedding step.
Career ladder
The titles this track maps onto. Levels differ between companies — the useful part is the direction, and what each step adds to the one before it.
- Data Engineer
- AI Data Engineer
- Senior Data Engineer
- Staff Data Engineer
- Principal Data Architect
Tech stack
What the work is actually done with. Grouped by the job each tool does, so the list reads as a system rather than a pile of names.
- Ingestion
- AirflowDagsterdbt
- Storage
- PostgreSQLS3Delta LakeBigQuery
- Vectors
- pgvectorPineconeWeaviateMilvus
- Graph
- Neo4jKnowledge graphs
- Quality
- Great ExpectationsLineagePII detection
The delivery flow
The order the work actually happens in. Each step is where a decision gets made and written down, not a chapter heading.
- Source inventory
- Ingestion
- PII classification
- Transformation
- Quality gate
- Chunk + embed
- Index
- Freshness monitor
Reference repository structure
A starting layout for this track. The directories are the ones that get added late and hurt — decisions, evals, policy, lineage — promoted to the top level where they are visible.
data-platform/ ├── sources/ # one connector per system of record ├── pipelines/ │ ├── ingest/ │ ├── transform/ # dbt models │ └── embed/ # chunking + embedding, versioned ├── contracts/ # schema contracts, breaking-change gate ├── quality/ # expectations, run per pipeline ├── lineage/ # source -> chunk -> answer, traceable ├── pii/ # classification rules and redaction └── warehouse/
Reference implementations
Citadel’s open-source repositories for this track — Terraform modules, MCP servers, and reference architectures you can read, fork, and run. Apache/MIT licensed; check each repository for its terms.
- GitHubmcp-server-databaseMCP server for database operations - PostgreSQL, MySQL, and SQLite queries and schema managementmcp-server-database on github.com (external site, opens in a new tab)
- GitHubmcp-server-vector-dbMCP server for vector database operations - Pinecone, Weaviate, Qdrant, and ChromaDBmcp-server-vector-db on github.com (external site, opens in a new tab)
- GitHubterraform-gcp-cloud-sqlTerraform module for Google Cloud SQL with PostgreSQL, MySQL, high availability, read replicas, and backupsterraform-gcp-cloud-sql on github.com (external site, opens in a new tab)
- GitHubterraform-aws-dynamodbAWS DynamoDB Terraform module with global tables, DAX, PITR, contributor insights, and auto-scalingterraform-aws-dynamodb on github.com (external site, opens in a new tab)
- GitHubterraform-aws-rds-auroraProduction-grade AWS Aurora Terraform module with MySQL/PostgreSQL, global database, RDS Proxy, Activity Streams, and auto-scalingterraform-aws-rds-aurora on github.com (external site, opens in a new tab)
Primary sources
The standards, framework documents, and vendor references this track is built against. Go here when you need the authoritative wording rather than a summary — in a security review or an audit, the source is what counts.
- dbt documentation (external site, opens in a new tab)
- Apache Airflow documentation (external site, opens in a new tab)
- pgvector (external site, opens in a new tab)
- Neo4j documentation (external site, opens in a new tab)
- Great Expectations (external site, opens in a new tab)
- Delta Lake documentation (external site, opens in a new tab)