LLM Engineering
Prompting, context, embeddings, and evaluation, the craft of making a model behave predictably.
Who this is for
Engineers who own model behaviour: quality, grounding, and the regression suite behind it.
What you should be able to do
Hold a model to a measured quality bar with an eval set that fails the build when it slips.
Career ladder
The titles this track maps onto. Levels differ between companies. The useful part is the direction, and what each step adds to the one before it.
- Prompt Engineer
- LLM Engineer
- Senior LLM Engineer
- Staff LLM Engineer
- Principal LLM Engineer
Tech stack
What the work is actually done with. Grouped by the job each tool does, so the list reads as a system rather than a pile of names.
- Prompting
- System promptsStructured outputsTool schemas
- Context
- ChunkingRerankingContext windowsToken budgeting
- Adaptation
- LoRAFine-tuningDistillation
- Vectors
- pgvectorPineconeWeaviateMilvusQdrant
- Evaluation
- Golden setsLLM-as-judgeGuardrailsHallucination scoring
The delivery flow
The order the work actually happens in. Each step is where a decision gets made and written down, not a chapter heading.
- Task definition
- Prompt draft
- Structured output schema
- Retrieval grounding
- Golden set
- Automated eval
- Guardrails
- Regression gate
Reference repository structure
A starting layout for this track. The directories are the ones that get added late and hurt. Decisions, evals, policy, lineage. Promoted to the top level where they are visible.
llm-workspace/ ├── prompts/ │ ├── system/ # one file per persona, versioned │ ├── task/ │ └── CHANGELOG.md # every prompt change, with the eval delta ├── schemas/ # structured output + tool definitions ├── context/ │ ├── chunking/ │ └── rerank/ ├── evals/ │ ├── golden/ # inputs + expected properties │ ├── judges/ # rubric prompts │ └── report/ # scored runs, kept for comparison ├── guardrails/ # input and output validators └── experiments/
Reference implementations
Citadel’s open-source repositories for this track. Terraform modules, MCP servers, and reference architectures you can read, fork, and run. Apache/MIT licensed; check each repository for its terms.
- GitHub ★ 1rag-pipeline-hallucination-mitigationProduction RAG with 5-layer hallucination mitigation. Hybrid retrieval (RRF), cross-encoder reranking, NLI validation. 94.3% faithfulness. Python + Azure AI Search.rag-pipeline-hallucination-mitigation on github.com (external site, opens in a new tab)
- GitHubmcp-server-vector-dbMCP server for vector database operations - Pinecone, Weaviate, Qdrant, and ChromaDBmcp-server-vector-db on github.com (external site, opens in a new tab)
- GitHubAI-Agent-SpecializationsAI agent specialization patterns and prompt engineering frameworksAI-Agent-Specializations on github.com (external site, opens in a new tab)
Primary sources
The standards, framework documents, and vendor references this track is built against. Go here when you need the authoritative wording rather than a summary. In a security review or an audit, the source is what counts.
- Anthropic prompt engineering guide (external site, opens in a new tab)
- OpenAI evals framework (external site, opens in a new tab)
- Hugging Face documentation (external site, opens in a new tab)
- pgvector (external site, opens in a new tab)
- Weaviate documentation (external site, opens in a new tab)
- Qdrant documentation (external site, opens in a new tab)
- Ragas evaluation toolkit (external site, opens in a new tab)