LLM Engineering
Prompting, context, embeddings, and evaluation — the craft of making a model behave predictably.
Who this is for
Engineers who own model behaviour: quality, grounding, and the regression suite behind it.
What you should be able to do
Hold a model to a measured quality bar with an eval set that fails the build when it slips.
Career ladder
The titles this track maps onto. Levels differ between companies — the useful part is the direction, and what each step adds to the one before it.
- Prompt Engineer
- LLM Engineer
- Senior LLM Engineer
- Staff LLM Engineer
- Principal LLM Engineer
Tech stack
What the work is actually done with. Grouped by the job each tool does, so the list reads as a system rather than a pile of names.
- Prompting
- System promptsStructured outputsTool schemas
- Context
- ChunkingRerankingContext windowsToken budgeting
- Adaptation
- LoRAFine-tuningDistillation
- Vectors
- pgvectorPineconeWeaviateMilvusQdrant
- Evaluation
- Golden setsLLM-as-judgeGuardrailsHallucination scoring
The delivery flow
The order the work actually happens in. Each step is where a decision gets made and written down, not a chapter heading.
- Task definition
- Prompt draft
- Structured output schema
- Retrieval grounding
- Golden set
- Automated eval
- Guardrails
- Regression gate
Reference repository structure
A starting layout for this track. The directories are the ones that get added late and hurt — decisions, evals, policy, lineage — promoted to the top level where they are visible.
llm-workspace/ ├── prompts/ │ ├── system/ # one file per persona, versioned │ ├── task/ │ └── CHANGELOG.md # every prompt change, with the eval delta ├── schemas/ # structured output + tool definitions ├── context/ │ ├── chunking/ │ └── rerank/ ├── evals/ │ ├── golden/ # inputs + expected properties │ ├── judges/ # rubric prompts │ └── report/ # scored runs, kept for comparison ├── guardrails/ # input and output validators └── experiments/
Reference implementations
Citadel’s open-source repositories for this track — Terraform modules, MCP servers, and reference architectures you can read, fork, and run. Apache/MIT licensed; check each repository for its terms.
- GitHub ★ 1rag-pipeline-hallucination-mitigationProduction RAG with 5-layer hallucination mitigation. Hybrid retrieval (RRF), cross-encoder reranking, NLI validation. 94.3% faithfulness. Python + Azure AI Search.rag-pipeline-hallucination-mitigation on github.com (external site, opens in a new tab)
- GitHubmcp-server-vector-dbMCP server for vector database operations - Pinecone, Weaviate, Qdrant, and ChromaDBmcp-server-vector-db on github.com (external site, opens in a new tab)
- GitHubAI-Agent-SpecializationsAI agent specialization patterns and prompt engineering frameworksAI-Agent-Specializations on github.com (external site, opens in a new tab)
Primary sources
The standards, framework documents, and vendor references this track is built against. Go here when you need the authoritative wording rather than a summary — in a security review or an audit, the source is what counts.
- Anthropic prompt engineering guide (external site, opens in a new tab)
- OpenAI evals framework (external site, opens in a new tab)
- Hugging Face documentation (external site, opens in a new tab)
- pgvector (external site, opens in a new tab)
- Weaviate documentation (external site, opens in a new tab)
- Qdrant documentation (external site, opens in a new tab)
- Ragas evaluation toolkit (external site, opens in a new tab)