Vector Databases for RAG: pgvector, Pinecone, Weaviate & Milvus
A vector database is the retrieval backbone of a Retrieval-Augmented Generation (RAG) system: it stores the numerical embeddings that represent your documents and returns the passages most semantically similar to a user's query, so a language model can answer from your own knowledge rather than its training data alone. Choosing among pgvector, Pinecone, Weaviate, and Milvus is less about which is "best" and more about matching a tool to your scale, your operational appetite, and your governance requirements. This page explains what a vector database does inside RAG, how the leading options differ conceptually, and the pitfalls that quietly degrade retrieval quality in production.
What a vector database is
An embedding model converts text (or images, code, audio) into a high-dimensional vector — a list of numbers that captures meaning, so that semantically similar content sits close together in that space. A vector database indexes those vectors and answers a specific question quickly: given this query vector, which stored vectors are nearest? "Nearest" is measured by a distance metric such as cosine similarity or dot product. The database's job is to make approximate nearest-neighbor (ANN) search fast enough to run inside a live request, using index structures like HNSW graphs or IVF partitions that trade a small amount of recall for a large gain in speed.
In a RAG pipeline the vector database sits between ingestion and generation. During ingestion you chunk source documents, embed each chunk, and store the vectors alongside metadata. At query time you embed the user's question, retrieve the top-matching chunks, and pass them to the model as grounding context. If you are weighing retrieval against retraining the model itself, see RAG vs fine-tuning; the two solve different problems and are often combined.
When and why the choice matters
The vector store is where retrieval quality, latency, and cost concentrate, so the wrong fit shows up as slow answers, ballooning infrastructure bills, or an operational burden your team cannot sustain. A few conceptual distinctions guide the decision:
- pgvector is a Postgres extension. Its defining advantage is that vectors live in the same database as your relational data, so you can filter, join, and enforce access control with SQL you already run. For teams already operating Postgres, it removes a moving part entirely. It suits small-to-moderate corpora and workloads where transactional consistency and simplicity outweigh specialized scale.
- Pinecone is a fully managed, serverless service. You trade direct control of the infrastructure for someone else running, scaling, and tuning it. That is attractive when you want to avoid operating a database and can accept a managed dependency and its cost model.
- Weaviate is an open-source, schema-aware vector database with built-in hybrid search (combining keyword and vector signals) and optional module integrations. It fits teams that want a purpose-built store with richer querying and the option to self-host or use its managed tier.
- Milvus is an open-source system built for very large scale, with a distributed architecture that separates storage and compute. It earns its added operational complexity when you genuinely have billions of vectors or demanding throughput requirements.
The honest heuristic: start with pgvector if you already run Postgres and your corpus is modest; reach for a dedicated store when scale, hybrid search, or throughput exceed what a general-purpose database handles comfortably. Avoid choosing the most powerful option "to be safe" — you inherit its operational weight regardless of whether you use its ceiling.
How it works in practice
Retrieval quality depends far more on the pipeline around the database than on the database brand. Common pitfalls include chunking documents too coarsely (burying the relevant sentence in noise) or too finely (losing context); mixing embedding models between ingestion and query time, which silently breaks similarity; and ignoring metadata filtering, so retrieval pulls semantically similar but contextually wrong passages across tenants or time periods. Index parameters also embody a recall-versus-latency trade-off that should be tuned against your own data, not left at defaults. Hybrid search — blending exact keyword matching with vector similarity — often recovers cases where pure semantic search misses exact identifiers, error codes, or proper nouns. For a deeper treatment of the full pipeline, our engineering guide on building production RAG pipelines walks through these decisions end to end.
Security and governance considerations
A vector store holds a re-representation of your source content, which means it inherits your source content's sensitivity. Embeddings are not anonymized data — with the right access they can reveal what the underlying documents contain. Governance therefore requires access control on retrieval (a user should only ever retrieve chunks they are entitled to see), tenant isolation so one customer's data cannot surface in another's answers, encryption in transit and at rest, and auditability of what was retrieved to produce a given answer. Where you host the store — self-managed versus a third-party managed service — is a data-residency and vendor-risk decision, not merely a convenience one. These concerns sit inside a broader enterprise AI security posture and connect to the same principles covered in our writing on AI security architecture.
How Citadel helps
Citadel Cloud Management designs RAG systems as governed AI agents rather than loose scripts bolted onto a model. That means the vector database is chosen for your actual scale and operating model, retrieval is scoped by identity and permission, and every answer is traceable to the sources that produced it — the same discipline we bring across our AI agents practice. If you are evaluating where to store embeddings, how to keep retrieval trustworthy, or how to bring an existing RAG prototype up to enterprise standards, book a discovery call and we will map the right architecture to your requirements.