Skip to content
anythingai.space
06Retrieval Systems·20 min read

Vector Databases

Vector databases are purpose-built systems for storing, indexing, and searching high-dimensional embeddings. The market has split into Managed Serverless and Specialized High-Performance engines. We no longer ask "Does it support vector search?" (Postgres, Redis, and Mongo all do). We ask "Does it scale to 100M+ vectors with sub-100ms P99 and full metadata filtering?"


What Is a Vector Database

A vector database stores embeddings (dense vectors) and enables fast similarity search over them.

text
Traditional DB: SELECT * FROM docs WHERE category = 'tech'
Vector DB: SELECT * FROM docs ORDER BY similarity(embedding, query_embedding) LIMIT 10

Core Capabilities

CapabilityPurpose
Vector storagePersist high-dimensional embeddings
Similarity searchFind nearest neighbors quickly
Metadata filteringCombine vector search with attribute filters
CRUD operationsUpdate embeddings as data changes
ScalingHandle millions to billions of vectors

Why Not General Databases?

Traditional databases can store vectors but lack optimized search:

ApproachSearch ComplexityPractical at Scale
Brute force (e.g., pgvector with no HNSW or IVFFlat index)O(n * d)OK to ~1M vectors
ANN index (dedicated vector DB)O(log n) or O(1)Yes, billions

Vector Search Fundamentals

Exact (brute force):

  • Compare query to every stored vector
  • O(n * d) per query
  • Perfect accuracy

Approximate Nearest Neighbor (ANN):

  • Use index structure to prune search space
  • Sub-linear complexity
  • Slightly lower recall (typically 95-99%)

Distance Metrics

MetricFormulaRangeBest For
Cosine1 - (a . b) / (norm(a) * norm(b))[0, 2]Text embeddings
Euclidean (L2)sqrt(sum((a - b)^2))[0, inf)Image embeddings
Dot producta . b(-inf, inf)Already normalized

For text embeddings: Use cosine similarity (or dot product if pre-normalized).

Recall vs Latency Tradeoff

text
^ Recall
|
100% | ------------------ Brute force
| * Well-tuned ANN
| *
| *
95% |* Fast ANN
|
+-----+-------+------> Latency
1ms 10ms

ANN indices trade some accuracy for speed. Tune for your requirements.


Indexing Algorithms

HNSW (Hierarchical Navigable Small World)

The most popular algorithm for production in-memory vector search.

How it works:

  1. Build a graph where nodes are vectors
  2. Connect to nearby neighbors
  3. Multiple layers of abstraction (hierarchical)
  4. Search: navigate from top layer down, greedy nearest neighbor
text
Layer 2: *--------*--------*
| | |
Layer 1: *--*--*--*--*--*--*
| | | | | | |
Layer 0: ******************** (all vectors)

Pros:

  • Excellent recall/latency tradeoff
  • No training required
  • Supports updates natively

Cons:

  • Memory-intensive: the full-precision vectors and the graph links both live in RAM
  • The graph itself is a small share at high dimensions. OpenSearch's sizing rule is 1.1 x (4 x dims + 8 x M) bytes per vector, about 12-19% above the raw float32 vectors at 1,536 dims for M of 16 to 64; low-dimensional vectors pay proportionally more
  • 10M vectors at 1,536 dims need ~70-75 GB of RAM per copy (61 GB of that is the float32 vectors), before replicas

Key parameters:

  • M: Max connections per node (16-64)
  • ef_construction: Build-time exploration (100-500)
  • ef_search: Query-time exploration (50-200)

DiskANN (SSD-based)

Built for billion-scale indexes that do not fit in RAM. The original paper served a billion SIFT points from one workstation with 64 GB of RAM and an SSD, at over 5,000 queries per second, under 3 ms mean latency and 95%+ 1-recall@1.

How it works:

  • Keeps the full-precision vectors and the graph on SSD (NVMe), with only compressed (PQ) codes in RAM to steer the search
  • Uses the Vamana algorithm for efficient disk-based graph traversal

Pros:

  • 10x cheaper than HNSW for billion-scale datasets with <5ms latency penalty
  • 90-95% reduction in RAM requirements vs HNSW

Cons:

  • Slightly higher latency than pure in-memory HNSW
  • Tail latency depends on NVMe IOPS, so size SSDs for query concurrency, not just capacity

Example: A 100-million-vector index with 1,536 dimensions needs roughly 700 GB of RAM per copy for HNSW (614 GB of float32 vectors plus graph and overhead). Using DiskANN, the RAM requirement drops by 90-95% while maintaining sub-10ms query times.

IVF (Inverted File Index)

Partition vectors into clusters, search only relevant clusters.

How it works:

  1. Use k-means to create centroids
  2. Assign each vector to nearest centroid
  3. At query time: find nearest centroids, search those clusters

Pros:

  • Lower memory than HNSW
  • Can use quantization (IVF-PQ)

Cons:

  • Requires training
  • Updates need re-clustering or hybrid approach

Key parameters:

  • nlist: Number of clusters (sqrt(n) rule of thumb)
  • nprobe: Clusters to search at query time

Product Quantization (PQ)

Compress vectors to reduce memory and speed up comparison.

How it works:

  1. Split vector into subvectors
  2. Quantize each subvector to a codebook
  3. Store codes instead of full vectors

Memory reduction: 4-32x typical

Tradeoff: Lower accuracy due to quantization loss

Low-Bit and Rotation-Based Quantization

Between May and October 2026 several engines shipped or extended 1 to 8-bit vector storage, mostly as opt-in or preview options rather than defaults. The newest trick is rotate first, then quantize. Embedding dimensions have very uneven variance, so per-dimension scalar quantization wastes bits on quiet dimensions and clips loud ones. Multiplying by a random or Hadamard rotation spreads the energy evenly across dimensions, after which a uniform 4-bit or 1-bit grid loses far less information.

EngineFeatureStatusFootprint
QdrantTurboQuant with Hadamard rotation (1.18, May 2026); Turbo4 storage type keeps only the 4-bit form (1.19, August 2026); turbo8 and turbo16 vector datatypes (1.20, October 2026)Opt-inFrom 36 bits per coordinate in 1.18 (float32 original plus 4-bit copy) to 4 bits with Turbo4, a ninefold cut
Weaviate4-bit Rotational Quantization (1.39, August 2026), adding to the 8-bit and 1-bit RQ it already had; 1.40 (October 2026) adds "RQ4 centered quantization"Preview in 1.39, HNSW only; the 1.40 notes give no status label, so confirm before relying on it1,536-dim vector from 6,144 bytes to 784 bytes (7.84x)
ElasticsearchDiskBBQ (bbq_disk, GA since 9.2) gained a symmetric 1-bit OSQ scorer and merge-time calibration (9.5, August 2026)GA; default index type for float vectors from 9.4 where the license includes it (Enterprise)~1 bit per dimension plus correction terms
turbopufferPlain i8 vector type (June 2026), no rotationShipped75% less storage and query cost than f32 for quantization-aware models (vendor)

Qdrant's own numbers show the tradeoff: 4-bit TurboQuant recall 0.9193 vs 0.9285 for 8-bit scalar quantization vs 0.9419 uncompressed on one dataset, at twice the compression of scalar quantization (vendor-reported). The design decision is whether to keep the float originals for rescoring. Keeping them on disk or object storage and rescoring the top few hundred candidates recovers most of the lost recall; dropping them (Turbo4) is the cheapest option and caps your maximum recall.

The RAM math changes accordingly: 100M vectors at 1,536 dims is ~614 GB as float32 but ~77 GB at 4 bits. At a billion vectors, that is the difference between a fleet of high-memory nodes and a few machines plus SSD.

Flat Index (Brute Force)

No approximation, exact search.

Use when:

  • Less than 100K vectors
  • Accuracy is critical
  • Latency budget is generous

Algorithm Comparison

AlgorithmMemoryBuild TimeQuery SpeedRecallUpdates
HNSWHighMediumVery fast95-99%Good
DiskANNLow (SSD)MediumFast95-99%Fair
IVFMediumFastFast90-98%Fair
IVF-PQLowFastFast85-95%Fair
FlatLowNoneSlow100%Instant

Competitive Landscape

Vector-Native (Dedicated)

DatabaseTypeBest ForPricing Model
PineconeManaged cloud (serverless standard); BYOC GA September 23, 2026 (data plane in your AWS, GCP or Azure account)Easy start, scale, managed SLAs; BM25 full-text search GA September 9, 2026Storage plus read and write units
QdrantOpen source / Cloud (Rust, high-perf)Self-hosted control, strong filtered search; vendor benchmarks cite ~12ms p99 at 10M vectors; 1.20 (October 2026) adds BM25 scoring on the full-text index, a code tokenizer, and per-element nested payload indexesPer GB (cloud) or free
WeaviateOpen source / CloudNative hybrid (BM25 + dense + metadata) in a single query, MMR diversity and Boost API (GA in 1.39), multimodal; 1.40 (October 2026) makes namespaces GA as a licensed feature, stops creating an implicit legacy vector index (auto-schema creates a default named vector), and defaults the vectorizer to none, so schemas that relied on the old defaults change on upgradePer dimension-hour
MilvusOpen source / Cloud (Zilliz)Distributed scale; 3.0 (July 2026) adds External Collections over Parquet, Iceberg, Lance and Vortex in object storage, plus online column backfillFree (self-host) or Zilliz Cloud
turbopufferManaged, object-storage-firstMany cold namespaces (per-tenant indexes) at low cost; native embedding GA September 29, 2026Usage-based
ChromaOpen source (Apache 2.0)Prototyping, local dev, embedded use, self-hosted or embedded only: Chroma's October 23, 2026 notice says Chroma Cloud stops accepting writes on December 7, 2026 and stops serving reads on December 22, 2026Free (self-host)

General-Purpose (Plugin/Extension)

DatabaseTypeBest ForPricing Model
pgvector (0.8.7)PostgreSQL extensionSmall scale, existing PG (HNSW + IVFFlat). Patch to 0.8.7 (October 1, 2026): it fixes CVE-2026-103484, an IVFFlat build overflow that any role allowed to create an index can triggerCompute only
Elasticsearch (9.5)Search engineHybrid via the rrf and linear retrievers, DiskBBQ quantized vectors, semantic_text fields that embed automatically at ingestLicense-based
MongoDB Atlas Vector SearchDocument DBExisting Mongo shops; Automated Embedding GA August 13, 2026 (Atlas embeds and re-embeds documents with Voyage models on write)Atlas cluster pricing

Where the Market Moved in 2026

Three shifts change how you evaluate an engine:

  1. Embedding moved into the database. MongoDB Atlas Automated Embedding (GA), turbopuffer native embedding (GA; turbopuffer cites Readwise cutting median embedding latency 8x by embedding in parallel with query execution), Elasticsearch semantic_text, and Milvus 3.0's Hugging Face Inference Providers integration all remove the separate embed-and-sync pipeline. The cost is coupling: the vendor now picks the model version, so ask how a model change triggers re-embedding before you adopt it.
  2. The index moved toward the lakehouse. Milvus 3.0 External Collections query Parquet, Iceberg, Lance and Vortex data in place (zero-copy, read-only), and turbopuffer's September 30 "RIP, vector database" post describes a v3 storage design where the ANN index becomes a secondary index rather than the primary data layout (announced; it passes CI but is not in production yet, and the scale figures in the post are the vendor's). The argument shifts from "which vector DB" to "index in place over object storage, or copy into a serving store."
  3. Low-bit quantization spread across vendors (see above), changing the RAM budget for HNSW sizing.
  4. The migration path moved inside Postgres. Pinecone's pg_pinecone (October 27, 2026, Apache 2.0) is a Postgres extension for tables that already have pgvector columns: each table picks a reversible engine (native, which runs in Postgres with no network calls or Pinecone account; pgvector, which uses your existing HNSW or IVFFlat index; or remote, which syncs to a Pinecone serverless index in the background) and runs BM25, vector or hybrid search from SQL. "Start on pgvector, migrate later" becomes a per-table switch with Postgres as the system of record. The price is a superuser extension (PostgreSQL 17 or 18, shared_preload_libraries for remote mode), which many managed Postgres plans will not allow, and the published performance figures are Pinecone's own, from one instance type and a 1M-vector dataset.

The same month taught the opposite lesson: a managed store can disappear on short notice. Chroma's October 23, 2026 wind-down notice gave Chroma Cloud users about six weeks until writes stop (December 7) and two more weeks until reads stop (December 22); open-source ChromaDB continues under Apache 2.0, and the notice says Replit will keep contributing to it (Dealroom reports Replit is acquiring Chroma; reported, not vendor-confirmed). Put an export path and the re-embed and re-index cost into the selection criteria, not just features.


Detailed Database Comparison

Feature Matrix

FeaturePineconeQdrantWeaviateMilvuspgvector
LanguageProprietaryRustGoGo/C++C
Hosted optionYesYesYesYes (Zilliz)Via cloud PG
Self-hostedBYOC only (data plane in your cloud)YesYesYesYes
ServerlessYes (Best)YesYesYes (Zilliz)No
Cloud-NativeAnyAnyAnyK8s for distributed; Docker standaloneAny
Metadata filteringGoodExcellentGoodGoodVia SQL
Hybrid searchSparse-dense, or BM25 FTS with client-side fusionNativeNativeNativeMulti-stage (limited)
Max vectorsBillionsBillionsBillionsBillions~10M
HNSW indexYesYesYesYesYes

Metadata Filtering

Critical for multi-tenant and filtering use cases.

python
# Pinecone
results = index.query(
vector=query_embedding,
top_k=10,
filter={"tenant_id": "123", "category": {"$in": ["tech", "science"]}}
)
# Qdrant (Query API; the legacy /search endpoint is removed from the 1.19 REST schema)
results = client.query_points(
collection_name="documents",
query=query_embedding,
limit=10,
query_filter=Filter(
must=[
FieldCondition(key="tenant_id", match=MatchValue(value="123")),
FieldCondition(key="category", match=MatchAny(any=["tech", "science"]))
]
)
).points

Performance impact: Filtering happens during search, not after. Pre-filtered indices are faster but less flexible.

Why metadata filtering is often the bottleneck: In naive vector search, we find the "Top K" nearest neighbors and THEN filter by metadata. If the filter is very restrictive, we might find 0 results after filtering. Specialized databases now use Pre-Filtering with HNSW, traversing the graph but only considering nodes that satisfy the boolean metadata constraint. This requires specialized bitmasks or hardware acceleration (SIMD) to keep latencies low.

Disk-Native Metadata: Modern DBs tier data between RAM and disk per component. Qdrant 1.19 replaced the old on_disk and always_ram flags with one memory setting (pinned, cached or cold) that applies to vectors, HNSW graphs, sparse and payload indexes, so complex filters (e.g., full-text + geo + vector) run without holding every payload index in RAM.


Query Patterns

python
def semantic_search(query: str, top_k: int = 5) -> list[Document]:
query_embedding = embed(query)
results = vector_db.search(query_embedding, top_k=top_k)
return [Document(id=r.id, text=r.payload["text"], score=r.score) for r in results]
python
def filtered_search(query: str, filters: dict, top_k: int = 5) -> list[Document]:
query_embedding = embed(query)
results = vector_db.search(
query_embedding,
top_k=top_k,
filter=filters # {"tenant_id": "abc", "created_after": "2025-01-01"}
)
return results

Pattern 3: Hybrid Search (Dense + Sparse)

python
def hybrid_search(query: str, alpha: float = 0.5, top_k: int = 5) -> list[Document]:
# Dense (semantic)
dense_embedding = embed(query)
dense_results = vector_db.search(dense_embedding, top_k=top_k * 2)
# Sparse (keyword)
sparse_results = bm25_search(query, top_k=top_k * 2)
# Combine with reciprocal rank fusion
combined = reciprocal_rank_fusion(
[dense_results, sparse_results],
weights=[alpha, 1 - alpha]
)
return combined[:top_k]

Some databases (Weaviate, Qdrant, Milvus, Elasticsearch) fuse dense and keyword results natively. Pinecone's BM25 full-text search ranks by one scoring type per request, so you run the lexical and vector searches separately and fuse them client-side (e.g., with RRF).

python
# Weaviate native hybrid (Python client v4)
docs = client.collections.use("Document")
results = docs.query.hybrid(
query=query,
alpha=0.5, # 0 = BM25 only, 1 = vector only
limit=5,
)

Pattern 4: Multi-Vector Query

For parent-child or multi-aspect retrieval:

python
def multi_vector_search(queries: list[str], top_k: int = 5) -> list[Document]:
all_results = []
for query in queries:
embedding = embed(query)
results = vector_db.search(embedding, top_k=top_k)
all_results.extend(results)
# Dedupe and rerank
unique = dedupe_by_id(all_results)
reranked = rerank(queries[0], unique) # Use primary query for reranking
return reranked[:top_k]

Production Operations

Capacity Planning

python
def estimate_resources(
num_vectors: int,
dimensions: int,
metadata_size_bytes: int = 500
) -> dict:
# Vector storage
vector_size = dimensions * 4 # float32
total_vector_storage = num_vectors * vector_size
# Index overhead (HNSW ~1.5x)
index_overhead = total_vector_storage * 1.5
# Metadata
metadata_storage = num_vectors * metadata_size_bytes
# Total
total_gb = (total_vector_storage + index_overhead + metadata_storage) / 1e9
# QPS estimate (rough)
qps_per_gb = 50 # depends heavily on config
estimated_qps = total_gb * qps_per_gb
return {
"storage_gb": total_gb,
"estimated_qps": estimated_qps,
"recommended_replicas": max(1, int(total_gb / 50)) # ~50GB per replica
}

Index Maintenance

python
class VectorDBMaintenance:
def __init__(self, client):
self.client = client
def add_documents(self, documents: list[Document]):
"""Upsert documents with batching."""
batch_size = 100
for i in range(0, len(documents), batch_size):
batch = documents[i:i + batch_size]
embeddings = embed_batch([d.text for d in batch])
self.client.upsert([
{
"id": doc.id,
"vector": embedding,
"payload": doc.metadata
}
for doc, embedding in zip(batch, embeddings)
])
def delete_documents(self, doc_ids: list[str]):
"""Delete by document ID."""
self.client.delete(ids=doc_ids)
def update_metadata(self, doc_id: str, metadata: dict):
"""Update metadata without re-embedding."""
self.client.set_payload(
collection_name="documents",
payload=metadata,
points=[doc_id]
)

High Availability

text
+-------------------------------------------------------------+
| Load Balancer |
+----------------------------+--------------------------------+
|
+----------------+----------------+
v v v
+--------------+ +--------------+ +--------------+
| Replica 1 | | Replica 2 | | Replica 3 |
| (Read) | | (Read) | | (Primary) |
+--------------+ +--------------+ +--------------+
|
(Replication)
|
+-----v-----+
| Storage |
+-----------+

Key patterns:

  • Leader-follower for writes
  • Read replicas for query scaling
  • Async replication for HA

Patching

Treat the vector store like any other database on the CVE feed. pgvector 0.8.7 (October 1, 2026) fixes CVE-2026-103484, a buffer overflow during IVFFlat index builds that can lead to code execution for any database role allowed to create an index; earlier 2026 patches fixed a parallel HNSW build overflow (0.8.2) and possible index corruption during HNSW vacuuming (0.8.3). On managed Postgres, confirm the provider has shipped 0.8.7, and do not grant CREATE INDEX to application roles in multi-tenant schemas.

For Milvus, the floor is 3.0.5 (October 30, 2026). 3.0.5 fixes upserts on CMEK-enabled collections writing the delete part (primary keys and timestamps) to the WAL unencrypted, which quietly breaks an encryption-at-rest claim; 3.0.4 (October 26) upgraded Go and golang.org/x/net for several CVEs and, with authorization on, made client telemetry admin APIs root-only and hid import job details from users without Import permission. 3.0.3 (October 14) rejects malformed sparse vectors in Search and HybridSearch that could cause a heap out-of-bounds write on the query node, and it stops DescribeCollection from showing metadata of collections the caller cannot access (3.0.1 had already fixed unauthenticated access through streaming gRPC calls on the external proxy port). Budget for the 3.0.3 breaking changes in the same upgrade: the legacy /api/v1/* routes on the metrics port are gone (point health probes at /healthz or /livez), and binlog imports now need the cluster-level ImportBinlog privilege. Qdrant 1.20 changes a security default in the other direction: service.enforce_internal_auth is now on whenever an API key is configured, so verify internal cluster traffic carries credentials before you roll it out.

The trust store is part of the patch surface too. After weak private keys were found on eight eMudhra intermediate CAs in October 2026, Mozilla and Chrome distrusted them and moved to drop the emSign ECC Root CA - G3 root, and Weaviate shipped patch releases on its 1.36 to 1.40 lines between October 22 and 28. Rebuild vector-store and client images on refreshed base images, and expect TLS to fail to any endpoint that chains only to the removed root.

Monitoring

python
VECTOR_DB_METRICS = [
"query_latency_p50",
"query_latency_p99",
"queries_per_second",
"index_size_gb",
"vector_count",
"filter_latency",
"upsert_latency",
"cache_hit_rate"
]
def alert_rules():
return {
"query_latency_p99_high": {
"condition": "query_latency_p99 > 500ms",
"severity": "warning"
},
"query_latency_p99_critical": {
"condition": "query_latency_p99 > 2000ms",
"severity": "critical"
},
"low_recall": {
"condition": "bench_recall < 0.90",
"severity": "warning"
}
}

Managed vs Self-Hosted (TCO Analysis)

Cost Comparison

AspectPinecone (Serverless)Self-Hosted (Qdrant/Milvus)
Ops OverheadZeroHigh (Requires K8s + SRE)
ScalingInstant (Scale to zero)Manual (Node provisioning)
Cost (Small)$0 - $100/mo$50/mo (Minimum instance)
Cost (Scale)High per token/vectorLow unit cost

Managed Service Pricing (indicative, always verify on provider pages)

ProviderModelExample: 10M vectors, 1536 dims
PineconeServerless (Node.js SDK v9, Go SDK v7 and Terraform provider v5, September to October 2026, can no longer create pod-based indexes; existing pods keep working, and Pinecone Local, the Docker emulator, reached end of life October 7, so CI tests need a separate project or a mock)~$70-150/month serverless
Qdrant CloudPer GB~$50/month (20GB)
Weaviate CloudPer dimensions~$100/month
Zilliz (Milvus)Per CU~$75/month

Managed RAG Pipelines (Build vs. Buy)

A newer option skips the vector database decision entirely: a managed pipeline that parses, chunks, embeds, indexes and retrieves, billed by ingested tokens, storage and queries. Cloudflare AI Search went GA on October 1, 2026; on November 2 Cloudflare postponed billing to December 1, 2026 (prices unchanged) and raised the default sync interval from 6 to 24 hours, so set it explicitly if new documents must be searchable sooner:

MeterPriceIncluded per month (Workers plans)
Ingestion$0.75 per 1M tokens (+$0.50 per 1M for images)5M tokens
Storage$2.00 per GB-month10 GB-month
Semantic, vector or hybrid queries$0.75 per 1K1,000
Full-text queries$0.10 per 1K1,000

The per-query meter dominates at scale: 10M hybrid queries a month is ~$7,500 in query fees alone, versus a self-hosted index whose marginal cost per query is close to zero once provisioned. Managed pipelines win on time-to-first-answer and for low or spiky volume; at steady high QPS, compare the query meter against node costs plus embedding API spend. Pinecone Nexus and Cohere Compass Cloud take the same "buy the pipeline" approach further; see Agentic RAG.

Self-Hosted Costs

python
def estimate_self_hosted_cost(
vectors: int,
dimensions: int,
cloud: str = "aws"
) -> dict:
storage_gb = (vectors * dimensions * 4 * 2.5) / 1e9 # 2.5x for index
# Instance sizing: an in-RAM HNSW index must fit in memory with headroom
if storage_gb < 12:
instance = "r6g.large" # 16 GB RAM
elif storage_gb < 25:
instance = "r6g.xlarge" # 32 GB RAM
elif storage_gb < 50:
instance = "r6g.2xlarge" # 64 GB RAM
else:
instance = "r6g.4xlarge" # 128 GB RAM; past ~100 GB, shard, quantize, or use DiskANN
return {
"storage_gb": storage_gb,
"instance": instance,
"monthly_compute": instance_pricing[instance],
"monthly_storage": storage_gb * 0.10, # EBS
"total_monthly": instance_pricing[instance] + storage_gb * 0.10
}

Decision: Managed vs Self-Hosted

FactorManagedSelf-Hosted
Ops overheadLowHigh
Cost at small scaleHigherLower
Cost at large scaleVariableOften lower
ControlLessFull
ComplianceDependsFull control
Vendor lock-inYesNo (if open source)

Verdict: Start with Serverless. Only self-host if you have >500M vectors or strict On-Prem/GPU-Local requirements. If the requirement is "data stays in our cloud account" rather than "we run it," BYOC (Pinecone, GA September 2026) now covers that without taking on operations.


Selection Framework

Decision Tree

text
Need < 100K vectors?
+-- Yes -> pgvector (if already using PostgreSQL)
| +-- Chroma (prototyping, self-hosted or embedded)
|
+-- No -> Need managed service?
+-- Yes -> Cloud-first?
| +-- Yes -> Pinecone (easiest)
| +-- No -> Qdrant Cloud or Zilliz
|
+-- No -> Need enterprise features?
+-- Yes -> Milvus on Kubernetes
+-- No -> Qdrant or Weaviate self-hosted

Evaluation Criteria

CriterionWeightQuestions to Ask
ScaleHighHow many vectors now? In 1 year?
LatencyHighWhat are p99 requirements?
Ops capacityHighCan we operate this?
CostMediumBudget constraints?
FeaturesMediumHybrid search? Multimodal?
Lock-in and exit riskMediumOpen source preferred? Is there a bulk export path, and what does a re-embed and re-index cost if the vendor exits?

Proof of Concept Checklist

Before committing to a vector database:

  • Load representative data volume
  • Benchmark query latency at target QPS, with your real filters and top-k depth (vendor numbers rarely include restrictive filters; Qdrant-FineWeb-10B, released September 2026 with exact top-1,000 ground truth over 10.07B vectors, is a public benchmark at production scale)
  • Test metadata filtering performance
  • Verify update/delete performance
  • Test failure recovery
  • Evaluate monitoring and observability
  • Calculate total cost of ownership

Interview Questions

Q: How would you choose between Pinecone and a self-hosted solution?

Strong answer: Decision depends on several factors:

Choose Pinecone when:

  • Team lacks ops capacity for stateful infrastructure
  • Need to move quickly (days not weeks)
  • Scale is moderate (under 100M vectors)
  • Budget allows managed service premium
  • Compliance allows cloud-vendor dependency

Choose self-hosted (Qdrant, Milvus) when:

  • Have Kubernetes and ops expertise
  • Cost sensitivity at scale
  • Need full control over the software, not just data location
  • Specific compliance requirements
  • Want to avoid vendor lock-in

Three 2026 changes narrow the gap: Pinecone BYOC (GA September 23) runs the data plane in your own cloud account with no inbound access from Pinecone, and Pinecone's BM25 full-text search (GA September 9) removes the "we need a separate keyword engine" argument, though you still fuse lexical and vector results client-side. And Pinecone's pg_pinecone extension (October 27) adds a route for Postgres shops: keep pgvector tables and move individual tables to a remote Pinecone engine only when they outgrow Postgres.

For most startups, I would start with Pinecone or Qdrant Cloud for velocity, then evaluate migration if costs become prohibitive at scale. The switching cost is the re-embed and re-index, not the API: query surfaces are similar, but in-database embedding features tie you to the vendor's model choice.

Q: Explain how HNSW works and when you would not use it.

Strong answer: HNSW builds a hierarchical graph of vectors:

How it works:

  1. Insert vectors as nodes in a multi-layer graph
  2. Higher layers have fewer nodes, larger jumps
  3. Search: start at top layer, greedily navigate to nearest neighbor
  4. Descend layers until bottom (all vectors)

Why it is good:

  • O(log n) query complexity
  • No training required
  • Supports real-time updates
  • Excellent recall/latency tradeoff

When not to use:

  • Very small datasets (<10K): brute force is fine
  • Extremely memory constrained: HNSW keeps every full-precision vector plus its graph links in RAM
  • Need exact search: HNSW is approximate
  • Heavy update workload with tight latency: updates can cause temporary degradation

Alternatives:

  • IVF-PQ for memory constraints
  • DiskANN for billion-scale with cost efficiency
  • Flat index for exact search
  • LSH for very high-dimensional sparse vectors

Q: When would you use a Disk-based index (like DiskANN) over a RAM-based index (HNSW)?

Strong answer: I would use a Disk-based index when the memory cost of the index exceeds the budget or the capacity of a single high-memory node. For example, a 100-million-vector index with 1,536 dimensions needs roughly 700 GB of RAM per replica for HNSW (614 GB is the float32 vectors alone). Using DiskANN, I can keep the full vectors and the graph on NVMe SSDs and only compressed codes in RAM, reducing the RAM requirement by 90-95% while maintaining sub-10ms query times. This represents a massive TCO (Total Cost of Ownership) reduction for any workload that can absorb a few extra milliseconds per query, which covers most RAG traffic, since the LLM call dominates end-to-end latency.

Q: Why is metadata filtering often the bottleneck in vector databases?

Strong answer: In naive vector search, we find the "Top K" nearest neighbors and THEN filter them by metadata (e.g., "only documents from 2024"). If the filter is very restrictive, we might find 0 results after filtering. Specialized databases now use Pre-Filtering with HNSW, traversing the graph but only considering nodes that satisfy the boolean metadata constraint. This is computationally expensive because it breaks the "short-circuit" logic of HNSW, requiring specialized bitmasks or hardware acceleration (SIMD) to keep latencies low.

Q: How do you handle multi-tenancy in a vector database?

Strong answer: Three main approaches:

1. Metadata filtering (most common):

python
results = db.search(
vector=query,
filter={"tenant_id": current_tenant}
)
  • Pros: Simple, single index
  • Cons: All tenants share resources, potential for bugs exposing data

2. Collection per tenant:

python
results = db.collection(f"tenant_{tenant_id}").search(vector=query)
  • Pros: Strong isolation, per-tenant scaling
  • Cons: Many collections, operational overhead

3. Namespace per tenant (Pinecone):

python
results = index.query(vector=query, namespace=tenant_id)
  • Pros: Isolation within single index
  • Cons: Vendor-specific

I would choose:

  • Metadata filtering for most cases (simple, cost-effective)
  • Separate collections for high-security requirements
  • Never post-filter (retrieve all, filter after) due to leakage risk

The hybrid-search catch: BM25 scores depend on corpus-wide IDF statistics. In a shared collection, one large tenant's vocabulary shifts every other tenant's keyword scores, and term statistics leak across the boundary. Use an engine with per-tenant IDF (Qdrant 1.19 added it for sparse and BM25 search) or give large tenants their own collection.


References


Previous: Embedding Models | Next: Hybrid Search

From the AI System Design Guide by Om Bharatiya · MIT License ·view source