I heard about a team that developed a RAG demo for internal document search. Several hundred PDFs, a handful of people testing it. The monthly cost was $340. Everyone was so excited, it was signed off by leadership and shipped to production.

The bill was $61,000 per month six months later.

No one did anything out of the ordinary. No one has multiplied the number of documents by a factor of 100 in a single night. What happened is what happens with almost every RAG system after it moves out of the demo phase: The cost math that seemed to work at small scale doesn't work when real traffic and a growing corpus appear. The part that surprises most teams is not the LLM bill — everybody plans for that one. It's the vector database lying beneath it.

The bill nobody models correctly

Most teams have a pretty good estimate of their embedding costs, and it's the easiest number to estimate before launch. It is the most deceptive one to concentrate on, too. A couple of cents is the cost of embedding a million tokens. Low cost, low risk, low remuneration.

In real production systems, the vector database consumes around 40-50% of the overall application cost, trailing only LLM calls, if not even then. That's a difference of $340 versus $61,000.

Why the bill grows faster than your users do

A RAG system is not scalable with traffic. It scales with traffic times documents times lookups-per-request — and all three tend to grow at the same time, without anyone realizing it.

AI agents tend to perform 5 to 15 vector lookups per user request, as they link together multiple retrieval steps to finally provide an answer. A system that appears to be a million queries per minute to the user is really ten or fifteen million queries to the database. Typically, vector databases charge for that second number, not the first, and that's where budgets go astray.

Storage adds up the same way. A single embedding can easily be a few thousand bytes. Multiply that by a few million documents, add metadata, multiple embedding model versions, and document history, and most teams end up storing 10 to 50 times more vector data than they originally estimated.

The real cost driver: memory

Memory is the dominant cost in a real deployment of a vector database. For example, in terms of actual dollars spent, memory represents about 85-90 percent of the total cost. Depending on which index type you choose for your vectors, you will be using four times as much memory as with another type of index for the exact same set of data. As well, there are many different types of indexes available for use with graphs. However, the HNSW, or Hyperbolic Nearest Neighbor Search, graph based index used by most groups is also going to require a lot more memory than a compressed version. This means that switching from an uncompressed index to a compressed version can reduce your memory costs by approximately seventy percent, while at worst resulting in a few percent decrease in search accuracy.

Most teams never make that tradeoff on purpose. They pick the default because it's fast, ship it, and only notice the cost six months later.

Managed vs. self-hosted is a cost cliff, not a preference

As you grow in scale, the costs of the same technology start to look different. An example was a company that paid about $4,200 per month (for an example) on a managed service and then saved approximately $48,000 per year by moving their load to self-hosting; it cost them less than $200/month. The trend seems to hold up: A managed service that works well at a million vectors will likely be three to five times more expensive than a self-hosted alternative when working with 100-million vectors.

The extras nobody budgets for

There are a few other expenses that creep in once the architecture is set:

• Filtered search (find the nearest match where category \= X) is much more expensive than plain similarity search, as it requires scanning and filtering during the search.
• But the second pass over your top results to re-rank them, really does improve quality — but only when you add it on to fix a quality complaint, which is a cost most teams never anticipate.
• The cost of serverless “scale to zero” pricing may appear to be free when it is idle, but during the “cold start” the first query may incur as much as two seconds of latency. The savings are, in any case, offset by the cost of always-on capacity in most production systems.

What actually helps

This doesn't involve any rebuilding of your architecture. There are a few things that always make a difference:

• Switch to a compressed index. It is the biggest lever because the majority of the cost resides in memory.
• Add semantic caching. It can save a significant portion of your most valuable cost, such as hit rates of 20-40% and repetitive workloads such as customer support hitting 50-70%.
• Don't assume that it's not worth the engineering effort, actually run the self-hosted math after a few tens of millions of vectors.

The takeaway

RAG is not the issue and vector databases are not a pitfall. Their pricing models are based on abstract concepts in a planning meeting, like dimensions, lookups, index types; and become very real on the monthly invoice.

The team that had $340 demo didn't do anything wrong early on. They just never ran into the case where a few hundred documents turn into a few million, and one query turns into ten lookups. It is here that nearly all RAG cost surprises actually occur; not in the part that is budgeted for, but in the part that only appears when real usage does.