"Most Of Your Vectors Are Steerage. Why Are They In First Class?"

Source: dev.to · Published: 2026-08-01

<p>I was on a call last month with a startup CTO who had just gotten their AWS bill. They had built a beautiful RAG application: semantic search, conversational AI, the works. Their vector index was humming along with about 50 million embeddings. Then they hit product-market fit.</p>

<p>Within six weeks, they scaled to 500 million vectors. Their monthly infrastructure costs went from $2,000 to $20,000. The real kicker? When we looked at the access patterns, over 80% of those vectors were queried less than once a week. They were paying hot-storage prices for data that was, by any honest measure, cold.</p>

<p>The standard advice here is "just use a cheaper vector database." The more interesting question is: why are you storing all your vectors at the same temperature in the first place?</p>

<h2> The Cost-Recall-Latency Triangle </h2>

<p>Vector search forces a three-way tradeoff. You can optimize for cost, recall, and latency, but you only get to pick two. Want high recall and low latency? That costs money (in-memory HNSW graphs with full-precision vectors eating RAM). Want high recall at low cost? Latency goes up. Want cheap and fast? Recall suffers.</p>

<p>Most teams pick a single point on this triangle and apply it uniformly to every vector in their index. That decision made sense when vector databases offered a single storage tier. It makes the same amount of sense as storing your entire filesystem on NVMe SSDs because some files need fast access.</p>

<p>The conventional wisdom says you pick your point on the triangle and live with it. But the conventional wisdom was written before vector storage got interesting. The better approach: tier your vectors the same way you already tier your storage. Different access patterns deserve different economics. The same embedding that costs $0.12/month in RAM might cost $0.004/month on disk and $0.0002/month in object storage. When you have 500 million of them, those decimals matter.</p>

<h2> The Hot Tier: In-Memory HNSW and Exact k-NN </h2>

<p>For vectors that get hit constantly (your user-facing search, your real-time recommendations, anything in the critical path of a page load), Amazon OpenSearch Service stores HNSW graphs entirely in native memory. On a single r6g.8xlarge node with 113 million vectors at 1,024 dimensions (<a href="https://opensearch.org/blog/Reduce-Cost-with-Disk-based-Vector-Search/" rel="noopener noreferrer">source</a>), this configuration delivered P90 latency of 25 ms and recall of 0.95 at 300 queries per second. Throughput scales linearly with replica shards.</p>

<p>There is also a case the HNSW literature tends to gloss over: exact k-NN. Exact k-NN compares every candidate by brute force rather than navigating a graph. When your query includes filters (a tenant ID, a category, a date range) that reduce the candidate set below roughly 100,000 vectors, exact k-NN outperforms approximate search. The brute-force scan finishes faster than an HNSW traversal at that scale, uses less RAM (no graph to build or maintain, no <code>ef_construction</code> tuning, no <code>ef_search</code> parameter), and returns perfect recall.</p>

<p>The hot tier is the right home for vectors with high query frequency, latency requirements under 50 ms, or both.</p>

<p>Quantization methods (FP16, binary quantization, product quantization) give you further control over the RAM-versus-recall tradeoff at each tier. That topic deserves its own article.</p>

<h2> The Warm Tier: On-Disk Mode (Where Most of Your Vectors Should Live) </h2>

<p>Here is the part of this story that deserves more attention than it gets.</p>

<p>OpenSearch Service's on-disk mode keeps a quantized navigation graph in RAM and stores the full-precision vectors on SSD. The default is binary quantization at 32× compression (each float dimension collapses to a single bit), but you can choose 2×, 4×, 8×, or 16× to trade more RAM for higher recall. When a query arrives, OpenSearch walks the compressed graph in memory, identifies candidates, then rescores against the full vectors on disk.</p>

<p>On the same 113M-vector benchmark (<a href="https://opensearch.org/blog/Reduce-Cost-with-Disk-based-Vector-Search/" rel="noopener noreferrer">source</a>), on-disk mode at 8× compression delivered P90 latency of 96 ms with 0.98 recall. At 32×, latency was 104 ms with 0.94 recall. Compare that to the in-memory tier's 25 ms. You trade 70-80 ms of latency for a memory reduction approaching two thirds: 1 million vectors at 256 dimensions drop from 1.31 GB (FP32) to 0.18 GB for the navigation graph, plus the <a href="https://opensearch.org/blog/do-more-with-less-save-up-to-3x-on-storage-with-derived-vector-source/" rel="noopener noreferrer">derived source</a> optimization saves up to two-thirds more by deduplicating vector storage across replicas.</p>

<p>Most workloads belong here. In a RAG pipeline where LLM generation takes 2-3 seconds, 100 ms for vector retrieval is invisible. The only workloads that need 25 ms retrieval are the ones in the critical path of a user-facing page load with tight SLAs.</p>

<p>To enable on-disk mode,</p>

<h2> The Cold Tier: S3 Vectors for Archive Scale </h2>

<p>For vectors that are accessed rarely, Amazon S3 Vectors provides native vector storage and search at S3 economics. You specify <code>"engine": "s3_vector"</code> in your index mapping on OpenSearch 2.19+ with <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/or1.html" rel="noopener noreferrer">O-series instances</a>, and OpenSearch manages the rest. OpenSearch offloads vector data to S3, keeps metadata on the cluster for filtering, and routes the k-NN portion of each query to S3 transparently. From the application's perspective, you query the same <code>_search</code> API you use for in-memory and on-disk indexes.</p>

<p>Response times are sub-second (500-800 ms typical), and storage costs drop by up to 70% compared to in-memory indexes. S3 Vectors uses pay-per-query pricing, so you never pay for idle capacity. That model is ideal for large datasets with low to moderate query traffic (up to thousands of queries per day). For workloads with sustained high throughput (hundreds of queries per second), on-disk mode is the better complement because its costs are capacity-based. The two tiers work together. S3 Vectors handles large, infrequently accessed data, and on-disk mode handles data with steady query traffic.</p>

<p>S3 Vectors also works as a standalone vector store outside the OpenSearch integration. For teams using S3 Vectors independently, a one-click export moves data into OpenSearch Serverless, where queries run at sub-200ms latency. This lets teams take advantage of OpenSearch Serverless throughput when access patterns shift and cold data needs to become hot.</p>

<h2> The Decision Framework </h2>

<p>Where each vector lives should follow from two questions: how often does the vector get queried, and what latency does the caller need?</p>

<p>Vectors that get hit constantly and need sub-30ms responses belong in memory with in-memory HNSW on OpenSearch Service (FP16 recommended). Exact k-NN is worth testing when query filters narrow candidates below 100,000, which is common in multi-tenant applications. Vectors with steady query traffic where 100ms latency is acceptable belong on disk with 32× compression. Think internal search tools, batch RAG pipelines, analytics, product catalogs. On-disk mode is the default tier for most production workloads at scale. Vectors that are large in volume, accessed at low to moderate frequency, and tolerant of 500-800ms latency belong in S3 Vectors with the <code>"engine": "s3_vector"</code> integration: historical archives, compliance retention, seasonal catalogs outside their season. If access patterns change, a one-click export promotes cold data to OpenSearch Serverless.</p>

<p>The beauty of this tiering is that the calling application does not need to know about it. Every tier exposes the same OpenSearch <code>_search</code> API. The application sends a query. OpenSearch routes the query to the right storage layer based on which index you target. Migrations between tiers are operational decisions, not application code changes.</p>

<h2> What Happened to the $20,000 Bill </h2>

<p>The CTO from my opening story did the access-pattern audit. About 15% of their 500 million vectors were hot: they powered the real-time search and needed in-memory speed. Around 60% were warm: knowledge-base vectors for their RAG pipeline where 100 ms was invisible inside a 3-second LLM call. The remaining 25% were seasonal product embeddings from marketing campaigns that had ended months ago.</p>

<p>They moved the warm vectors to on-disk mode and the seasonal vectors to S3 Vectors. Their monthly bill dropped by roughly two-thirds without changing a single query from the application layer. The application still called the same <code>_search</code> API. The tiering was invisible to the calling code.</p>

<p>If you are running a vector workload today, the one thing worth doing this week is pulling up your access-pattern metrics. Look at which embeddings get queried at high frequency versus which ones are sitting idle. The answer tells you exactly which tier each vector belongs in. The math tends to be obvious once you look, and the savings tend to surprise people who assumed all their vectors needed to be hot.</p>

<p>What does your vector access-pattern distribution look like? I am curious whether others are seeing the same 80/20 split between rarely-touched and actively-queried embeddings.</p>