Migrating embedding models without breaking search

Embedding models are not permanent infrastructure. They get superseded, deprecated, and outperformed by newer releases at a pace the rest of your stack doesn’t match. The model that was state-of-the-art for your domain eighteen months ago may be producing meaningfully worse search results than what’s available today — and if you’re not periodically migrating, that gap compounds silently while your users wonder why search is getting worse.

The problem is that embedding model migration is not like deploying a new API version. Every document in your index was embedded with the old model. Switching to a new model without reindexing means vectors from two incompatible embedding spaces are living in the same collection. Distance computations between them are meaningless. You cannot migrate incrementally without a strategy — you have to be deliberate about the transition.

This post covers the full migration lifecycle: evaluating whether to migrate at all, running a dual-write period, background reindexing without downtime, validating the new model before cutover, and executing an atomic collection swap.

Should you migrate?

Embedding model migration has real cost and operational risk. Before starting, answer these questions honestly:

Is retrieval quality actually a problem? If your RAG pipeline has a measured retrieval precision at k issue — relevant documents are not being returned — and you’ve ruled out chunking strategy and query preprocessing as causes, then a better embedding model may help. If retrieval quality is acceptable, migration is speculative.

What does the new model actually improve? Read the model card and benchmarks critically. MTEB scores are useful but measure general retrieval across many domains. What matters is performance on text that looks like yours. A model that tops MTEB on Wikipedia-derived benchmarks may not outperform your current model on dense technical documentation or short support tickets.

What’s the embedding dimension and context window of the new model? A dimension change (e.g., 768 → 1536) means you can’t reuse your existing vector index schema. You need to provision new storage. A context window change affects whether long documents that were truncated by the old model will be handled differently — this can cause changes in chunking strategy, not just the embedding call.

What’s the cost difference? Larger embedding models cost more per token. At scale, embedding costs are non-trivial. A model with 5% better retrieval quality that costs 3x more may not be worth it.

If you’ve validated that migration is worthwhile, proceed. If you’re not sure, run the offline evaluation first (described below) before committing to the operational work.

Step 1: Offline evaluation before touching production

Evaluate the new model against your actual documents and queries before writing a line of migration code. The evaluation protocol:

1. Sample your document corpus. Take a representative sample — at minimum 1,000 documents, ideally stratified across content types if you have heterogeneous content.
2. Build a reference query set. Collect 100–200 real queries that users have submitted to your search. If you have relevance feedback (clicks, thumbs-up on results, user ratings), use that to label which documents are relevant for each query.
3. Embed the document sample with both models. Create two separate in-memory indexes (FAISS or similar, no production systems involved).
4. Run all reference queries against both indexes. Collect the top-10 results for each query from each model.
5. Score retrieval quality. Compute Recall@5, Precision@5, and MRR against your relevance labels. If you don’t have explicit relevance labels, use a reranker as a proxy judge: pass the retrieved results through a cross-encoder and compare the relevance scores across models.

This evaluation tells you whether the new model is actually better for your data, not just better on a benchmark. It also gives you a quantitative threshold for cutover: “we will cut over if the new model achieves ≥5% improvement in Recall@5.”

If the offline evaluation shows no improvement, stop here. You’ve saved yourself a complex migration.

Step 2: Dual-write for new documents

Once you’ve decided to migrate, the first production change is dual-write: every new document that enters your system gets embedded with both the old model and the new model and written to two separate vector collections.

old_collection: "docs_v1"   # old embedding model, all historical docs
new_collection: "docs_v2"   # new embedding model, only new docs so far

During dual-write, your search pipeline continues to read from docs_v1. The docs_v2 collection is write-only — it accumulates new documents but is not queried yet. This is safe because docs_v2 is incomplete (historical documents haven’t been reindexed yet), and searching an incomplete index produces worse results than the complete old index.

Dual-write has two costs to account for: the additional embedding API calls (roughly 2x your embedding cost during the transition period) and the storage for the second collection. Budget for both in advance.

How long does dual-write run?

Dual-write runs until you complete background reindexing of all historical documents and validate the new collection is ready for cutover. For most teams, this is days to weeks, depending on corpus size and your reindexing throughput budget.

Step 3: Background reindexing

Background reindexing is the operational heart of the migration. You’re re-embedding every document in the old collection using the new model and writing the result to docs_v2, while production traffic continues to flow normally against docs_v1.

Key implementation considerations:

Rate limiting the reindex job. Embedding APIs have rate limits. Your reindex job should consume only a fraction of your embedding API quota to avoid interfering with production traffic, which also uses the embedding API for query embedding. A conservative starting point: use 20% of your embedding API rate limit for the reindex job.

Checkpointing. The reindex job will take time and may fail partway through due to transient API errors, network issues, or restarts. Store a checkpoint after each batch (e.g., every 1,000 documents) so a restart resumes from the last checkpoint rather than starting over.

Batch sizes. Most embedding APIs support batching multiple texts in a single request. Use the maximum batch size the API supports. For a corpus of 500,000 documents, the difference between batch size 1 and batch size 100 is the difference between 500,000 API calls and 5,000 API calls.

Monitoring progress. Track: documents reindexed, documents remaining, current throughput (documents/minute), estimated time to completion, and error rate. Error rate should be near zero — repeated errors for specific documents usually indicate encoding issues (binary content accidentally in your corpus, excessively long documents that exceed the model’s context window after chunking).

A simple progress tracking schema:

CREATE TABLE reindex_progress (
  collection_name TEXT PRIMARY KEY,
  total_docs      INTEGER,
  indexed_docs    INTEGER,
  last_doc_id     TEXT,
  started_at      TIMESTAMP,
  updated_at      TIMESTAMP,
  status          TEXT  -- 'running', 'paused', 'complete', 'failed'
);

Idempotency. If a document is reindexed twice (because of a checkpoint race or retry), the second write should overwrite the first cleanly. Upsert semantics in your vector store, keyed on document ID, ensures idempotency.

Handling documents that change during reindexing

Your documents don’t freeze during reindexing. Documents get updated, deleted, and created while the reindex job runs. The dual-write handles new documents — they go to both collections. Updated and deleted documents in docs_v1 need to propagate to docs_v2.

The cleanest approach: extend your existing document update and delete handlers to write to both collections during the transition period. This is the same dual-write pattern applied to mutations. If you already have a change feed or event stream for document mutations, consuming it in your reindex controller is straightforward.

Step 4: Shadow querying for online validation

Before cutting over, run the new collection in shadow mode: for a fraction of your production queries, execute the search against both docs_v1 and docs_v2 in parallel. Don’t return docs_v2 results to users — just collect them alongside the docs_v1 results and compute quality metrics.

Metrics to compare in shadow mode:

• Result overlap at k: what fraction of the top-5 results from docs_v2 also appear in the top-5 from docs_v1? Expect ~60-80% overlap — complete divergence suggests a problem, but some divergence is expected and desirable.
• Reranker scores: pass both result sets through your cross-encoder reranker and compare the distribution of relevance scores. Are docs_v2 results scoring higher?
• Null result rate: does docs_v2 return zero results for any queries that docs_v1 answers? This could indicate incomplete reindexing.

Shadow querying is your final pre-cutover validation. It uses real production queries — not a sampled eval set — against the complete new index. If shadow metrics confirm improvement, proceed to cutover.

Step 5: Atomic collection cutover

Cutover is the moment you switch query traffic from docs_v1 to docs_v2. The goal is to make this change atomic from the application’s perspective: one instant it’s querying the old collection, the next it’s querying the new one, with no mixed state.

The implementation pattern depends on how your application references the collection:

Configuration-driven cutover: your search service reads its target collection name from a configuration store (environment variable, feature flag, or config service). To cut over, update the configuration value. The change propagates to all instances within seconds to minutes, depending on your config propagation latency. Zero code deploy required.

Feature flag cutover: use a feature flag to control which collection is queried. This gives you percentage-based rollout (send 5% of traffic to docs_v2, monitor, then increase to 25%, 50%, 100%) and instant rollback (set the flag back to 0%).

Percentage rollout is preferable to instant cutover for high-traffic production systems. It lets you catch unexpected quality regressions or performance problems at small blast radius before they affect all users.

Rollback plan

Before cutting over, define your rollback condition explicitly: “if retrieval quality metrics (measured by reranker scores on production traffic) drop by more than 10% within one hour of cutover, we roll back.” A rollback is a single configuration change — point traffic back to docs_v1, which has been running in parallel and remains fully valid.

Keep docs_v1 alive and receiving dual-writes for at least 72 hours after successful cutover. This is your rollback window. After 72 hours of stable metrics on docs_v2, you can decommission docs_v1 and the dual-write path.

Step 6: Decommission and cleanup

After a stable cutover period:

1. Remove the dual-write path — new and updated documents now go only to docs_v2
2. Remove the shadow query path if you implemented one
3. Delete the docs_v1 collection (coordinate with your vector store to release storage)
4. Update your embedding client to use only the new model
5. Document the migration: new model version, date of cutover, and the quality improvement observed

Documentation matters here because you will do this again. Embedding model migration is a recurring operational task, not a one-time event. Having a runbook from the previous migration makes the next one faster.

Where ManyLayers fits

ManyLayers Workspace manages connector-sourced document pipelines, chunking configurations, and embedding model selection per-connector. When you’re ready to migrate, the dual-write and collection cutover patterns described here can be configured through the connector pipeline settings, with background reindexing jobs managed by the platform rather than custom scripts you maintain. Gateway handles the query-side routing, so shadow querying and percentage cutover are routing-layer changes rather than application code changes. For teams running ManyLayers on their own infrastructure via the Deploy module, embedding models run on your own hardware — migration cost is compute rather than API fees.

Embedding model migration is operational work, but it doesn’t have to be risky work. Evaluate before committing, run dual-write as your safety net, validate with production traffic before cutover, and keep rollback trivially fast. The teams that do this well run migrations quarterly and stay ahead of the model curve. The teams that avoid it fall further behind with every release cycle.