Embeddings & semantic search

How Maidan generates embeddings, how models are stored, and how to switch embedding models in production (the migration / reindex story).

Providers

MAIDAN_EMBEDDING_PROVIDER selects the active provider (default hash-v1):

Value	Use	Notes
`hash-v1`	Dev / tests / offline	SHA-256-derived 1024-d pseudo-embedding. No network, deterministic. Not semantically meaningful — for plumbing, not relevance.
`openai-compatible`	Production	Any OpenAI-style `/embeddings` endpoint (OpenAI, Azure OpenAI, vLLM, text-embeddings-inference, Ollama, …).

`openai-compatible` configuration

Env	Required	Default	Meaning
`MAIDAN_EMBEDDING_ENDPOINT`	yes	—	Full URL of the embeddings endpoint.
`MAIDAN_EMBEDDING_MODEL`	yes	—	Model id sent as `model` (also the registry key — see below).
`MAIDAN_EMBEDDING_API_KEY`	no	—	Sent as `Authorization: Bearer …` when set.
`MAIDAN_EMBEDDING_DIM`	no	auto-detected	Output dimension. If unset, the server probes the endpoint once at boot to learn it.
`MAIDAN_EMBEDDING_TIMEOUT_SECS`	no	`15`	Per-request timeout.

Dimension auto-detect. When MAIDAN_EMBEDDING_DIM is unset, the provider issues one sentinel embed at startup and uses the returned vector length. This means a wrong model id or an unreachable endpoint fails at boot with a clear error, not silently on every message. Set MAIDAN_EMBEDDING_DIM explicitly to skip the probe (e.g. air-gapped boot, or to assert the expected dimension) — text-embedding-3-small is 1536, text-embedding-3-large is 3072, BGE/GTE small models are 384/768.

Live indexing, retry and repair

The live indexer embeds posted and edited messages in batches, off a bounded queue. A failed provider call or upsert is retried with exponential backoff while the worker holds the batch — the queue stops draining rather than growing. What still fails is picked up by the repair sweep, which embeds, on a timer, the newest live messages that have no embedding for the active model. Replicas share the work through a Postgres advisory lock. A full reindex is only needed for a model change.

Variable	Default	What it does
`MAIDAN_INDEXER_QUEUE_CAPACITY`	`1024`	Bound on queued messages; a full queue applies backpressure to the bus.
`MAIDAN_INDEXER_BATCH_SIZE`	`32`	Messages per provider call.
`MAIDAN_INDEXER_RETRIES`	`3`	Further attempts after a failed call or upsert (max 10).
`MAIDAN_INDEXER_RETRY_BASE_MS`	`250`	First retry delay; each doubles, capped at 10 s.
`MAIDAN_EMBED_REPAIR_INTERVAL_SECS`	`300`	How often the repair sweep runs; `0` disables it.
`MAIDAN_EMBED_REPAIR_BATCH`	`256`	Messages the sweep embeds per pass.

Metrics: maidan_indexer_embed_failed_total (gave up after retries), maidan_indexer_embed_retries_total, maidan_indexer_embed_repaired_total (recovered by the sweep).

Per-model table scheme

Each embedding model gets its own table and a row in the maidan_embedding_models registry (model, dimension, table_name). The table is maidan_emb_<slug> where <slug> is the model id with non-alnum characters folded to _ (text-embedding-3-small → maidan_emb_text_embedding_3_small).

Consequences:

Models coexist. Switching models does not destroy the old vectors; the old table stays queryable. You can run old + new side by side.
Dimension is pinned per model. The registry records the dimension on first registration. Re-registering the same model id with a different dimension is rejected (DimensionMismatch) — pick a new model id instead.
Queries target a model. Semantic search resolves the table for its embedding_model argument (the active provider's model is the default). A query against an unregistered model returns no hits rather than an error.

Startup registration

On boot the server calls Search::ensure_model for the active provider, which creates the per-model table + index and inserts the registry row if absent. So a freshly-configured model is queryable before the first message is written, and a DimensionMismatch surfaces in the startup logs. Registration is best-effort and non-fatal: if it fails, messaging still serves and the per-message write path retries ensure_model lazily.

Switching models (migration / reindex)

Choose a new model id. Use a distinct MAIDAN_EMBEDDING_MODEL (a different real model, or a suffix like text-embedding-3-small@v2 if you must re-embed under the same model with different params). Reusing an id with a changed dimension is rejected by design.
Configure and restart. Set the openai-compatible env vars and restart. Boot registers the new model's table (empty) and the live indexer begins embedding new messages under it immediately.
Backfill existing messages into the new model's table. Either:
- HTTP (operator): POST /operator/reindex-embeddings with {"workspace_id": "<uuid>"} (workspace-scoped, needs workspace:write) or {} (whole instance, needs operator:global). Returns a ReindexJob; poll GET /operator/reindex-embeddings/{job_id} for processed/failed. The job re-embeds using the server's active provider, in batches.
- CLI (offline / large): maidan reindex-embeddings --embedding-provider openai-compatible [--workspace-id <uuid>] with the same MAIDAN_EMBEDDING_* env. Runs against its own pool, so it won't contend with the live server's statement-timeout cap.
Verify, then optionally cut over reads. Semantic search uses the active model by default; pass embedding_model to target a specific table during validation. Once the new model is fully backfilled, it is the default — no read-side flag needed.
Clean up (optional). The old model's table can be dropped manually once you're confident; nothing references it after cutover.

Backfill runs on its own task/queue and never enters the live indexer's bounded queue, so a large-workspace reindex does not delay live indexing.

The live indexer is a tap projector of the event log. It verifies every backfill row on the per-workspace hash chain, projects only message_posted / message_edited / message_tombstoned, and drains history before applying live frames. A pruned gap or chain break fails closed (RebuildRequired on IndexerHandle); Lagged without a durable log is an error, not a warn-and-continue. Rebuild from the messages table via the existing reindex path — do not keep serving a gapped suffix.

HNSW index parameters (Postgres)

The HNSW build params (m, ef_construction) are applied when a model's table

index are first created and are fixed for that table — see Query-Tuning.md for the env vars. To change them you must rebuild: drop the model's table and reindex, or register a new model id. Query -time ef_search is tunable without a rebuild.