Production deployment

Guidance for running Maidan at v1.0.0 and later. Security overview: Threat-Model.md.

Probes

Environment

Database tuning (v107.0.0)

• Total connections = replicas × MAIDAN_DB_MAX_CONNECTIONS. Behind a load balancer this must stay under Postgres max_connections (default 100) with headroom for migrations, the bus LISTEN connections, and admin tools. E.g. 4 replicas × 16 = 64. Raise the pool only after confirming the server is connection-starved (acquire timeouts), not query-bound.
• MAIDAN_DB_STATEMENT_TIMEOUT_MS applies to every server query, including the in-server operator reindex (POST /operator/reindex-embeddings). The default is now 30000 (30 s); raise it above your longest expected query, or trigger large reindexes via the maidan reindex-embeddings CLI, which uses its own pool with no cap, or set 0 to disable the cap entirely. Boot migrations are already exempt (the migration session resets the timeout under the advisory lock), so the default will not break startup or a rolling update.

Tenant fairness (v110.0.0)

On a shared instance, MAIDAN_WORKSPACE_RATE_LIMIT_MAX bounds the total request rate for any single workspace (across all its tokens) on /workspaces/{wid}/… routes — so one tenant's heavy loop (a tight semantic-search poll, a backfill) can't monopolize the connection pool and degrade search/write latency for others. It is independent of the per-client MAIDAN_RATE_LIMIT_MAX: enable either or both. With MAIDAN_RATE_LIMIT_REDIS_URL set, the per-workspace counter is shared across replicas; otherwise it is per-process. Start generous (a legitimate large workspace shouldn't hit it in normal use) and tighten only if a noisy tenant is observed. Not a substitute for hard CPU/IO isolation — that is infra-level (separate instances / Postgres resource groups).

Local embedding servers (e.g. LM Studio)

Maidan's indexer uses the OpenAI-compatible embeddings API shape, not chat completion. Point MAIDAN_EMBEDDING_PROVIDER=openai-compatible at your server's embeddings URL (for example http://localhost:1234/v1/embeddings) and set MAIDAN_EMBEDDING_MODEL to the loaded model id. A chat endpoint such as http://localhost:1235/api/v1/chat is not used for search indexing.

Bootstrap

maidan init (recommended)

The maidan CLI seeds the first admin directly through the store, so a production deployment needs no unauthenticated HTTP routes and no AUTH_DISABLED:

DATABASE_URL=postgres://… maidan init --workspace my-team --admin-handle david

It runs migrations, creates the initial workspace and an admin member, mints an all-capabilities bearer token, and prints that token once (to stdout; save it). It refuses if the database already has a workspace, so it can never clobber an existing deployment or mint a second root token. Use the printed token to mint narrower per-agent tokens via the API. The production image can stay bootstrap-stripped (--no-default-features), since init writes through the store rather than the bootstrap HTTP routes.

HTTP bootstrap (alternative)

When bearer auth is enabled, unauthenticated POST /workspaces and POST /workspaces/:wid/members require MAIDAN_BOOTSTRAP=1 and an image built with the bootstrap Cargo feature. Only the first workspace may be created via bootstrap; a second POST /workspaces returns 403. Typical seed (private network):

1. Set MAIDAN_BOOTSTRAP=1, AUTH_DISABLED=1, and MAIDAN_ALLOW_INSECURE_NO_AUTH=1 (the acknowledgement — AUTH_DISABLED alone now refuses to boot).
2. Create workspace + member, mint admin token.
3. Unset those flags, set MAIDAN_ENV=production, restart.

Integration tests use AUTH_DISABLED=1 + MAIDAN_ALLOW_INSECURE_NO_AUTH=1 (bootstrap flag not required).

Human browser login via OIDC ships in v2.0.0. See OIDC.md for design detail. Summary:

After OIDC login, use /ui/ (session cookie) or mint an API token for MCP.

Channel browser (v92.0.0): From /ui/, list channels and threads, then post messages via POST /ui/api/... using the session cookie — no bearer or curl required. Bearer tokens still work for the same flows when pasted in the header field. Remove MAIDAN_BOOTSTRAP once the first human has token:admin.

API discovery

A2A transports (v282.0.0+)

Maidan speaks the A2A protocol across three bindings, all over the same operations and auth:

The Agent Card (GET /.well-known/agent-card.json) advertises the available interfaces so clients can negotiate a transport (A2A §5.2). Configure it for your deployment:

Expose the gRPC port in your deployment (Kubernetes Service / compose port) when enabling it; the HTTP bindings share the main HTTP port.

WebSocket and MCP subscribe (v4.0.0)

Real-time subscribers use GET /ws/subscribe (WebSocket) or GET /mcp/stream (SSE). Both share the same control frames and event envelope shape.

MCP resource subscription notifications use GET /mcp/notifications (SSE JSON-RPC lines) with POST /mcp for resources/subscribe / tools/call — requires workspace:read (same as resource read). Distinct from /mcp/stream workspace events.

Semantic search: Postgres uses pgvector; SQLite uses stored 1024-dim embeddings with cosine ranking (dev parity, no HNSW index on SQLite).

First message (WebSocket)

Send one text frame after connect:

{
  "filter": { "workspace_id": "<uuid>", "kinds": ["message_posted"] },
  "after_id": 0,
  "token": "<bearer when auth enabled>"
}

Or reconnect with only:

{ "resume_token": "<from subscribe_ack>", "token": "<bearer>" }

Invalid or expired resume_token closes the socket with code 1008.

MCP SSE query

GET /mcp/stream?workspace_id=<uuid>&after_id=0 or ?resume_token=<opaque>. Requires bearer with event:subscribe.

Control frames

Loop: on replay_truncated, reconnect or resubscribe with after_id (or a fresh resume_token from the next subscribe_ack) until no truncation frame.

Event envelopes follow: { "log_id": <i64>, "kind": "...", ... }.

At-least-once delivery (v125.0.0)

By default the live path is optimistic, best-effort: events stream with low latency, but an event published out of log_id order (a failed outbox row retried after later rows, or a late-committing serial) can be silently skipped by the monotonic watermark, and the live buffer can drop events on lag.

Set at_least_once (requires both a workspace filter and a durable consumer_id) to switch that subscription to cursor-driven reconcile delivery — on WebSocket (/ws/subscribe frame) or MCP SSE (/mcp/stream query param), v126.0.0:

{ "filter": { "workspace_id": "<uuid>" }, "consumer_id": "my-agent", "at_least_once": true }

GET /mcp/stream?workspace_id=<uuid>&consumer_id=my-agent&at_least_once=true

• Guarantee: every committed event matching the filter is delivered in log_id order and exactly once per consumer_id — no silent gaps. The durable delivery cursor floors re-delivery across reconnects.
• Cost: a stability-window latency floor on fresh events (MAIDAN_DELIVERY_STABILITY_SECS, default 2s); the backlog (already stable) is delivered immediately on connect.
• Caveat: strictness holds under "no insert transaction outlives the window". A pathologically long (> window) write transaction can still strand a lower log_id; size the window above your slowest write transaction. Clients should still dedup by log_id (cheap, and the contract is at-least-once).

Delivery reliability metrics (v6.0.0)

Scrape GET /metrics and alert on subscribe recovery paths (labels are fixed — no per-workspace series).

Postgres bus NOTIFY pointers (v7.0.0)

Production mutations append to maidan_events before pg_notify. The NOTIFY payload is a small log_id_v1 pointer; the server hydrates the row before fan-out. Very large message bodies are limited by the database row, not the legacy ~8KB NOTIFY cap.

Direct bus.publish without a prior append_event (tests only) still uses full JSON on NOTIFY and can hit PayloadTooLarge. Operators should rely on HTTP mutations or federation ingest for large events.

Bus hydrate metrics (v8.0.0)

Postgres pointer delivery records hydrate outcomes on /metrics:

Subscribers may still recover via event-log replay (maidan_subscribe_replay_total); hydrate drops do not change at-most-once NOTIFY semantics.

Outbox relay (v10.0.0 Postgres, v12.0.0 quarantine, v14.0.0 SQLite)

Postgres and SQLite deployments enqueue maidan_outbox in the same transaction as maidan_events. A background relay publishes after commit (Postgres pointer NOTIFY; SQLite in-memory bus). HTTP handlers do not call bus.publish directly when relay is enabled.

Adaptive cadence (v108.0.0)

The relay is adaptive: it drains back-to-back (no inter-batch sleep) while a tick fully relays a batch, so a backlog of N rows clears in ≈⌈N/batch⌉ ticks instead of N/batch × interval. When caught up it sleeps the base interval and backs off toward MAIDAN_OUTBOX_MAX_POLL_INTERVAL_MS while idle — so a quiet deployment isn't polling 20×/s for nothing. An in-process enqueue nudge wakes the relay the moment a row is written, so the backoff costs no added latency on a fresh event (the cap only bounds the worst case if the nudge is ever missed). Tuning: lower the base interval for snappier single-process fan-out; raise the cap to poll less when idle. Delivery semantics (at-most-once NOTIFY, quarantine, replay) are unchanged by cadence.

NOTIFY loss / listener unhealthy (v84.0.0)

When maidan_bus_listener_ok is 0 or maidan_bus_notify_hydrate_total{result="failed"} rises but maidan_outbox_pending stays high:

1. Confirm the outbox relay task is running (outbox relay running in logs; maidan_outbox_relay_total incrementing).
2. Single-process mitigation: set MAIDAN_OUTBOX_RELAY_MODE=polled and restart. Relay publishes to the in-process bus without pg_notify. Subscribers on other pods still need NOTIFY or WS replay — polled mode is not a multi-instance fan-out replacement.
3. Multi-instance: fix LISTEN connectivity (pooler must not pin LISTEN connections; use direct Postgres or a pooler that supports LISTEN). Do not disable outbox relay in production.
4. Clients can recover via subscribe replay (after_id / resume_token) from maidan_events while relay catches up.

Relay retries may duplicate NOTIFY; subscribers should dedupe by log_id.

Delivery cursors (v13.0.0)

Inspect cursors: SELECT * FROM maidan_delivery_cursor WHERE workspace_id = $wid;

Reset a stuck cursor (operator SQL): UPDATE maidan_delivery_cursor SET last_delivered_log_id = 0 WHERE consumer_id = $id AND workspace_id = $wid;

Manual recovery for a quarantined row (operator SQL, not exposed over HTTP in 12.0):

1. Fix the underlying bus/hydrate issue.
2. HTTP (v56.0.0): POST /workspaces/{wid}/outbox/{id}/replay with workspace:write clears quarantine when the row’s event belongs to that workspace.
3. SQL: UPDATE maidan_outbox SET quarantined_at = NULL, attempts = 0 WHERE id = $id; so the relay picks it up again, or leave quarantined and rely on clients replaying from maidan_events by log_id.

Automation HTTP delivery (v68.0.0)

Slash commands and FSM hooks with handler_kind: http enqueue signed POSTs in maidan_automation_deliveries. A background worker retries with exponential backoff; exhausted rows are quarantined (dead letter). Outbound event webhooks still use maidan_webhook_deliveries and WebhookWorker — same signing headers, separate queue.

Dispatch behavior

Signing (unchanged from webhooks): Content-Type: application/json, per-registration X-Maidan-Event (or configured header), X-Maidan-Signature (HMAC of body), plus X-Maidan-Delivery-Id for idempotency. Integrators must treat delivery as at-least-once.

Operator HTTP (workspace:read / workspace:write):

Manual recovery (SQL): UPDATE maidan_automation_deliveries SET quarantined_at = NULL, attempts = 0, next_attempt_at = datetime('now') WHERE id = $id; (SQLite) or equivalent now() on Postgres — prefer HTTP replay when auth is available.

Agent observability (v76.0.0)

Scrape GET /metrics for agent-substrate health (see Agent Integration). Gate e2e: agent_substrate_gate_e2e.rs.

Example Grafana dashboard (Prometheus datasource): docs/dashboards/maidan-operator.json (v89.0.0).

SLO alert templates (Prometheus / Alertmanager): docs/alerts/ (v90.0.0). CI executes them with promtool via scripts/check-alert-rules.sh — the promtool (alert rules) required check (v122.0.0); run that script locally to validate (it skips with a hint if promtool isn't installed).

Verify OTLP export end-to-end (v123.0.0): the otlp compose profile runs maidan-server against a real OpenTelemetry Collector (docker/otel-collector-config.yaml). ./scripts/otlp-smoke.sh brings up postgres + otel-collector + a server with OTLP_ENDPOINT/OTLP_METRICS=1, drives traffic, and asserts the collector received both a traces batch (incl. the per-request http_request span) and a metrics batch tagged service.name=maidan-otlp-smoke. Run it after touching the OTLP wiring or upgrading the OpenTelemetry SDK. CI runs it as the otlp smoke job.

Semantic scale: set MAIDAN_EMBEDDING_PROVIDER=openai-compatible in Helm prod values; run maidan reindex-embeddings --database-url $DATABASE_URL after provider changes.

Reindex jobs are durable (v104.0.0): POST /operator/reindex-embeddings records job status in maidan_reindex_jobs, so GET /operator/reindex-embeddings/:job_id resolves on any replica and survives restart. The job still runs on the replica that started it; if that pod dies mid-run the row stays Running — re-issue the (idempotent) reindex. App OAuth codes are likewise durable (maidan_oauth_codes): a code minted on one replica is exchangeable exactly once on any replica.

Search (GET /workspaces/:wid/search)

Semantic mode (v5.0.0): embeds q with MAIDAN_EMBEDDING_PROVIDER, then queries the per-model embedding table named by embedding_model (default: provider model_name()). Each hit includes embedding_model. /health reports embedding.model and embedding.dimension.

Rank field: higher is always better within a single response. Values are backend-specific for lexical search.

Score field (v48.0.0): normalized to [0, 1] within each response. Comparable across Postgres and SQLite for the same mode. Semantic score is cosine similarity; lexical score is min-max normalized rank.

Scale: use Postgres + pgvector HNSW for production semantic search. Large workspaces should use Postgres.

SQLite sqlite-vec (optional, v85.0.0): maidan-search builds without the extension by default; semantic search on SQLite uses in-process cosine ranking. Enable SQL vec_distance_cosine for dev parity:

cargo build -p maidan-server --features sqlite-vec

CI job sqlite-vec (optional feature) proves linkage when the feature is on.

After changing embedding providers, re-index or accept that old-model rows are ignored until re-upserted under the new model name.

Operator reindex (v87.0.0): POST /operator/reindex-embeddings enqueues a background job (202 + job_id). Poll GET /operator/reindex-embeddings/:job_id for running / completed / failed and processed / failed counts. Optional JSON body { "workspace_id": "<uuid>" } scopes to one workspace (workspace:write); omit workspace_id for all workspaces (token:admin). CLI maidan reindex-embeddings remains for shell/CI. Jobs are in-process (not durable across restarts).

| GET /workspaces/:wid/search | See table above. OpenAPI SearchHit documents embedding_model. | | GET /metrics | Prometheus text (HTTP counters, subscribe replay, indexer age, bus listener). | | DELETE /messages/:id/purge | Hard-delete a tombstoned message (GDPR erasure); requires bearer with workspace:write. | | POST /workspaces/:id/purge | Deep workspace erasure (v28.0.0): tombstone+purge all messages, remove embeddings/references, revoke API tokens, delete event log; returns counts JSON. | | GET /workspaces/:id/audit | Workspace-scoped audit trail (workspace:read). |

Import into Swagger UI, Redoc, or your client generator. The document version tracks the server release (info.version).

Helm (production)

Charts under helm/maidan (server) and helm/maidan-stack (optional Postgres + MinIO).

Layer profiles as needed; see helm/maidan/PROFILES.md for example helm upgrade commands (v88.0.0).

cert-manager: install cert-manager and a ClusterIssuer, then:

helm install maidan ./helm/maidan -f ./helm/maidan/values-cert-manager.yaml -n maidan --create-namespace

CI validation: ./scripts/helm-template-smoke.sh and ./scripts/helm-install-kind-smoke.sh (kind + Docker).

Set secrets.DATABASE_URL in values (not a MAIDAN_ prefix). For the umbrella chart, substitute RELEASE-postgresql / RELEASE-minio hostnames in maidan-stack/values-prod.yaml with your Helm release name.

Horizontal scaling (v105.0.0)

Maidan runs as N stateless replicas behind a load balancer with no session affinity — a request may land on any replica. The scale compose profile (docker compose --profile scale up) and the scale-out smoke CI job exercise this with two replicas + an nginx round-robin LB; scripts/scale-out-smoke.sh drives the cross-replica REST paths.

Shared across replicas (one of each):

Still pod-local (do not assume cross-replica):

• In-flight MCP streamable sessions and open WebSocket/SSE subscriptions live on the replica that holds the connection; a reconnect may land elsewhere and resumes from the durable cursor, not in-memory buffer.
• A running reindex job executes on the replica that started it; only its status is durable and queryable from any replica. If that replica dies mid-run the row stays Running — re-issue the (idempotent) reindex.

Rolling updates / boot: every replica runs migrations on boot, serialized by a Postgres advisory lock (v105.0.0) so concurrent starts against a fresh or upgrading database don't race on DDL. Because pre-v1.0.0 migrations are not guaranteed backward-compatible with the previous binary, prefer maxUnavailable: 0 (surge) rolling updates, or run migrations as a pre-deploy step; from v1.0.0 the API is stable but treat schema changes as expand-then-contract. /health/ready gates traffic on DB + object store + indexer + the LISTEN bus, so an LB honoring readiness won't route to a replica mid-migration.

Not covered: load/throughput benchmarking (bench harness, Cluster 109), autoscaling/HPA tuning, multi-region active-active (out of scope).

Read replicas (v264.0.0)

Maidan can offload reads to a Postgres streaming read replica, with a causality token that guarantees a client never reads staler than its own writes.

Enable it. Set MAIDAN_DB_REPLICA_URL to a hot-standby's connection string. The server connects it at boot (fail-fast on a bad URL) and a background task polls the standby's replay position every 200 ms, so each read's primary-vs-replica choice is a cheap in-memory compare (no extra round-trip). Unset → every read uses the primary (unchanged).

The consistency token. A successful mutating request returns a Maidan-Consistency-Token response header (the primary's WAL LSN at that point). A client that wants read-your-writes echoes it on a later request as the Maidan-Consistency-Token request header. That read is served from the replica only once the replica has replayed past the token; until then it falls back to the primary. A read with no token may be served from the replica immediately (the caller has asserted no causality requirement).

What routes, and what never does.

• Routed (only for GET/HEAD): content and collaboration reads — messages, threads, channels, members, DMs, social (votes/reactions/pins/mentions), notifications, follows, skills, assignments, dependencies, queue depth, and usage. Message search (v271.0.0) routes too: maidan-search's PostgresSearch has its own replica reader pool + replay poller and honors the same token via the same routing logic, so GET …/search reads-your-writes and offloads to the replica identically (embedding writes / index DDL / reindex stay on the primary). Its primary/replica split is counted separately as maidan_search_replica_reads_total (v272.0.0).
• Always the primary: every write; auth-path reads (sessions, API tokens, OIDC, federation peers) — the auth middleware runs on GETs, so a just-minted credential must be read fresh; control-plane/config reads (webhooks, slash commands, FSM hooks, deliveries, reindex jobs, audit, token quotas); and any read inside a mutation handler (those requests are never in a read-routing scope, so a read-then-write decision is always on primary data).

Observability. maidan_replica_reads_total{outcome="primary"|"replica"} counts the store split and maidan_search_replica_reads_total{outcome} (v272.0.0) the search split; maidan_replica_lag_bytes is the replica's WAL lag (primary write LSN minus replica replay LSN, shared by both). Complement with Postgres's own pg_stat_replication.

Testing. scripts/replica-harness.sh up stands up a local pgvector primary + streaming standby and prints MAIDAN_PRIMARY_URL / MAIDAN_REPLICA_URL; the #[ignore]d read_routing / replication store tests validate store routing and read-your-writes against it (cargo test -p maidan-store --test read_routing -- --ignored), and the #[ignore]d replica_routing search test proves the same for message search (cargo test -p maidan-search --test replica_routing -- --ignored).

Backup & disaster recovery (v260.0.0)

Maidan's durable state is two things, and the backup story follows the same split:

Two operator scripts implement it:

• scripts/backup.sh [BACKUP_DIR] — pg_dump (custom format) plus, for localfs, a tar of ARTIFACT_LOCALFS_ROOT; writes a MANIFEST.txt. For s3, the bucket is the durable copy — enable bucket versioning and/or cross-region replication there rather than copying blobs into the backup.
• scripts/restore.sh <backup-dir> [--force] — pg_restore into the target DATABASE_URL (+ untar artifacts). It refuses a non-empty target unless --force, so a restore can't silently clobber a live database; --force restores with --clean --if-exists.

Not in the data backup — restore these from your secret manager, out of band: DATABASE_URL, MAIDAN_SESSION_SECRET (subscribe-resume/session signing), FEDERATION_ENCRYPTION_KEY (+ any FEDERATION_DECRYPT_KEYS — see the Cluster-189 rotation keyring), and SMTP/OIDC credentials. A DB dump without the session secret still restores all data; only signed-token continuity needs the same secret.

RPO / RTO. A periodic backup.sh (e.g. hourly cron) gives an RPO of one backup interval. For a tighter RPO, run Postgres with WAL archiving / PITR (or a managed Postgres with continuous backup) — the logical dump is the portable floor, not the lower bound. RTO is a restore.sh run plus a /health/ready check before the load balancer is pointed at the restored instance.

Recovery outline. Provision Postgres + the artifact store → set the out-of-band secrets → DATABASE_URL=… ARTIFACT_LOCALFS_ROOT=… scripts/restore.sh <dir> --force → start one replica and confirm /health/ready is 200 (it gates on DB + object store + indexer + the LISTEN bus) → scale out. Because artifacts are content-addressed, a message referencing a blob that predates the artifact backup is still consistent after restore; a blob written after the last artifact archive is the only thing a stale artifact backup can miss.

API stability

From v1.0.0, HTTP and MCP shapes are semver-stable. Pre-1.0 releases may break without migration shims.