Production deployment

Guidance for running Maidan at v1.0.0 and later. Security overview: Threat-Model.md.

Probes

Operator status

GET /operator/status (operator:global) is the page to read before a deploy or during an incident. It reports:

• Phase: serving, degraded (a readiness check fails) or draining.
• Readiness checks: the same ones /health/ready runs.
• Search backfill: the search tap's cursor against the event-log head, as a percentage and a count behind. Below 100% after a rebuild means the backfill is still running.
• Replica lag, in bytes, when a read replica is configured.
• Queue depths: the indexer queue, WebSocket connections against their ceiling, and the outbox.

It returns JSON, or a script-free page when the request sends Accept: text/html.

Rolling restarts

Kubernetes takes a terminating pod out of its Service at the same moment it sends SIGTERM, and the endpoint change takes a few seconds to reach every proxy. On SIGTERM the server therefore reports /health/ready as 503 {"status":"draining"}, keeps serving for MAIDAN_SHUTDOWN_DRAIN_SECS, and only then closes its listener and drains in-flight requests. Liveness stays 200 throughout. The Helm chart and k8s/base set the drain to 10 and terminationGracePeriodSeconds to 45; the default outside them is 0. The runtime image is distroless, so this is done in-process rather than with an exec preStop hook. A long-poll MCP call (up to 330 s) still open at the grace period is cut, and clients retry.

Environment

GET /metrics serves the Prometheus text exposition (HTTP, subscribe recovery, indexer and bus gauges) with fixed label cardinality — no workspace UUIDs.

Compiling OpenTelemetry out. OTLP export is a default-on cargo feature (otel) on maidan-server. Building with --no-default-features (or a custom feature set that omits otel) drops the OpenTelemetry/OTLP/tonic stack entirely for a leaner binary — plain tracing logs and the Prometheus /metrics scrape are unaffected. In such a build the OTLP_* variables above are inert (an OTLP_ENDPOINT that is set is reported to stderr at startup and otherwise ignored). Leave the feature on (the default) to keep OTLP traces + metrics push available.

SCIM 2.0 provisioning. An IdP (Okta / Entra ID / …) can provision and deprovision workspace members via SCIM 2.0 at /scim/v2/: ServiceProviderConfig, Users (create / read / list with userName, externalId or id eq filters / replace / patch / delete) and Groups (create / read / list / replace / patch / delete). Point the IdP's SCIM connector at https://<host>/scim/v2 with a token:admin bearer token; everything it does is confined to that token's workspace. No env config: the endpoint is always available, gated on token:admin. Every change writes its audit row (scim.user.*, scim.group.*, and one token.revoke per revoked token) in its own transaction, so a change that cannot be recorded does not happen and the IdP retries it.

• Users. A SCIM id is the Maidan member id and userName the member handle. Changing userName, by PUT (Okta) or by PATCH with "path": "userName" or a pathless value object (Entra ID), renames the member: the id stays, so tokens, messages, claims and group memberships stay with it, and every surface that shows the handle shows the new one. A userName another member of the workspace holds is 409 with scimType: uniqueness. Deactivation (active=false, also Entra ID's string "False") and delete revoke the member's API tokens; delete also removes the user from every group. Both also revoke the delegation grants the member holds as delegate, so a token exchanged under one stops working too, and both delete the member's browser sessions in the same transaction. Each revoke and delete has its own audit row, with reason scim_deactivate or scim_deprovision. A deactivated member cannot sign in through OIDC (the callback answers 403), and any session that survives anyway, such as one made by a write outside the SCIM path, is refused at its next request and deleted with reason member_deactivated. displayName is set at creation and not changed afterwards.
• Groups. A group is the IdP's named set of users it provisioned into the workspace. It records membership and grants nothing by itself: no channel access and no capability follow from it. Members must be SCIM users of the same workspace; any other id (another workspace's member, an agent, an unknown id) is 400 with scimType: invalidValue. PATCH accepts add, remove and replace of members in both the Okta form (remove with the filter path members[value eq "<id>"]) and the Entra ID form (path members with a value array, capitalized op), remove of members with no value empties the group, and replace of displayName and externalId with a path or a pathless value object. The list filters on displayName eq (not case-sensitive), externalId eq or id eq, pages with startIndex and count (at most 200), and honours excludedAttributes=members; any other filter is 400 with scimType: invalidFilter.
• Filters. Users filters on userName eq (not case-sensitive), externalId eq or id eq, and answers with that workspace's matching users only. Any other filter, on Users or Groups, is 400 with scimType: invalidFilter, never the whole list, so an IdP matching on an attribute Maidan does not filter cannot link to the wrong user.
• Not supported: Bulk, sorting, ETags, Schemas and ResourceTypes, nested groups, compound (and/or) filters, and paging on Users.

Database tuning (v107.0.0)

• Total connections = replicas × MAIDAN_DB_MAX_CONNECTIONS. Behind a load balancer this must stay under Postgres max_connections (default 100) with headroom for migrations, the bus LISTEN connections, and admin tools. E.g. 4 replicas × 16 = 64. Raise the pool only after confirming the server is connection-starved (acquire timeouts), not query-bound.
• MAIDAN_DB_STATEMENT_TIMEOUT_MS applies to every server query, including the in-server operator reindex (POST /operator/reindex-embeddings). The default is now 30000 (30 s); raise it above your longest expected query, or trigger large reindexes via the maidan reindex-embeddings CLI, which uses its own pool with no cap, or set 0 to disable the cap entirely. Boot migrations are already exempt (the migration session resets the timeout under the advisory lock), so the default will not break startup or a rolling update.

Tenant fairness (v110.0.0)

On a shared instance, MAIDAN_WORKSPACE_RATE_LIMIT_MAX bounds the total request rate for any single workspace (across all its tokens) on /workspaces/{wid}/… routes — so one tenant's heavy loop (a tight semantic-search poll, a backfill) can't monopolize the connection pool and degrade search/write latency for others. The budget is charged only once the caller is authenticated into that workspace, so naming the id is not enough to spend it. It is on by default at 6000 requests per 60 s (about 100 a second): five clients each at the per-client default of 1200 per 60 s, so a busy workspace's agents meet their own limits before the shared one, and well under what one node serves (666–1586 requests a second on the SQLite benchmark). Set a value to change it, or 0 to turn it off. It is independent of the per-client MAIDAN_RATE_LIMIT_MAX. With MAIDAN_RATE_LIMIT_REDIS_URL set, the per-workspace counter is shared across replicas; otherwise it is per-process, so N replicas without Redis admit up to N times the limit. Not a substitute for hard CPU/IO isolation — that is infra-level (separate instances / Postgres resource groups).

Local embedding servers (e.g. LM Studio)

Maidan's indexer uses the OpenAI-compatible embeddings API shape, not chat completion. Point MAIDAN_EMBEDDING_PROVIDER=openai-compatible at your server's embeddings URL (for example http://localhost:1234/v1/embeddings) and set MAIDAN_EMBEDDING_MODEL to the loaded model id. A chat endpoint such as http://localhost:1235/api/v1/chat is not used for search indexing.

Bootstrap

maidan init (recommended)

The maidan CLI seeds the first admin directly through the store, so a production deployment needs no unauthenticated HTTP routes and no AUTH_DISABLED:

DATABASE_URL=postgres://… MAIDAN_CONTENT_KEK=… maidan init --workspace my-team --admin-handle david

For containerized deployments, the separately published CLI image runs the same release binary without adding a shell or operator tooling to the distroless server image. Put it on a network that can reach the database and pin it to the server's exact tag:

MAIDAN_TAG=v413.0.0
MAIDAN_NETWORK=your_database_network
docker run --rm --network "$MAIDAN_NETWORK" \
  -e DATABASE_URL=postgres://maidan:…@postgres/maidan \
  -e MAIDAN_CONTENT_KEK \
  "ghcr.io/david-engelmann/maidan-cli:${MAIDAN_TAG}" \
  init --workspace my-team --admin-handle david

It runs migrations, creates the initial workspace and an admin member, mints an all-capabilities bearer token, and prints that token once (to stdout; save it). It refuses if the database already has a workspace, so it can never clobber an existing deployment or mint a second root token. Use the printed token to mint narrower per-agent tokens via the API. The production image can stay bootstrap-stripped (--no-default-features), since init writes through the store rather than the bootstrap HTTP routes. The CLI image is a one-shot operator tool, not a long-running sidecar.

On a release tag, GitHub Release publication waits for a smoke that pulls the published server, CLI, and Postgres images by that exact tag. It initializes a fresh database through the CLI image, requires the server to report the tag as healthy, authenticates /me with the one-time token, and confirms anonymous access is rejected. A source checkout or locally built image cannot satisfy that release gate.

A second workspace

maidan init stops after the first workspace. A later tenant is POST /operator/workspaces with the init token (it holds operator:global):

curl -sS -X POST "$MAIDAN_URL/operator/workspaces" \
  -H "Authorization: Bearer $MAIDAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"other-team","admin_handle":"ada"}'

The response is the workspace, its admin member, and that admin bearer token, shown once. The token can administer that workspace, including minting narrower tokens. It does not hold operator:global or audit:read-global, and a later mint cannot add those unless the caller already holds them. No MAIDAN_BOOTSTRAP, and the route exists on the production image, which is built without the bootstrap routes.

Connect an agent

An admin token (one holding token:admin) connects an agent in one call. The call creates the agent member and its maidan.agent.worker token together, and returns the token once:

curl -sS -X POST "$MAIDAN_URL/workspaces/$WORKSPACE_ID/agents" \
  -H "Authorization: Bearer $MAIDAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"handle":"builder","display_name":"Builder"}'

The response is the member and its token (token.secret, shown once). The member is always an agent. The route takes no capability list and no kind, so it can't create a human or grant more than the worker set. The member, the token and both audit rows (member.create, token.mint) commit together. A taken handle is a 409 and leaves nothing behind. Like the second-workspace call, it is on the production image and needs no MAIDAN_BOOTSTRAP. The console's Connect an agent uses it.

HTTP bootstrap (development only)

maidan init above is how a deployment gets its first token, on a private network too. The published server image is built without the bootstrap Cargo feature, so the routes below do not exist in it.

A development build with the feature (cargo run, or docker build --build-arg MAIDAN_ENABLE_BOOTSTRAP=1 -f crates/maidan-server/Dockerfile .) serves unauthenticated POST /workspaces and POST /workspaces/:wid/members when MAIDAN_BOOTSTRAP=1 is set. Only the first workspace may be created that way; a second POST /workspaces returns 403. Those routes mint no token, so a development seed over HTTP also runs without auth, which needs both variables:

1. AUTH_DISABLED=1 and MAIDAN_ALLOW_INSECURE_NO_AUTH=1 (the acknowledgement: AUTH_DISABLED alone refuses to boot, and with MAIDAN_ENV=production it refuses regardless), plus MAIDAN_BOOTSTRAP=1.
2. Create the workspace and member, mint an admin token.
3. Unset all three and restart with auth on.

Integration tests use AUTH_DISABLED=1 + MAIDAN_ALLOW_INSECURE_NO_AUTH=1 (bootstrap flag not required).

Human browser login via OIDC ships in v2.0.0. See OIDC.md for design detail. Summary:

Experimental Jev land-gate advisor

The decision-model spike is off by default and does not participate in the authoritative land-gate write or close paths. Enabling it adds only POST /threads/:id/land-gate/advice; see Agent integration — Experimental Jev advice for its request, response, calibration harness, and privacy boundary.

When enabled, a missing key, malformed threshold, unsafe base URL, or DNS resolution failure stops startup rather than silently enabling a partial configuration. With the flag unset or 0, none of the other variables are read and the server behaves exactly as before.

After OIDC login, use /ui/ (session cookie) or mint an API token for MCP.

The board (/ui/): channels and tasks, and the thread when one is open. Posts go through /ui/api/... on the session cookie. A token is pasted on the first-run card — those fields sit behind Change in the header once you are in — and exchanged for a session with that token's authority (POST /auth/session/from-token). The page does not keep the token. Revoking or rotating it ends the session.

Sessions. A session row (maidan_sessions) names its member and workspace and, for one made from a token, the token's id. It is checked on every request: an expired row, or one whose token is no longer live, is deleted and refused. Signing out (POST /auth/logout) deletes the row. CSRF is handled by SameSite=Lax, JSON request bodies, and refusing an unsafe session request or a session WebSocket from another origin; the session keeps no CSRF secret. Behind a proxy that rewrites Host, browsers still send Sec-Fetch-Site, which the check prefers. Accepting an approval gate takes the strict form of the check: a request that names no origin is refused, and only a session signed in through the identity provider (not one made from a token) accepts without approval:grant. Signing in to the console with a pasted token still declines and cancels gates. Creating a session writes an audit row in the same transaction (session.from_token, or session.create for an OIDC login). Remove MAIDAN_BOOTSTRAP once the first human has token:admin.

One instance built from main

The quickstart and the release images run a tagged release. A shared instance that tracks main is built from the repository at a pinned commit instead. The stack that runs it lives outside this repository; this section is what it needs from here.

1. Build the images at one commit. Record the full commit SHA where the stack is defined, and rebuild only by changing it. The server, the database image and the CLI come from the same commit, because maidan init applies that commit's migrations.

git clone https://github.com/david-engelmann/maidan.git && cd maidan
git checkout "$COMMIT"            # a full SHA on main
export MAIDAN_TAG="main-$(git rev-parse --short=12 HEAD)"

docker build -f crates/maidan-server/Dockerfile --build-arg MAIDAN_VERSION="$MAIDAN_TAG" \
  -t "maidan-server:$MAIDAN_TAG" .
docker build -f docker/Dockerfile.db -t "maidan-postgres:$MAIDAN_TAG" .   # pgvector

# The CLI image wraps a binary built for Debian bookworm, like CI's release CLI image.
docker run --rm -v "$PWD":/src -w /src -e CARGO_TARGET_DIR=/src/target-cli \
  rust:1.91-slim-bookworm sh -c \
  'apt-get update -qq && apt-get install -y -qq pkg-config libssl-dev >/dev/null \
   && cargo build --release -p maidan-cli'
mkdir -p cli-image && cp target-cli/release/maidan cli-image/maidan
cp docker/Dockerfile.cli cli-image/Dockerfile
docker build --build-arg MAIDAN_VERSION="$MAIDAN_TAG" -t "maidan-cli:$MAIDAN_TAG" cli-image

Put MAIDAN_TAG in the stack's .env so compose runs these images. The server image is built without the bootstrap feature, so it has no unauthenticated seed routes and MAIDAN_BOOTSTRAP does nothing.

2. Choose the database.

Use Postgres for an instance several agents write to at once. SQLite suits one operator and a few agents, and is one file to copy away (with the server stopped, or through scripts/backup.sh).

3. Write the secrets to files. Each secret-bearing variable is read from the file <NAME>_FILE names when it is set (see Environment), so the container's config holds paths and docker inspect shows no secret. The server runs as uid 65532; give it the files:

mkdir -p secrets
openssl rand -hex 32 > secrets/content_kek          # back this up apart from the data
openssl rand -hex 32 > secrets/session_secret
openssl rand -hex 24 > secrets/postgres_password
echo "postgres://maidan:$(cat secrets/postgres_password)@maidan-postgres:5432/maidan" \
  > secrets/database_url
# secrets/github_token, github_webhook_secret, slack_bot_token, slack_signing_secret:
# paste each value, one per file.
sudo chown 65532:65532 secrets/* && sudo chmod 600 secrets/*

MAIDAN_GITHUB_TOKEN is a personal access token with exactly two repository permissions, contents:write and pull_requests:write (fine-grained: Contents and Pull requests, read and write), on exactly the repositories the change flow may write to; Result delivery to GitHub and Slack says why nothing else. Postgres's own password goes through the Postgres image's POSTGRES_PASSWORD_FILE.

The token belongs to the instance, while egress allowlists belong to workspaces, so the operator also names the repositories the token may write to in MAIDAN_GITHUB_WRITE_REPOS, for example MAIDAN_GITHUB_WRITE_REPOS=example/app,example/skills. A write to any other repository is refused before a request is made, whichever workspace asks. To let the mark-ready app mark agent pull requests ready, set MAIDAN_MARK_READY_APP_ID to the id of that app (GET /workspaces/{wid}/apps lists it). To pin a repository's flips to one base whatever its allowlist blesses, list it in MAIDAN_MARK_READY_BASES, for example MAIDAN_MARK_READY_BASES=example/app=dev,example/skills=main. To let the same app put a ready pull request into a repository's merge queue, name the queue's label in MAIDAN_MERGE_QUEUE_LABELS, for example MAIDAN_MERGE_QUEUE_LABELS=example/app=ready-to-merge.

4. The services. Auth is on: nothing sets AUTH_DISABLED, and MAIDAN_ENV=production refuses it and the development KEK outright. With production the session cookie is Secure, so reach the board over HTTPS or on localhost; bearer tokens work either way.

services:
  maidan-postgres:
    image: maidan-postgres:${MAIDAN_TAG}
    environment:
      POSTGRES_USER: maidan
      POSTGRES_DB: maidan
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
    secrets: [postgres_password]
    volumes: [maidan_pg:/var/lib/postgresql/data]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U maidan -d maidan"]
      interval: 2s
      retries: 30

  # The image has no /data, so a new named volume belongs to root; hand it to
  # the server's uid before anything writes artifacts (or SQLite) there.
  maidan-volume-init:
    image: busybox:1.36
    command: ["chown", "65532:65532", "/data"]
    volumes: [maidan_data:/data]

  maidan:
    image: maidan-server:${MAIDAN_TAG}
    depends_on:
      maidan-postgres: { condition: service_healthy }
      maidan-volume-init: { condition: service_completed_successfully }
    environment:
      MAIDAN_ENV: production
      MAIDAN_BIND: 0.0.0.0:8080
      ARTIFACT_BACKEND: localfs
      ARTIFACT_LOCALFS_ROOT: /data/artifacts
      DATABASE_URL_FILE: /run/secrets/database_url
      MAIDAN_CONTENT_KEK_FILE: /run/secrets/content_kek
      MAIDAN_SESSION_SECRET_FILE: /run/secrets/session_secret
      MAIDAN_GITHUB_TOKEN_FILE: /run/secrets/github_token
      MAIDAN_GITHUB_WEBHOOK_SECRET_FILE: /run/secrets/github_webhook_secret
      MAIDAN_GITHUB_WRITE_REPOS: example/app,example/skills
      MAIDAN_MARK_READY_BASES: example/app=dev,example/skills=main
      MAIDAN_SLACK_BOT_TOKEN_FILE: /run/secrets/slack_bot_token
      MAIDAN_SLACK_SIGNING_SECRET_FILE: /run/secrets/slack_signing_secret
    secrets:
      - database_url
      - content_kek
      - session_secret
      - github_token
      - github_webhook_secret
      - slack_bot_token
      - slack_signing_secret
    volumes: [maidan_data:/data]
    healthcheck:
      test: ["CMD", "/usr/local/bin/maidan-server", "--health-check"]
      interval: 5s
      retries: 30
      start_period: 10s

  # One-shot, run once by hand: `docker compose run --rm maidan-init`.
  maidan-init:
    image: maidan-cli:${MAIDAN_TAG}
    profiles: [init]
    depends_on:
      maidan-postgres: { condition: service_healthy }
      maidan-volume-init: { condition: service_completed_successfully }
    command: ["init", "--workspace", "my-team", "--admin-handle", "ops"]
    environment:
      DATABASE_URL_FILE: /run/secrets/database_url
      MAIDAN_CONTENT_KEK_FILE: /run/secrets/content_kek
    secrets: [database_url, content_kek]
    volumes: [maidan_data:/data]

secrets:
  postgres_password: { file: ./secrets/postgres_password }
  database_url: { file: ./secrets/database_url }
  content_kek: { file: ./secrets/content_kek }
  session_secret: { file: ./secrets/session_secret }
  github_token: { file: ./secrets/github_token }
  github_webhook_secret: { file: ./secrets/github_webhook_secret }
  slack_bot_token: { file: ./secrets/slack_bot_token }
  slack_signing_secret: { file: ./secrets/slack_signing_secret }

volumes:
  maidan_pg:
  maidan_data:

For SQLite, drop maidan-postgres, its volume and password, and the depends_on entries that name it, and in both maidan and maidan-init replace DATABASE_URL_FILE and its secret with DATABASE_URL: sqlite:///data/maidan.db?mode=rwc (a path is not a secret). The two then share the database file through maidan_data.

A variable set both ways (MAIDAN_GITHUB_TOKEN and MAIDAN_GITHUB_TOKEN_FILE) or a file that cannot be read refuses boot and names the variable. The server logs which variables came from files, never their values.

5. The first workspace and token. Run init once, then start the server:

docker compose run --rm maidan-init
docker compose up -d --wait maidan

Init migrates the database, creates the workspace and its admin member, and prints the workspace id and an admin bearer token holding every capability, once. Keep the token in the stack's secret store; it is the token:admin the next step needs, and the one to mint narrower agent tokens from. A second run refuses. No MAIDAN_BOOTSTRAP and no unauthenticated route is involved.

6. Seed the egress allowlist. Nothing is allowed by default. Each row is one audited call, POST /workspaces/{wid}/egress-targets with {surface, selector} and a token:admin bearer, and writes an egress_target.allow audit row. A github_branch row (owner/name@base) lets the change flow commit to an agent branch of that repository and open a draft pull request into that base, one row per base; a github row (owner/name) allows result comments only; a slack row is a channel id. With example repositories:

MAIDAN=http://maidan:8080; WID=<workspace id from init>; ADMIN_TOKEN=<token from init>
URL="$MAIDAN/workspaces/$WID/egress-targets"
H1="Authorization: Bearer $ADMIN_TOKEN"; H2='Content-Type: application/json'

curl -fsS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github_branch","selector":"example/app@dev"}'
curl -fsS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github_branch","selector":"example/skills@main"}'
curl -fsS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github","selector":"example/app"}'
curl -fsS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github","selector":"example/skills"}'
curl -fsS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"slack","selector":"C0123456789"}'

GET "$URL" lists the rows; DELETE "$URL/<id>" revokes one.

7. Smoke check. From any container on the stack's network (the server image has no shell or curl; its own probe is maidan-server --health-check):

curl -fsS "$MAIDAN/health/ready"                                   # 200: database, artifacts, bus
test "$(curl -s -o /dev/null -w '%{http_code}' "$MAIDAN/me")" = 401  # auth is on
curl -fsS "$MAIDAN/me" -H "$H1" | jq -e --arg w "$WID" '.workspace_id == $w'
curl -fsS "$URL" -H "$H1" | jq -e 'length >= 5'                    # the seed is in place

And on the Docker host, that the container's config carries paths and no secret (with SQLite, take DATABASE_URL| out of the pattern):

docker inspect "$(docker compose ps -q maidan)" --format '{{json .Config.Env}}' | jq -e \
  'map(select(test("^(DATABASE_URL|MAIDAN_(CONTENT_KEK|SESSION_SECRET|GITHUB_TOKEN|GITHUB_WEBHOOK_SECRET|SLACK_BOT_TOKEN|SLACK_SIGNING_SECRET))="))) | length == 0'

Upgrading. Change COMMIT, rebuild the three images and restart the server; it migrates forward on boot. Back up first (see Backup & disaster recovery). Do not run init again.

API discovery

Import into Swagger UI, Redoc, or your client generator. The document version tracks the server release (info.version).

A2A transports (v282.0.0+)

Maidan speaks the A2A protocol across three bindings, all over the same operations and auth:

The Agent Card (GET /.well-known/agent-card.json) advertises the available interfaces so clients can negotiate a transport (A2A §5.2). Configure it for your deployment:

The gRPC listener speaks plaintext HTTP/2 and does not terminate TLS, and calls carry the bearer token in their metadata, so the server refuses to bind it off loopback until MAIDAN_A2A_GRPC_PLAINTEXT=1 says TLS is handled in front. Do not expose it to an untrusted network: put it behind a TLS-terminating ingress or proxy, and keep the hop from the proxy to Maidan on a trusted private network. Expose the gRPC port in your deployment (Kubernetes Service / compose port) only that way; the HTTP bindings share the main HTTP port.

A2A push configs seal their token and credentials with FEDERATION_ENCRYPTION_KEY; without it, creating a push config that carries either fails. Push targets pass the same egress checks as webhooks, and secret:// references in a pushed task are substituted like a webhook's (see Secret substitution on egress).

WebSocket and MCP subscribe (v4.0.0)

Real-time subscribers use GET /ws/subscribe (WebSocket) or GET /mcp/stream (SSE). Both share the same control frames and event envelope shape.

An open SSE stream keeps checking the credential it opened with. It does this before a frame goes out (at most once a second) and every 10 seconds while idle. A revoked or expired token, or a member deactivated through SCIM, ends the stream. Every SSE stream sends one last event: stream_ended with data: {"reason": "..."} and closes. The reason is token no longer valid, member deactivated or credential check failed. The SSE streams are /mcp/stream, /agui/stream, /mcp/notifications, both /mcp/streamable streams and A2A SubscribeToTask. A2A's gRPC SubscribeToTask ends with UNAUTHENTICATED instead. A client that reconnects goes through the normal authorization again.

MCP resource subscription notifications use GET /mcp/notifications (SSE JSON-RPC lines) with POST /mcp for resources/subscribe / tools/call — requires workspace:read (same as resource read). Distinct from /mcp/stream workspace events. A listener receives only what its own credential subscribed to, for resources it can still read. Behind a load balancer no affinity is needed: a stateless client's subscriptions are kept in the database (maidan_mcp_resource_subscriptions), and the replica holding its listener delivers them over the Postgres NOTIFY path. They lapse once no replica has had a listener open for the caller for MAIDAN_MCP_STREAMABLE_SESSION_TTL_SECS (default 3600). A 2024-11-05 session's subscriptions stay in the replica holding the session.

Semantic search: Postgres uses pgvector; SQLite uses stored 1024-dim embeddings with cosine ranking (dev parity, no HNSW index on SQLite).

First message (WebSocket)

Send one text frame after connect:

{
  "filter": { "workspace_id": "<uuid>", "kinds": ["message_posted"] },
  "after_id": 0,
  "token": "<bearer when auth enabled>"
}

Or reconnect with only:

{ "resume_token": "<from subscribe_ack>", "token": "<bearer>" }

Invalid or expired resume_token closes the socket with code 1008.

MCP SSE query

GET /mcp/stream?workspace_id=<uuid>&after_id=0 or ?resume_token=<opaque>. Requires bearer with event:subscribe.

Control frames

Loop: on replay_truncated, reconnect or resubscribe with after_id (or a fresh resume_token from the next subscribe_ack) until no truncation frame.

Event envelopes follow: { "log_id": <i64>, "kind": "...", ... }.

At-least-once delivery (v125.0.0)

By default the live path is optimistic, best-effort: events stream with low latency, but an event published out of log_id order (a failed outbox row retried after later rows, or a late-committing serial) can be silently skipped by the monotonic watermark, and the live buffer can drop events on lag.

Set at_least_once (requires both a workspace filter and a durable consumer_id) to switch that subscription to cursor-driven reconcile delivery — on WebSocket (/ws/subscribe frame) or MCP SSE (/mcp/stream query param), v126.0.0:

{ "filter": { "workspace_id": "<uuid>" }, "consumer_id": "my-agent", "at_least_once": true }

GET /mcp/stream?workspace_id=<uuid>&consumer_id=my-agent&at_least_once=true

• Guarantee: every committed event matching the filter is delivered in log_id order and exactly once per consumer_id — no silent gaps. The durable delivery cursor floors re-delivery across reconnects.
• Cost: a stability-window latency floor on fresh events (MAIDAN_DELIVERY_STABILITY_SECS, default 2s); the backlog (already stable) is delivered immediately on connect.
• Caveat: strictness holds under "no insert transaction outlives the window". A pathologically long (> window) write transaction can still strand a lower log_id; size the window above your slowest write transaction. Clients should still dedup by log_id (cheap, and the contract is at-least-once).

Delivery reliability metrics (v6.0.0)

Scrape GET /metrics and alert on subscribe recovery paths (labels are fixed — no per-workspace series).

Postgres bus NOTIFY pointers (v7.0.0)

Production mutations append to maidan_events before pg_notify. The NOTIFY payload is a small log_id_v1 pointer; the server hydrates the row before fan-out. Very large message bodies are limited by the database row, not the legacy ~8KB NOTIFY cap.

Direct bus.publish without a prior append_event (tests only) still uses full JSON on NOTIFY and can hit PayloadTooLarge. Operators should rely on HTTP mutations or federation ingest for large events.

Bus hydrate metrics (v8.0.0)

Postgres pointer delivery records hydrate outcomes on /metrics:

Subscribers may still recover via event-log replay (maidan_subscribe_replay_total); hydrate drops do not change at-most-once NOTIFY semantics.

Outbox relay (v10.0.0 Postgres, v12.0.0 quarantine, v14.0.0 SQLite)

Postgres and SQLite deployments enqueue maidan_outbox in the same transaction as maidan_events. A background relay publishes after commit (Postgres pointer NOTIFY; SQLite in-memory bus). HTTP handlers do not call bus.publish directly when relay is enabled.

Adaptive cadence (v108.0.0)

The relay is adaptive: it drains back-to-back (no inter-batch sleep) while a tick fully relays a batch, so a backlog of N rows clears in ≈⌈N/batch⌉ ticks instead of N/batch × interval. When caught up it sleeps the base interval and backs off toward MAIDAN_OUTBOX_MAX_POLL_INTERVAL_MS while idle — so a quiet deployment isn't polling 20×/s for nothing. An in-process enqueue nudge wakes the relay the moment a row is written, so the backoff costs no added latency on a fresh event (the cap only bounds the worst case if the nudge is ever missed). Tuning: lower the base interval for snappier single-process fan-out; raise the cap to poll less when idle. Delivery semantics (at-most-once NOTIFY, quarantine, replay) are unchanged by cadence.

NOTIFY loss / listener unhealthy (v84.0.0)

When maidan_bus_listener_ok is 0 or maidan_bus_notify_hydrate_total{result="failed"} rises but maidan_outbox_pending stays high:

1. Confirm the outbox relay task is running (outbox relay running in logs; maidan_outbox_relay_total incrementing).
2. Single-process mitigation: set MAIDAN_OUTBOX_RELAY_MODE=polled and restart. Relay publishes to the in-process bus without pg_notify. Subscribers on other pods still need NOTIFY or WS replay — polled mode is not a multi-instance fan-out replacement.
3. Multi-instance: fix LISTEN connectivity (pooler must not pin LISTEN connections; use direct Postgres or a pooler that supports LISTEN). Do not disable outbox relay in production.
4. Clients can recover via subscribe replay (after_id / resume_token) from maidan_events while relay catches up.

Relay retries may duplicate NOTIFY; subscribers should dedupe by log_id.

Delivery cursors (v13.0.0)

Inspect cursors: SELECT * FROM maidan_delivery_cursor WHERE workspace_id = $wid;

Reset a stuck cursor (operator SQL): UPDATE maidan_delivery_cursor SET last_delivered_log_id = 0 WHERE consumer_id = $id AND workspace_id = $wid;

Manual recovery for a quarantined row (operator SQL, not exposed over HTTP in 12.0):

1. Fix the underlying bus/hydrate issue.
2. HTTP (v56.0.0): POST /workspaces/{wid}/outbox/{id}/replay with workspace:write clears quarantine when the row’s event belongs to that workspace.
3. SQL: UPDATE maidan_outbox SET quarantined_at = NULL, attempts = 0 WHERE id = $id; so the relay picks it up again, or leave quarantined and rely on clients replaying from maidan_events by log_id.

Automation HTTP delivery (v68.0.0)

Slash commands and FSM hooks with handler_kind: http enqueue signed POSTs in maidan_automation_deliveries. A background worker retries with exponential backoff; exhausted rows are quarantined (dead letter). Outbound event webhooks still use maidan_webhook_deliveries and WebhookWorker — same signing headers, separate queue. All operator-supplied HTTP destinations use the public-egress guard described in the environment table above; a target that becomes private after registration fails delivery rather than being contacted.

Dispatch behavior

Signing (unchanged from webhooks): Content-Type: application/json, per-registration X-Maidan-Event (or configured header), X-Maidan-Signature (HMAC of body), plus X-Maidan-Delivery-Id for idempotency. Integrators must treat delivery as at-least-once.

Operator HTTP (workspace:read / workspace:write):

Manual recovery (SQL): UPDATE maidan_automation_deliveries SET quarantined_at = NULL, attempts = 0, next_attempt_at = datetime('now') WHERE id = $id; (SQLite) or equivalent now() on Postgres — prefer HTTP replay when auth is available.

Secret substitution on egress

Event webhooks, automation HTTP (slash commands and FSM hooks, the first POST and every queued retry) and A2A push notifications replace secret://<name> references in their JSON body with the sending workspace's secret value, at send time, when the delivery URL's host is on that workspace's secret-egress allowlist (maidan_secret_egress_hosts). Any other host gets the literal reference. Queued rows keep the reference, so a retry after a host is removed carries the literal; the value is never written to a queue, the event log or an audit row. Substitution needs FEDERATION_ENCRYPTION_KEY; without it every reference stays literal.

Workspaces manage their own lists (GET/POST /workspaces/:wid/secret-egress-hosts, DELETE …/:host; MCP list_/allow_/revoke_secret_egress_host). Adding a host needs secret:admin and secret:read. The operator's control is a ceiling:

Narrowing the ceiling does not delete workspace entries outside it; they stay listed and inert until the ceiling admits them again or a workspace removes them. Before this list existed the variable was the whole allowlist, for every workspace; a deployment that set it keeps the ceiling but substitutes nothing until each workspace lists its hosts.

Outbound timeouts and the retry budget

Every outbound delivery (event webhooks, automation HTTP, Slack and GitHub projector and result egress) gives up after 5 s waiting to connect and 10 s in all, so a receiver that accepts the connection and never answers cannot hold up the workers. An operator-supplied URL (a webhook, an automation target, a slash or FSM handler, a federation peer, an A2A push, web push, the advisor) also gives up after 5 s resolving the name, before those clocks start, so a nameserver that never answers cannot hold them either. A failed delivery is retried on its worker's backoff: webhooks and automation wait 2^attempts seconds (at most 256 s) up to their *_MAX_ATTEMPTS; mail and egress wait 30 s doubling to an hour, eight attempts.

Backoff spaces one delivery's attempts, not a destination's. When a host that was down comes back, everything queued for it is due at once. So every worker in a process shares one retry budget per destination host (the webhook or automation URL's host, slack.com, the GitHub API host, the SMTP relay): a host takes at most 10 retries at once and 2 more a second after that. A first attempt is never held back and does not count. A retry the budget refuses is deferred, not failed: its next attempt moves 1 to 60 s forward, spaced so the backlog returns at about the budget's rate, and its attempt count stays as it was, so waiting never pushes a delivery toward the dead-letter queue. The limits are constants in crates/maidan-server/src/retry_budget.rs, not environment variables.

The budget is in memory on each replica: with N replicas a recovering host can take N times the rate, and a restart starts every host's budget full.

Result delivery to GitHub and Slack

Result delivery and the GitHub projector post with MAIDAN_GITHUB_TOKEN, or as a GitHub App (below), and the egress worker runs only when a Slack, GitHub or Linear sender is configured. Each sender needs only its outbound credential: the GitHub sender MAIDAN_GITHUB_TOKEN or the three MAIDAN_GITHUB_APP_* variables, the Slack sender MAIDAN_SLACK_BOT_TOKEN. The inbound secrets (MAIDAN_GITHUB_WEBHOOK_SECRET, MAIDAN_SLACK_SIGNING_SECRET) turn on only the webhook routes, so a deployment that delivers results and takes nothing back leaves them unset. They are read from the server's environment, or from the files MAIDAN_GITHUB_TOKEN_FILE and the rest name (see Environment); set them through your platform's secret mechanism like every other secret. Maidan never logs the token, and cuts it out of any error text it records on a delivery or an audit row.

The change flow (Result Delivery) commits a coding result to a branch and opens a draft pull request with this token, so commits and pull requests are authored by the token's owner. Use a personal access token with exactly two repository permissions, contents:write and pull_requests:write (in a fine-grained token: Contents and Pull requests, read and write), on exactly the repositories the change flow may write to. Grant nothing else. The flow stops at a draft pull request: it requests no review and needs no approval (a token owner cannot approve their own pull request), and the producer marks the pull request ready. The comment and check-run paths use the same token; a check run needs checks:write, which a personal access token does not carry, so the check run fails and the comment still posts.

Authenticating as a GitHub App. Set MAIDAN_GITHUB_APP_ID, MAIDAN_GITHUB_APP_INSTALLATION_ID and MAIDAN_GITHUB_APP_PRIVATE_KEY (or MAIDAN_GITHUB_APP_PRIVATE_KEY_FILE), and every outbound GitHub call, the change flow, comments, reviews and check runs alike, authenticates as the app instead of with MAIDAN_GITHUB_TOKEN:

• Maidan signs a JWT with the app's key (RS256, nine minutes long) and exchanges it at POST /app/installations/{id}/access_tokens for an installation token. That token reaches only the repositories the installation was granted, with only the app's permissions, and lasts an hour.
• The token is kept until five minutes before it expires, then exchanged again. One exchange runs at a time, so a burst of deliveries makes one call.
• Commits, pull requests and comments are authored by the app's bot account, not a person, and a check run works once the app is granted checks:write. Grant the app Contents, Pull requests, Issues and Checks (read and write) on the repositories the change flow may write to, and nothing else.
• MAIDAN_GITHUB_WRITE_REPOS still bounds every write, before a token is fetched. A refused write costs no exchange.
• With the app configured, MAIDAN_GITHUB_TOKEN is unused, and boot logs a warning if it is also set. A partial app config, an installation id that is not a number, or a key that does not parse refuses boot rather than falling back to the token. A refused exchange (a wrong key, a removed installation) is a misconfiguration, like a revoked token: the delivery is not retried. A rate-limited or failing exchange (429, 5xx) is retried like any other GitHub call.

The app key is a secret: mount it as a file. Maidan never logs it, and cuts the installation token out of any error text it records.

Guards in Maidan's code, whatever the token can do. The token may be able to push anywhere in those repositories, so the limits are enforced by Maidan, not by GitHub settings, and none of them is configurable:

• The only ref Maidan creates or moves is the target branch, and it must match feature/agent-[a-z0-9][a-z0-9-]*. prod, main, master, staging and dev are never written. A branch is never its own base.
• prod is never a base, for any repository; the allowlist refuses to bless it.
• A pull request is opened only into the base the allowlist names for that repository. An open pull request for the branch into any other base refuses the change before anything is written.
• Branch moves are fast-forwards (force: false), with one exception: a squash-rebase round moves an agent branch with force: true, only while its head is still the last commit the change flow delivered to its open pull request in that workspace, the pull request is not in the merge queue and the new commit sits on the base branch's current head. The head is re-read right before the move (Result Delivery). Maidan has no code that merges, marks a pull request ready, requests a review, deletes a branch or changes repository settings or protection.

A refused change is recorded as skipped with the reason on the delivery (GET /threads/:id/deliveries) and said in the Slack thread.

Seed the allowlist. Nothing is blessed by default, and nothing seeds it in code. Each blessing is one audited token:admin call, POST /workspaces/:wid/egress-targets, made by an operator per workspace. A github_branch selector is owner/name@base: the repository and the one base change pull requests into it may target (one row per base). A github row is the repository alone and allows result comments only, never commits. For example, with two repositories, one whose change pull requests target main and one that targets dev, and $SLACK_CHANNEL the channel id (C…) the replies go to, the seed is:

H1="Authorization: Bearer $ADMIN_TOKEN"; H2='Content-Type: application/json'
URL="$MAIDAN/workspaces/$WORKSPACE_ID/egress-targets"

# Commits and draft pull requests, one row per repository and base.
curl -sS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github_branch","selector":"example/skills@main"}'
curl -sS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github_branch","selector":"example/app@dev"}'

# Result comments on a pull request (the review loop). These allow no commits.
curl -sS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github","selector":"example/skills"}'
curl -sS -X POST "$URL" -H "$H1" -H "$H2" -d '{"surface":"github","selector":"example/app"}'

# The Slack channel the replies go to; a thread_ts needs no row of its own.
curl -sS -X POST "$URL" -H "$H1" -H "$H2" -d "{\"surface\":\"slack\",\"selector\":\"$SLACK_CHANNEL\"}"

Each call writes an egress_target.allow audit row. GET /workspaces/$WORKSPACE_ID/egress-targets lists the blessings and DELETE …/egress-targets/:tid revokes one; a revoked branch row stops a change that is already queued, because the allowlist is checked again before the write.

Agent observability (v76.0.0)

Scrape GET /metrics for agent-substrate health (see Integration). Gate e2e: agent_substrate_gate_e2e.rs.

Example Grafana dashboard (Prometheus datasource): docs/dashboards/maidan-operator.json (v89.0.0).

SLO alert templates (Prometheus / Alertmanager): docs/alerts/ (v90.0.0). CI executes them with promtool via scripts/check-alert-rules.sh — the promtool (alert rules) required check (v122.0.0); run that script locally to validate (it skips with a hint if promtool isn't installed).

Verify OTLP export end-to-end (v123.0.0): the otlp compose profile runs maidan-server against a real OpenTelemetry Collector (docker/otel-collector-config.yaml). ./scripts/otlp-smoke.sh brings up postgres + otel-collector + a server with OTLP_ENDPOINT/OTLP_METRICS=1, drives traffic, and asserts the collector received both a traces batch (incl. the per-request http_request span) and a metrics batch tagged service.name=maidan-otlp-smoke. Run it after touching the OTLP wiring or upgrading the OpenTelemetry SDK. CI runs it as the otlp smoke job.

Semantic scale: set MAIDAN_EMBEDDING_PROVIDER=openai-compatible in Helm prod values; run maidan reindex-embeddings --database-url $DATABASE_URL after provider changes.

Reindex jobs are durable (v104.0.0): POST /operator/reindex-embeddings records job status in maidan_reindex_jobs, so GET /operator/reindex-embeddings/:job_id resolves on any replica and survives restart. The job still runs on the replica that started it; if that pod dies mid-run the row stays Running — re-issue the (idempotent) reindex. App OAuth codes are likewise durable (maidan_oauth_codes): a code minted on one replica is exchangeable exactly once on any replica.

MCP request log

Every MCP request writes one info line on the maidan_mcp::request target, so the default MAIDAN_LOG keeps it. The line carries the method, the tool, the names of the arguments, the workspace, member, actor, app installation and delegation grant, the transport (http, streamable, stdio or slash), the profile, the latency and the outcome (ok, tool_error or error with its JSON-RPC code). It never carries an argument value or a result. The client chooses every name in a request, so only names the server declares are printed. A method it does not answer is (other), a tool not in the catalog is (unknown), and an argument name the tool does not declare is counted, not printed. To drop the line, add maidan_mcp::request=warn to MAIDAN_LOG.

Full frames are a developer tool, not a setting. Build with --features mcp-frame-capture and set MAIDAN_LOG=info,maidan_mcp::frame=debug to log each request and response. Secret-shaped keys are redacted at any depth, inside JSON result text too, and a tool that carries credentials (its name says secret, token, ticket or grant) has its arguments and result withheld. A release build that enables the feature fails to compile, so no published image can capture frames.

DNS rebinding

A page on another site can point its own name at this machine and send requests past the browser's same-origin rule, with Host and Origin naming that site. It holds no bearer token and gets none of this server's cookies, so it gains only what a request with no credential may do. That is everything under AUTH_DISABLED, and reading under MAIDAN_DEV_ANONYMOUS_MCP_WORKSPACE. In those two modes a request with no credential is refused (403) when its Host is a dotted name that is neither a loopback name (localhost or a name under .localhost) nor in MAIDAN_ALLOWED_HOSTS, and when it carries an Origin that is not its Host. A rebinding page presents a public name it controls, which always has a dot, so an IP address, a name without a dot (a compose service such as maidan) and a request with no Host pass. This holds however the server is bound, on every interface included. A public dev instance names itself in MAIDAN_ALLOWED_HOSTS. For the anonymous reader, a request with a credential is not judged by Host, since it takes the bearer path, so a reverse proxy that forwards a public Host keeps working. Under AUTH_DISABLED every request is judged, because no credential is checked there and a page can attach any header, so a proxy in front of such a server lists its name in MAIDAN_ALLOWED_HOSTS.

Search (GET /workspaces/:wid/search)

Semantic mode (v5.0.0): embeds q with MAIDAN_EMBEDDING_PROVIDER, then queries the per-model embedding table named by embedding_model (default: provider model_name()). Each hit includes embedding_model. /health reports embedding.model and embedding.dimension.

Rank field: higher is always better within a single response. Values are backend-specific for lexical search.

Score field (v48.0.0): normalized to [0, 1] within each response. Comparable across Postgres and SQLite for the same mode. Semantic score is cosine similarity; lexical score is min-max normalized rank.

Scale: use Postgres + pgvector HNSW for production semantic search. Large workspaces should use Postgres.

SQLite sqlite-vec (optional, v85.0.0): maidan-search builds without the extension by default; semantic search on SQLite uses in-process cosine ranking. Enable SQL vec_distance_cosine for dev parity:

cargo build -p maidan-server --features sqlite-vec

CI job sqlite-vec (optional feature) proves linkage when the feature is on.

After changing embedding providers, re-index or accept that old-model rows are ignored until re-upserted under the new model name.

Operator reindex (v87.0.0): POST /operator/reindex-embeddings enqueues a background job (202 + job_id). Poll GET /operator/reindex-embeddings/:job_id for running / completed / failed and processed / failed counts. Optional JSON body { "workspace_id": "<uuid>" } scopes to one workspace (workspace:write); omit workspace_id for all workspaces (operator:global; before, a workspace's token:admin could start and read an instance-wide job). CLI maidan reindex-embeddings remains for shell/CI. A job runs in-process on the replica that started it; its record is durable (see below), so a replica that dies mid-run leaves it Running.

Hosted on Cloudflare

dev.maidan.world runs main on Cloudflare Containers. A Worker (deploy/cloudflare/) takes every request and hands it to one Maidan container, which sleeps after 15 minutes without a request and starts again on the next one. Nothing the server keeps lives in the container: Postgres is a Neon project (it scales to zero too) and artifacts are an R2 bucket through the S3 backend. A sleep or a restart drops only presence and open live connections, which clients reconnect.

Deploying

Every push to main that touches the server runs deploy dev (.github/workflows/deploy-dev.yml): it builds the linux/amd64 image with a warm cache, pushes it to the account's registry, points wrangler.jsonc at it and deploys, then waits until /health reports the build it deployed (main-<first 8 of the commit>). The job reads CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from the dev environment, which only main may use; the token is scoped to Workers scripts, containers and the maidan.world zone's routes. A pull request that changes the deploy builds the image only.

The very first deploy of an instance creates the Durable Object and the container application together, and the application can fail to find the namespace (DURABLE_OBJECT_NOT_FOUND). The second attempt finds it, so the job retries once.

Without local Docker, push a CI-built image with crane: download the maidan-server-image artifact, get a short-lived push credential from POST /accounts/<id>/containers/registries/registry.cloudflare.com/credentials ({"expiration_minutes": 30, "permissions": ["push", "pull"]}), crane push the tarball to registry.cloudflare.com/<account id>/maidan-server:<tag>, and deploy with that reference as image.

From a checkout, npx wrangler deploy in deploy/cloudflare/ builds the image from source with local Docker instead, which is slow on a machine that is not linux/amd64.

Secrets

The server's secrets are Worker secrets, set once and kept across deploys:

cd deploy/cloudflare
for name in DATABASE_URL MAIDAN_CONTENT_KEK MAIDAN_SESSION_SECRET \
            S3_ENDPOINT S3_ACCESS_KEY_ID S3_SECRET_ACCESS_KEY; do
  npx wrangler secret put "$name"
done

The Worker passes only the names it lists in src/index.js to the container, plus the fixed settings there (MAIDAN_ENV=production, JSON logs, the bind address and the proxy hop). Plain settings (the public origin, the bucket) are vars in wrangler.jsonc.

The first admin

maidan init writes through the store, so it runs from any machine that can reach Neon, with the same DATABASE_URL and MAIDAN_CONTENT_KEK as the Worker secrets (Bootstrap).

What a sleeping instance means

• The first request after a sleep waits for the container to start and the server to boot against Neon, which may itself be waking.
• Any open request keeps the container awake: a WebSocket, an SSE stream or MCP's notification stream counts until it closes, and the container sleeps 15 minutes after the last one does. An agent that stays connected keeps its instance running, and is billed as running.
• Work a sleeping server would have done waits for the next request or the hourly cron.
• Any request wakes it, a crawler's too. Scanners find a new domain within hours of its certificate and probe for /.env, /.git/config and PHP files; the Worker answers paths no Maidan route serves with 404 itself (src/scanner.js), so those probes never reach the container.

Helm (production)

Charts under helm/maidan (server) and helm/maidan-stack (the server plus an optional Postgres and MinIO of its own).

Layer profiles as needed; see helm/maidan/PROFILES.md for example helm upgrade commands (v88.0.0).

Production refusals. values-prod.yaml sets production: true, and the chart then refuses to render: a development image (image.repository: maidan-server, or a tag of dev, latest or empty without image.digest); and, unless existingSecret names a Secret holding DATABASE_URL and MAIDAN_CONTENT_KEK, an unset secrets.DATABASE_URL, the development default postgres://maidan:maidan@postgres:5432/maidan, or any empty secrets value. config values, image.tag and image.digest holding CHANGE_ME fail every render. secrets and contentKek values holding CHANGE_ME fail when existingSecret is unset. Each refusal names the value to set. maidan-stack/values-prod.yaml sets the same flag (maidan.production) and pins the same release.

cert-manager: install cert-manager and a ClusterIssuer, create the maidan-secrets Secret (DATABASE_URL, MAIDAN_CONTENT_KEK), then:

helm install maidan ./helm/maidan \
  -f ./helm/maidan/values-prod.yaml \
  -f ./helm/maidan/values-cert-manager.yaml \
  --set existingSecret=maidan-secrets \
  -n maidan --create-namespace

values-cert-manager.yaml alone refuses to render: values-prod.yaml is what names the release image.

CI validation: ./scripts/helm-template-smoke.sh and ./scripts/helm-install-kind-smoke.sh (kind + Docker).

Pin by digest. image.digest: sha256:… renders the reference as repository@sha256:…, which a re-pointed tag cannot change; image.tag is then informational. Find a release's digest with docker buildx imagetools inspect ghcr.io/david-engelmann/maidan-server:<tag>, and verify its signature first (README, "Prebuilt image").

Set secrets.DATABASE_URL in values (not a MAIDAN_ prefix), or name an existingSecret that already holds it. Rendering does not check that Secret or its keys.

The chart mounts every secret as a file under /run/secrets/maidan and names it in <KEY>_FILE, so no secret is in the pod's environment. An existingSecret is mounted with the keys in secretFiles.existingSecretKeys, and a key the Secret lacks stops the pod. k8s/base does the same for maidan-secrets. See helm/maidan/README.md, "Secrets are files".

The umbrella stack's own stores. maidan-stack can run Postgres and MinIO as single-replica StatefulSets of its own (postgresql.enabled, minio.enabled; values-prod.yaml enables both):

• Postgres runs ghcr.io/david-engelmann/maidan-postgres (docker/Dockerfile.db, built on pgvector/pgvector, so migration 0003's CREATE EXTENSION vector works), pinned to the same release as the server; the Service is <release>-postgresql.
• MinIO runs cgr.dev/chainguard/minio by digest, the build compose and k8s/ use; the Service is <release>-minio. A post-install and post-upgrade Job creates minio.defaultBuckets with cgr.dev/chainguard/minio-client.
• The server reaches them through a Secret the stack renders, <release>-datastores: DATABASE_URL, and ARTIFACT_BACKEND=s3 with the S3_* settings, overriding maidan.secrets and maidan.config. With maidan.existingSecret, the Secret you create needs only MAIDAN_CONTENT_KEK.
• A production render refuses an empty or development password for either store (postgresql.auth.password, minio.auth.rootPassword) and a Postgres or MinIO image without a release tag or digest.

Moving an existing release from the earlier Bitnami subcharts to these StatefulSets is a replacement, not an in-place upgrade. The StatefulSet selector labels differ (Kubernetes rejects changing them), and MinIO's volume claim is data-<release>-minio-0, not Bitnami's <release>-minio. Dump the database and copy the buckets (mc mirror) out, install the new release, and restore; keep the old volumes until you have checked the restore. Postgres reads POSTGRES_PASSWORD only when it first creates the data directory, so changing it later does not change the database's password: run ALTER ROLE on the live role before changing postgresql.auth.password and upgrading. Changing a store password does not restart the server: run kubectl rollout restart deployment/<release>-maidan after the upgrade so it reads the new DATABASE_URL. A cluster-connected helm upgrade restarts MinIO when its root user or password changes, because the pod template carries a hash of a rollout nonce from the Secret, not of the password, and that nonce changes only when the live credentials do. helm template cannot see the Secret, so it renders one stable nonce: applying those manifests again does not roll MinIO, and a credential change there does not either. Restart it with kubectl rollout restart statefulset/<release>-minio.

minio.persistence.enabled, size and storageClass cannot change on an existing StatefulSet. Kubernetes rejects an update to volumeClaimTemplates, and a cluster-connected upgrade refuses the change before sending it. Copy the buckets out (mc mirror), delete the StatefulSet with --cascade=orphan, delete PVC data-<release>-minio-0 when the new pod needs a new volume, and upgrade again. The chart README has the steps. Growing a volume, when the storage class allows it, is a change to that PVC; leave size as it is. This is not the Bitnami replacement above.

The stack's Postgres has no replicas, backups or PITR; for those, run your own (or a managed) Postgres, turn postgresql.enabled off and set DATABASE_URL as for the server chart. The install command and the full list are in helm/maidan-stack/README.md. The helm install (kind) CI job installs the stack with both stores and waits for /health/ready.

Horizontal scaling (v105.0.0)

Maidan runs as N stateless replicas behind a load balancer with no session affinity — a request may land on any replica. The scale compose profile (docker compose --profile scale up) and the scale-out smoke CI job exercise this with two replicas + an nginx round-robin LB; scripts/scale-out-smoke.sh drives the cross-replica REST paths.

Shared across replicas (one of each):

Still pod-local (do not assume cross-replica):

• In-flight MCP streamable sessions and open WebSocket/SSE subscriptions live on the replica that holds the connection; a reconnect may land elsewhere and resumes from the durable cursor, not in-memory buffer.
• The outbound retry budget (10 retries at once, then 2 a second, per destination host) is held per replica, so N replicas give a recovering host N budgets (Outbound timeouts and the retry budget).
• A running reindex job executes on the replica that started it; only its status is durable and queryable from any replica. If that replica dies mid-run the row stays Running — re-issue the (idempotent) reindex.

Rolling updates / boot: every replica runs migrations on boot, serialized by a Postgres advisory lock (v105.0.0) so concurrent starts against a fresh or upgrading database don't race on DDL. A surge (maxUnavailable: 0) starts the new binary, which migrates, while the previous binary is still serving. That overlap follows Migrations: the migration that runs then only expands, and a drop, rename, retype, or rewrite is a later contract, after the previous binary is gone. A migration that cannot expand is a cutover: stop the previous binary before the new one migrates. There is no HTTP or MCP compatibility promise before the product's own 1.0 gate (Decisions, F-54). /health/ready gates traffic on the database, the object store, the indexer, and the LISTEN bus, so a load balancer that honors readiness does not route to a replica mid-migration.

Not covered: load/throughput benchmarking (bench harness), autoscaling/HPA tuning, multi-region active-active (out of scope).

Read replicas (v264.0.0)

Maidan can offload reads to a Postgres streaming read replica, with a causality token that guarantees a client never reads staler than its own writes.

Enable it. Set MAIDAN_DB_REPLICA_URL to a hot-standby's connection string. The server connects it at boot (fail-fast on a bad URL) and a background task polls the standby's replay position every 200 ms, so each read's primary-vs-replica choice is a cheap in-memory compare (no extra round-trip). Unset → every read uses the primary (unchanged).

The consistency token. A successful mutating request returns a Maidan-Consistency-Token response header (the primary's WAL LSN at that point). A client that wants read-your-writes echoes it on a later request as the Maidan-Consistency-Token request header. That read is served from the replica only once the replica has replayed past the token; until then it falls back to the primary. A read with no token may be served from the replica immediately (the caller has asserted no causality requirement).

Not Maidan-Room-LSN. That header is your workspace's event-log high-water (decimal, including SQLite) so a subscriber or webhook consumer can see projector / broadcast lag. It is scoped to the caller's room rather than the instance — the instance head was not comparable to anything a client had seen, so a caught-up consumer could never reach it. It is not a WAL LSN, is not gated on a replica, and must not be echoed as Maidan-Consistency-Token. See Integration.md (subscribe).

What routes, and what never does.

• Routed (only for GET/HEAD): content and collaboration reads — messages, threads, channels, members, DMs, social (votes/reactions/pins/mentions), notifications, follows, skills, assignments, dependencies, queue depth, and usage. Message search (v271.0.0) routes too: maidan-search's PostgresSearch has its own replica reader pool + replay poller and honors the same token via the same routing logic, so GET …/search reads-your-writes and offloads to the replica identically (embedding writes / index DDL / reindex stay on the primary). Its primary/replica split is counted separately as maidan_search_replica_reads_total (v272.0.0).
• Always the primary: every write; auth-path reads (sessions, API tokens, OIDC, federation peers) — the auth middleware runs on GETs, so a just-minted credential must be read fresh; control-plane/config reads (webhooks, slash commands, FSM hooks, deliveries, reindex jobs, audit, token quotas); and any read inside a mutation handler (those requests are never in a read-routing scope, so a read-then-write decision is always on primary data).

Observability. maidan_replica_reads_total{outcome="primary"|"replica"} counts the store split and maidan_search_replica_reads_total{outcome} (v272.0.0) the search split; maidan_replica_lag_bytes is the replica's WAL lag (primary write LSN minus replica replay LSN, shared by both). Complement with Postgres's own pg_stat_replication.

Testing. scripts/replica-harness.sh up stands up a local pgvector primary + streaming standby and prints MAIDAN_PRIMARY_URL / MAIDAN_REPLICA_URL; the #[ignore]d read_routing / replication store tests validate store routing and read-your-writes against it (cargo test -p maidan-store --test read_routing -- --ignored), and the #[ignore]d replica_routing search test proves the same for message search (cargo test -p maidan-search --test replica_routing -- --ignored).

Event-log chain verification

Every event carries a prev_hash / content_hash chain. Verifying it end to end is GET /workspaces/:wid/events/verify, which is streaming rather than whole-log-in-memory.

The search tap used to re-walk and re-verify the entire log on every process start — which meant verification happened only when a process happened to bounce. That is not a control you can schedule, alert on, or say when it last ran, so the tap now resumes from a cursor and verification became an explicit job:

MAIDAN_CHAIN_VERIFY_SECS=86400   # daily; unset = off

It walks every workspace that has events, continues past a break so one tenant's tamper does not hide the others, and emits maidan_chain_verify_total{outcome="ok"|"broken"|"error"}.

Alert on broken, investigate error separately. broken means a chain verified and failed — a tamper or corruption. error means the verification could not run, which is usually a database problem. An alert that cannot tell them apart gets ignored.

A broken chain is logged with its workspace_id, break_at and reason; the recovery path is a rebuild from the domain tables, not a repair of the log.

Backup & disaster recovery (v260.0.0)

Maidan's durable state is two things, and the backup story follows the same split:

Two operator scripts implement it:

• scripts/backup.sh [BACKUP_DIR] — pg_dump (custom format), or for SQLite a VACUUM INTO snapshot (below), plus, for localfs, a tar of ARTIFACT_LOCALFS_ROOT; writes a MANIFEST.txt. For s3, the bucket is the durable copy — enable bucket versioning and/or cross-region replication there rather than copying blobs into the backup.
• scripts/restore.sh <backup-dir> [--force] — pg_restore into the target DATABASE_URL, or for SQLite the snapshot put in place of the file (+ untar artifacts). It refuses a non-empty target unless --force, so a restore can't silently clobber a live database; --force restores with --clean --if-exists.

Not in the data backup — restore these from your secret manager, out of band: DATABASE_URL, MAIDAN_SESSION_SECRET (subscribe-resume/session signing), FEDERATION_ENCRYPTION_KEY (+ any FEDERATION_DECRYPT_KEYS — see the rotation keyring), MAIDAN_EXPORT_SIGNING_KEY (and any MAIDAN_EXPORT_VERIFY_KEYS pin), MAIDAN_CONTENT_KEK (+ any MAIDAN_CONTENT_KEK_PREVIOUS; without it no message words can be read), and SMTP/OIDC credentials. A DB dump without the session secret still restores all data; only signed-token continuity needs the same secret.

RPO / RTO. A periodic backup.sh (e.g. hourly cron) gives an RPO of one backup interval. For a tighter RPO, run Postgres with WAL archiving / PITR (below, or a managed Postgres with continuous backup) — the logical dump is the portable floor, not the lower bound. RTO is a restore.sh run plus a /health/ready check before the load balancer is pointed at the restored instance.

Recovery outline. Provision Postgres + the artifact store → set the out-of-band secrets → DATABASE_URL=… ARTIFACT_LOCALFS_ROOT=… scripts/restore.sh <dir> --force → start one replica and confirm /health/ready is 200 (it gates on DB + object store + indexer + the LISTEN bus) → scale out. Because artifacts are content-addressed, a message referencing a blob that predates the artifact backup is still consistent after restore; a blob written after the last artifact archive is the only thing a stale artifact backup can miss.

SQLite

Do not copy a live SQLite file. In WAL mode (which Maidan sets) recent writes sit in maidan.db-wal until a checkpoint, so a copy of the file alone misses them, and a copy taken during a write can catch a torn page.

Back up with the server running: DATABASE_URL=sqlite:///data/maidan.db scripts/backup.sh runs VACUUM INTO, which writes a consistent, compacted snapshot (maidan.sqlite) from one read transaction, waiting out a write in progress, and checks it with PRAGMA integrity_check. It needs the sqlite3 CLI. Take it on a schedule; the RPO is the interval, as for pg_dump.

Restore with the server stopped: DATABASE_URL=sqlite:///data/maidan.db scripts/restore.sh <backup-dir> --force checks the snapshot, puts it in place of the file, and deletes the old -wal and -shm. Deleting them is not housekeeping: a -wal left from a server that was killed (or beside a file someone removed) is replayed over whatever file has that name the next time it is opened, and the restored database comes back as the old one, or corrupt. Without --force it refuses a target that already has tables, or that is not a readable SQLite database; with --force it does not open the target, so it replaces a corrupt file too. The restored file keeps the owner and mode of the file it replaces. A new target belongs to whoever runs the script, so run it as the server's user (or chown the file after). A path in DATABASE_URL is percent-decoded, as the server decodes it (%3F is a ? in the file name). Start the server, confirm /health/ready, and it migrates forward if the snapshot is older than the binary.

The drill. scripts/sqlite-backup-drill.sh takes a snapshot while a writer is inserting, then restores it over a killed server's database, over an orphaned -wal and over a file that is not a database, and to a percent-encoded path, and fails unless exactly the snapshot's rows come back. sqlite_backup (a store test) shows a snapshot of a migrated Maidan database passes integrity_check and foreign_key_check and opens and migrates as it stands. CI runs the drill as sqlite backup drill.

Point-in-time recovery

With WAL archiving on, a base backup plus the archive restores the database to any moment since that backup, not only to the last dump.

Enable it. Set wal_level=replica, archive_mode=on and an archive_command that copies each finished segment somewhere durable, and set archive_timeout so a quiet database still closes a segment at least once a minute. compose.pitr.yaml does this for the compose stack, archiving to a volume. In production, point archive_command at storage off the database host, through WAL-G, pgBackRest or a managed service: an archive on the same disk survives a bad deploy, not a lost disk. Take a base backup (pg_basebackup -X none -c fast) when you enable it and on a schedule; recovery replays WAL from the most recent one.

Restore to a moment.

1. Stop the server and put the base backup in a fresh data directory.
2. Create recovery.signal in it.
3. Start Postgres with restore_command = 'cp /archive/%f %p' (or your tool's fetch command), recovery_target_time = '<timestamp with zone>' and recovery_target_action = promote.
4. When SELECT pg_is_in_recovery() returns false, the database is at the target and writable. Start one replica, confirm /health/ready, then scale out.

Artifacts are content-addressed, so a restored message still names the blob it had. Restore the artifact store to a point no earlier than the database target.

The drill. scripts/pitr-drill.sh [image] runs this whole procedure with Docker. It writes a row, notes the time, writes another, restores to that time, and fails unless exactly the first row is back. CI runs it on every PR against the maidan-postgres image built from the tree.

Signed workspace export

A tenant portability file is not a pg_dump. GET /workspaces/:id/export (token:admin) writes a maidan.workspace.export/1 Ed25519 envelope a fresh GHCR instance can verify with POST /workspaces/export/verify and import with POST /workspaces/import — no callback to the origin host.

Tokens die on export. The bundle omits API tokens and secrets. After import, mint new tokens (token:admin). A pg_dump restore does preserve hashed tokens (same DATABASE_URL / same instance); a signed export is a different machine and must not.

Set MAIDAN_EXPORT_SIGNING_KEY on the origin. On a destination that should accept only your key, set MAIDAN_EXPORT_VERIFY_KEYS to that public key (GET /operator/export-public-key on the origin). A blank instance with neither key still verifies integrity (tamper-evident). See Integration.md.

Crypto-shredding

A message's words (body, metadata, content blocks) are encrypted with a key of their own (XChaCha20-Poly1305) before the event is hashed. Withdrawing the message destroys that key, so the words are gone from the event log, admin and peer catch-up, exports, snapshots and search, while the hash chain and signatures still verify. Workspace purge destroys every key in the workspace; its audit row counts them (metadata.content_keys_destroyed). A replica that ingests the origin's tombstone shreds its copy. Under a legal hold the preserved copy keeps the words, as described below.

Each workspace has its own key. A message's key lives in maidan_content_keys, wrapped by its workspace's key; the workspace keys live in maidan_workspace_keys, each wrapped by MAIDAN_CONTENT_KEK, the instance KEK. So one workspace's key opens nothing of another's, erasing or purging a workspace deletes its keys, and rotating a workspace's key rewraps only key rows. An operator holding the database alone opens nothing: the workspace keys need the instance KEK, which is not in the database. A server holds a workspace key in memory for at most five minutes after it last used it, and only while the stored row matches what it opened.

Message keys written before workspace keys existed are wrapped by the instance KEK directly. They still open, and the startup pass moves each under its workspace's key.

The server refuses to start without the KEK. The Helm chart refuses to render without contentKek or an existingSecret holding MAIDAN_CONTENT_KEK, and the k8s base reads it from maidan-secrets. Only an explicit MAIDAN_ALLOW_INSECURE_DEV_KEK=1 (the compose files and local development set it) falls back to the built-in development key, which is public; production refuses that flag.

A withdrawal also deletes the notification mail about the message, sent or pending, and a notification about a withdrawn message is never queued. (Mail bodies name only the notification kind and event number, never the words; a send already in flight completes.)

Check for leftovers. maidan verify-shredding --database-url … [--workspace-id …] reads every withdrawn message and lists any copy of its words still outside the sealed event log: a message row not blanked, earlier versions (kept on purpose under a legal hold), an unsealed event payload, a queued webhook, egress or mail copy, a search entry or an embedding. It exits non-zero when it finds one.

Rotate a workspace's key. maidan rotate-workspace-key --database-url … --workspace-id <id>, with the server's MAIDAN_CONTENT_KEK, retires the workspace's key, makes a new one, moves the workspace's message keys under it, destroys the old key and writes a workspace_key.rotate audit row. No sealed message is rewritten. Running servers pick up the new key on their next read.

Rotate the instance KEK. Generate a new key (openssl rand -hex 32), set it as MAIDAN_CONTENT_KEK, move the old one to MAIDAN_CONTENT_KEK_PREVIOUS, and roll the replicas. Each rewraps the workspace keys still under an old KEK at startup: one row per workspace, not one per message. Remove the old KEK once no key needs it:

SELECT count(*) FROM maidan_workspace_keys
 WHERE wrapped_key IS NOT NULL AND wrapping_key_id <> '<new kek id>';
SELECT count(*) FROM maidan_content_keys
 WHERE wrapped_key IS NOT NULL AND workspace_key_id IS NULL;

The KEK id is logged at startup. A key wrapped by a KEK the server does not have fails the read with a 500; it is never shown as withdrawn.

Backups. A database backup taken before a withdrawal or an erasure still holds those keys. Anyone with that backup and the instance KEK can read the words. Keep the KEK out of data backups, and expire backups within your erasure deadline.

Artifacts. Artifacts are deduplicated across workspaces. DELETE /artifacts/{sha} (token:admin, audited as artifact.erase) removes the calling workspace's reference; the bytes are deleted only when the last workspace lets go (last_reference: true). An upload of the same bytes waits for an erase in progress and then writes the bytes back, so a new reference never points at deleted bytes; workspace purge uses the same protocol.

Legal holds

A legal hold stops a workspace's data from being destroyed while litigation is pending or reasonably anticipated (FRCP 37(e); GDPR Art. 17(3)(e) exempts such data from erasure). Holds are per matter: place one for each, and lift each on its own, as a workspace admin (token:admin):

POST /workspaces/{id}/legal-holds
{"reason": "matter 2026-114"}
GET /workspaces/{id}/legal-holds
DELETE /workspaces/{id}/legal-holds/{hold_id}

While a workspace has any hold:

• workspace purge and erase, message purge, artifact erase, and an import that replaces the workspace are refused (409), inside the destroying transaction;
• its event-log rows and its audit rows are exempt from retention pruning, and its own retention policy prunes nothing. Other workspaces' rows, and instance-level audit rows, still prune;
• a withdrawn message keeps its words. When a member (or a moderator) tombstones a message, it disappears for everyone exactly as it would unheld, but its last body and every earlier version are kept. Nothing in the product tells the member the words were kept.

Read what the holds kept with GET /workspaces/{id}/legal-holds/preserved (token:admin): one entry per withdrawn message, with its author, thread, channel, times, last words and earlier versions. Each read writes a legal_hold.preserved_read audit row first; if the row cannot be written, nothing is returned. Listing the holds is token:admin too: the members a hold binds are not shown that it exists.

Lifting one matter's hold releases nothing while another stands. Lifting the last disposes of what the holds kept: the preserved bodies, and the earlier versions of messages withdrawn while held. That lift's audit row records how many of each (metadata.disposed). Without a hold, a withdrawal takes the message's earlier versions with it.

Retention

Nothing but unreferenced media is pruned unless you set a retention, or a workspace sets its own. Each knob is an age in days; the sweeper runs every MAIDAN_RETENTION_SWEEP_SECS (default 86400) and deletes in batches of MAIDAN_RETENTION_BATCH (default 5000), so a first sweep over a long-unpruned table does not lock it. Each sweep counts what it deleted in maidan_retention_pruned_total{table} (events, audit, notifications, deliveries, messages and media_artifacts).

Partitioned event log (Postgres)

On Postgres the event log, maidan_events, is partitioned by calendar month (UTC) of occurred_at (migration 0162), as maidan_events_pYYYYMM. The server keeps the current month and three more ready, at boot and at the start of each retention sweep. A DEFAULT partition, maidan_events_default, takes any row no month covers: one dated more than three months ahead, or one older than the oldest month still kept. When a month is created, the rows DEFAULT holds for it move into it. The rows the table had before 0162 stay where they were, in one partition, maidan_events_legacy.

With MAIDAN_RETENTION_EVENTS_DAYS set, a sweep drops a month whole when the month ends by the cutoff and nothing in it is above the delivery-cursor floor or in a held workspace. Other partitions are deleted from in batches, as before: a month that doesn't qualify, the month the cutoff falls in, _legacy and DEFAULT. A sweep waits up to five seconds for the table lock a drop needs, and deletes instead if it can't get it. Each partition has its own autovacuum settings (vacuum and insert-vacuum at 5%, analyze at 2%). Autovacuum never analyzes the partitioned parent itself, so after a large import run ANALYZE maidan_events by hand. To see the partitions:

SELECT inhrelid::regclass, pg_get_expr(c.relpartbound, c.oid)
FROM pg_inherits JOIN pg_class c ON c.oid = inhrelid
WHERE inhparent = 'maidan_events'::regclass;

A row in DEFAULT normally means maintenance hasn't run since its month came due. Check the sweep's logs for retention: partition maintenance failed.

Audit rows belong to the workspace stamped on them when they were written. On upgrade, migration 0123 backfilled older rows from what they reference: a workspace target, metadata.workspace_id, the target's own row (token, member, channel, thread, message, share ticket, grant, egress target, app installation, result delivery, reindex job, hold), then the actor's or subject's membership. A row none of those resolved (its members and target since erased) has no workspace: it is instance-level, shown only in GET /operator/audit, and pruned under the instance cutoff whatever holds stand. Count them with SELECT count(*) FROM maidan_audit WHERE workspace_id IS NULL.

MAIDAN_RETENTION_NOTIFICATIONS_DAYS is off when unset. It deletes read notifications older than that many days and leaves unread ones, snoozed ones and a held workspace's rows. The usage ledger is not pruned.

A delivered egress row goes with its dedup key, so the same source event could be queued again only if the router replayed an event older than the cutoff. It does not: it starts from the log head and replays only across a subscriber lag. The external comment or message a result delivery edits in place is tracked on maidan_result_deliveries, which retention does not touch.

Per-workspace retention

A workspace administrator can set a shorter retention for that workspace's messages, events and finished deliveries with PUT /workspaces/{wid}/retention (token:admin, audited; see Integration). The instance knobs above are the ceiling: a workspace value longer than the instance's for that kind is refused, and if you later lower an instance knob below a workspace's value, the instance's applies. MAIDAN_RETENTION_MESSAGES_DAYS is that ceiling for messages. When it is set, the instance sweep erases messages past it in every workspace that is not held, whether or not the workspace set a policy. When it is unset, only a workspace policy prunes messages, and that policy may be any value from 1 to 3650 days. A workspace may set a shorter messages_days and not a longer one. After the instance sweep, each sweep prunes every workspace that set a stricter policy past its own cutoff; a held workspace keeps everything. Message pruning erases a message the way a purge does, including its embeddings, references and content key. Policies are stored in maidan_retention_policies (migration 0134) and removed with their workspace. A workspace may also set a shorter media_days and a lower media_quota_bytes (migration 0170), held to MAIDAN_RETENTION_MEDIA_DAYS and MAIDAN_MEDIA_QUOTA_BYTES the same way.

Media retention and quota

Attachments, screenshots and recordings are media. They fill storage fast, and much of what an agent uploads is never pointed at again, so media expires once nothing uses it, and each workspace has a media quota. Transcripts, code dumps and context snapshots are evidence. Neither rule touches them.

What counts as a reference. A workspace's copy of a media artifact is referenced while either of these holds:

• a thread in that workspace links the sha (maidan_thread_artifacts, the thread's evidence), and the thread is not tombstoned
• a message in that workspace names the sha in its metadata, and the message is not withdrawn. The shapes are the ones the context assemblers read: a string artifact_sha256 or sha256, or an artifacts array holding sha strings or {"sha256": "..."} objects.

A withdrawn message has its metadata blanked, so it stops protecting the sha. A reference in another workspace protects only that workspace's copy.

The TTL. Each sweep erases a workspace's unreferenced media older than the workspace's TTL. The TTL is the workspace's own media_days when it set one, held to MAIDAN_RETENTION_MEDIA_DAYS, and otherwise MAIDAN_RETENTION_MEDIA_DAYS (30 days unless set, and 0 turns it off). The age counts from the workspace's latest upload of the bytes, so uploading the same screenshot again restarts the clock. Link the artifact to a thread or name it in a message to keep it past the TTL.

The erase. An expired copy is erased the way DELETE /artifacts/{sha} erases it. The workspace loses access, its share tickets stop granting the artifact, and an artifact.expire audit row with no actor is written in the same transaction. The bytes are deleted only when no other workspace references them. Each candidate is checked again inside the transaction that erases it, so a reference made after the sweep listed it keeps it. Expiry runs in batches of MAIDAN_RETENTION_BATCH.

The legal hold. A workspace under legal hold keeps all its media, whatever its age and whatever its policy says.

The quota. A media upload is refused when the workspace's media plus the upload would pass its quota. The quota is the workspace's own media_quota_bytes when it set one, held to MAIDAN_MEDIA_QUOTA_BYTES, and otherwise MAIDAN_MEDIA_QUOTA_BYTES (20 GiB unless set, and 0 lifts it). Usage is the size of each media sha the workspace references, counted once by the kind the workspace gave it. Bytes the workspace already references cost nothing to upload again, since dedup stores them once. Erasing an artifact frees its space at once. The refusal is a 409 problem with the numbers a client needs to free room:

{
  "type": "https://maidan.dev/problems/media-quota-exceeded",
  "title": "Media Quota Exceeded",
  "status": 409,
  "detail": "this upload of 60 bytes would take the workspace's media to 120 bytes, past its quota of 100 (60 used). Erase media it no longer needs, or raise the quota.",
  "used_bytes": 60,
  "quota_bytes": 100,
  "upload_bytes": 60
}

A single-shot POST /artifacts is refused before its bytes are written. A multipart upload is checked when it completes, and its bytes are then deleted unless another workspace holds them. MCP upload_artifact and complete_artifact_multipart apply the same quota and answer with an invalid params error naming the three numbers. The check reads usage before the upload writes, so uploads running at the same moment can each pass and together overshoot the quota by at most what they upload. The quota bounds storage. It is not an exact ledger.

Event-log hash chain

Every stored event is accompanied by {id, lsn, prev_hash, content_hash} (SHA-256, sha256:<hex>). lsn is the event-log id, not a WAL Maidan-Consistency-Token. The chain is hashed, not signed: a peer that has seen a prefix detects a splice or payload rewrite without trusting the host. Authorship of a wholly fabricated but consistent chain is the signed-export envelope above.

GET /workspaces/:wid/events/verify (workspace:read) walks the retained suffix and 409s (event-log-broken) on a break. Federation ingest checks origin hashes the same way before remap. After retention prune, verify the remaining suffix — snapshot catch-up of a dropped prefix is described below.

No extra env vars. See Integration.md and Threat-Model.md.

Log snapshot

A log snapshot (GET /workspaces/:id/snapshot) is a different artifact: a hashed checkpoint of the same content graph plus the retained event-log floor/head, so a peer can catch up after retention pruned the prefix. It is not Ed25519-signed (391 is authorship). include_graph=true is token:admin or a federation peer — the default header + graph_hash is enough to verify a later graph fetch. See Integration.md.

API stability

From v1.0.0, HTTP and MCP shapes are semver-stable. Pre-1.0 releases may break without migration shims.