TheProtocol — Operator Setup Manual

Audience: anyone standing up their own TheProtocol registry from the public repository. Status: authoritative. Where this manual and the Quickstart, Configuration or Federation pages disagree, this manual wins: it is the one checked against the source on the dates below. Last verified against source: 2026-06-28 (§2, §3 and §9 items 3, 5 and 8: 2026-09-14).

This manual covers the parts of a deployment that the code does not generate for you — the things an operator must decide, configure, and place by hand: deployment mode, environment + secrets, TLS/DNS, the public reverse proxy, and (only if you federate) the nginx mTLS sidecars and SPIRE identity bootstrap.

---

0. Pick your deployment mode FIRST

There are two supported topologies. They are different compose files with different container sets. Choose before you do anything else.

	Standalone (recommended to start)	Federated
Compose	scripts/distribution/docker-compose.standalone.yml	scripts/distribution/docker-compose.operator.yml
Containers	11 (plus the one-shot redpanda-init)	9
Your own Event Store	yes (local CQRS ledger)	no — writes to a parent frame's Event Store
Token minting	enabled (your treasury issues your own unit)	disabled (federated registries cannot mint)
SPIRE / nginx mTLS	none	spire-agent + cert-writer + nginx-federation sidecars
Needs anything from the network operator	no — fully self-serve / air-gappable	yes — a one-time onboarding step on the hub mints your SPIRE identity
Use it when	you want a sovereign, independent registry & token economy	you want your agents/AVT to interoperate with the existing federation

The two container sets are NOT the same nine. Standalone = registry, teg-layer, event-store, db, teg-db, es-db, redis, pgbouncer, redpanda. Federated = registry, teg-layer, db, teg-db, redis, pgbouncer, spire-agent, cert-writer, nginx-federation (no local Event Store; minting off).

⚠️ docker-compose.production.yml at the repo root is NOT an operator template — it is the maintainer's multi-frame mainframe (≈80 services: SPIRE server, OPA, sandbox tiers, cert-writers, per-service nginx, Frame-B/C cross-frame networks). Do not try to run it as an operator. Likewise theprotocol-orchestrator/ is a client-side ppao CLI for hiring agents, not a way to deploy a registry.

The rest of this manual is organized as: §1 prerequisites · §2 get the software · §3 Standalone walkthrough · §4 configuration reference (env, secrets, DB, TLS) — applies to both modes · §5 Federated add-ons (nginx sidecars + SPIRE) · §6 verify · §7 operations · §8 troubleshooting · §9 known issues.

---

1. Prerequisites

• Host: Linux x86-64, Docker Engine ≥ 24 with the Compose v2 plugin (docker compose, not docker-compose). Standalone needs ~3 GB RAM and ~2 vCPU to be comfortable; it boots in ~2 minutes.
• A domain (e.g. api.theprotocol.cloud) with an A record pointing at your host — required if you want browser/API access over HTTPS (see §4.4). Not needed for a purely local/LAN trial.
• Open ports: the registry container publishes plain HTTP on host :8000 by default. You will front it with your own TLS reverse proxy on 443 (§4.4). Only expose 443 publicly; keep 8000 bound to localhost or a private interface.
• git, openssl, bash on the host for cloning + secret generation.
• For federated mode only: you must first apply to the network operator and run a one-time onboarding handshake (see §5) — you cannot self-issue a federation identity.

---

2. Obtain the software

There are two ways to get the three application images (registry, teg-layer, event-store). Both end with the same compose files and env; only where the images come from differs.

2a. Pre-built images

There is no public image registry to pull the three application images from. The network's own image host answers only to the pull credentials it issues to federated operators at onboarding (§5); everyone else builds the images from the repository (§2b), which is one command.

2b. Build from source (the documented path)

The repository ships the build contexts. From the repo root, the three images are built from:

scripts/distribution/Dockerfile.registry.dist     <-  ./AgentVault
scripts/distribution/Dockerfile.teg.dist          <-  ./AgentVault/agentvault_teg_layer_mvp
scripts/distribution/Dockerfile.eventstore.dist   <-  ./agentvault-event-store

scripts/distribution/build.sh builds all three, proves that each one imports its application (a boot gate), and tags them theprotocol/{registry,teg-layer,event-store}:{VERSION,latest,stable}. That prefix is a LOCAL name, and it is what docker-compose.standalone.yml runs by default, so nothing is pulled from anywhere. --registry <host/namespace> (or IMAGE_REGISTRY) changes the prefix, and --push pushes, only to a registry you name. The registry image builds the Vue frontend itself in a node:20 stage (Docker with BuildKit is the only requirement), compiles the application to bytecode (com.theprotocol.distribution="bytecode-only"), and keeps alembic/ so migrations run on first boot (§4.3). The first build downloads the Python and npm dependencies and takes a while; later builds reuse Docker's cache.

---

3. Standalone setup: step by step

This is the fully self-serve path. No SPIRE, no nginx, no hub dependency.

# 1. enter the distribution dir of the kit you downloaded (this repository, unpacked)
cd scripts/distribution

# 2. build the three images (§2b)
./build.sh

# 3. generate your operator env + secrets (writes .env.operator)
./operator-setup.sh my-registry standalone
#    (this calls generate-operator-env.sh, which fills in random per-operator secrets)

# 4. edit .env.operator: set the values the generator cannot guess (see §4.1):
#    - MAIL_* (defaults are smtp.example.com / CHANGE_ME and WILL block prod-mode startup)
#    - your public BASE_URL / domain
#    - TRUST_DOMAIN: your DNS name (the generated <name>.local suits a local trial only); the registry signs its
#      card with it and mints its agents' DIDs under it
#    - REGISTRY_HTTP_PORT=<port> if 8000 is taken on this host; the kit's host ports are AUDITOR_HTTP_PORT (8045),
#      PROMETHEUS_HTTP_PORT (9090), GRAFANA_HTTP_PORT (3000) and ALERTMANAGER_HTTP_PORT (9093), all loopback only
nano .env.operator

# 5. launch
docker compose -f docker-compose.standalone.yml --env-file .env.operator up -d

# 6. wait ~2 min for the registry to pass its healthcheck, then grab the first-admin password
docker logs my-registry-registry 2>&1 | grep -A3 "FIRST-BOOT ADMIN PASSWORD"

# 7. open the API/UI
curl -s http://localhost:8000/health        # expect {"status":"ok"...}
#    then put TLS in front of it before exposing publicly  (§4.4)

# 8. log in as that admin and mint the registry's initial supply. A standalone treasury starts empty, and
#    until it holds tokens every new agent's genesis grant reports "partial":
#      POST /api/v1/teg/treasury/mint  {"amount": "1000000", "reason": "initial supply"}   (admin bearer)

# 9. no SMTP yet? verify a registrant from the admin rail instead of the mailed link:
#      POST /api/v1/admin/developers/{id}/force-email-verify

What just happened automatically (you do NOT do these by hand):

• generate-operator-env.sh created all app secrets (DB passwords, API_KEY_SECRET, TEG signing keys, internal keys) and set API_KEY_SECRET == TEG_REGISTRY_JWT_SECRET (they MUST match — §4.2).
• On first up, the registry container runs migration_bridge → alembic upgrade head → operator_seed → serve. Migrations create every table and seed the event_emission_policies rows (migration 0077). operator_seed creates your first admin admin@<slug>.dev with a random password printed once to the logs and re-seeds emission policies idempotently. There is no manual DB init, no manual key generation, no manual policy load.
• TEG and Event Store bootstrap their own schemas the same way (TEG via alembic; Event Store via create_all()).

You are responsible for (the "not in code" parts): §4.1 env you must set, §4.4 TLS/DNS front door, §7 backups, and rotating the first-admin password (step 5) before the registry is reachable from the internet.

---

4. Configuration reference (applies to both modes)

4.1 Environment variables you MUST set

The shipped */.env.example files are partial (the registry's lists about 20 variables; the stack reads about 40). In the operator path, generate-operator-env.sh produces a complete .env.operator; you only hand-edit the values it cannot generate. The categories:

Generated for you (random, per-operator) — leave as generated:PG_PASS_* (DB passwords) · API_KEY_SECRET / SECRET_KEY · TEG AVTP_SYSTEM_API_KEY, TOKEN_SIGNING_KEY, REGISTRY_JWT_SECRET · EVENT_STORE_INTERNAL_API_KEY. MINTING_AUTHORITY=teg-layer (standalone) / disabled (federated). GENESIS_GRANT_ENABLED=true, GENESIS_GRANT_AMOUNT=5000.0.

You must edit before going to production:

Variable(s)	Why
BASE_URL / PUBLIC_URL	Your real domain. Setting a non-localhost BASE_URL flips the app into production mode, which then requires the mail vars below or the registry refuses to boot.
MAIL_SERVER, MAIL_USERNAME, MAIL_PASSWORD, MAIL_FROM	Production-mode hard requirement (verification emails, magic links). Generator defaults are smtp.example.com / CHANGE_ME.
ALLOWED_ORIGINS / CORS	Must list your real front-end origin and must not contain * in production.
REGISTRY_SIGNING_KEY_PASSPHRASE (recommended)	If unset, the registry's auto-generated EdDSA card-signing key is stored plaintext in the DB (with a warning). Set it to encrypt the key at rest (HKDF→AES-GCM).

Fully optional / feature-gated off by default: all fiat rails (STRIPE_*, MOLLIE_*, FIAT_ONRAMP_ENABLED, FIAT_GENESIS_REGISTRY), monitoring (GRAFANA_ADMIN_PASSWORD), and the AGORA exchange / FX flags. Turn these on deliberately, later.

Frame identity (only relevant if you customize currency/role): FRAME_CURRENCY (absent ⇒ AVT), IS_CENTRAL_REGISTRY (federation-hub role; leave false for a normal operator), REGISTRY_NAME / REGISTRY_FEDERATION_NAME.

The authoritative full list lives in the compose file you run (every ${VAR} without a :-default is mandatory) plus the app config (AgentVault/agentvault_registry/src/agentvault_registry/config.py, …/agentvault_teg_layer_mvp/src/teg_core_service/core/secure_config.py).

4.2 Generating secrets by hand (if you are NOT using generate-operator-env.sh)

Generate each secret with:

openssl rand -hex 32        # or:  python3 -c 'import secrets; print(secrets.token_hex(32))'

Two load-bearing constraints the code enforces:

1. API_KEY_SECRET (registry) and TEG_REGISTRY_JWT_SECRET (TEG) MUST be identical. This is the shared service-JWT secret the registry uses to call the TEG. Mismatch ⇒ all registry→TEG calls 401.
2. TEG TOKEN_SIGNING_KEY and AVTP_SYSTEM_API_KEY must each be ≥ 32 characters (pydantic min_length=32). The app refuses to start otherwise.

API_KEY_SECRET must also not contain test/example/demo/sample in production. SPIRE/SVID certs and the registry EdDSA card-signing key are not env secrets — they are issued/generated by the stack at runtime.

4.3 Database bootstrap — automatic (do not do it by hand)

On first up, each service migrates itself via its compose command:. You do not run alembic, create databases, seed policies, or initialize a treasury manually.

• Registry: migration_bridge (safe re-stamp; no-op on a fresh DB) → alembic upgrade head (all tables + event_emission_policies seed in migration 0077) → operator_seed (first admin + idempotent policy re-seed) → serve.
• TEG: migration_bridge → alembic upgrade head → serve.
• Event Store: schema via SQLAlchemy create_all() at startup (append-only ledger; no admin).

4.4 TLS, DNS, and the public reverse proxy — YOUR responsibility

This is the single biggest gap in the older docs, and it is not in any operator compose: the registry container serves plain http://…:8000. There is no built-in 443 / certbot / browser-facing TLS in the operator stack. You must put your own reverse proxy in front:

1. Point your domain's A record at the host.
2. Run a TLS terminator (host nginx, Caddy, Traefik — your choice) on :443 that:
   • obtains a certificate (Let's Encrypt / your CA),
   • proxy_pass to http://127.0.0.1:8000 (the registry),
   • forwards Host, X-Forwarded-For, X-Forwarded-Proto.
3. Set BASE_URL/PUBLIC_URL (§4.1) to the https:// domain.
4. Do NOT enable ssl_verify_client (mTLS) on this public/browser vhost. That is only for federation sidecars (§5). Turning it on for the browser front door makes browsers pop a client-certificate picker and breaks normal users (this exact mistake is documented in the cert-prompt history; see §8).

The only TLS+certbot instructions currently in the repo (scripts/distribution/DEPLOYMENT_GUIDE.md) are for standing up a private image registry host — a different concern. There is no operator front-door doc; this section is it.

---

5. Federated add-ons — nginx mTLS sidecars + SPIRE

Skip this entire section for Standalone. You only need it to join the existing federation, and it cannot be completed without a one-time step performed on the network hub (see §5.4) because node identity is bootstrapped from a private x509pop CA the operator does not hold.

A whole frame on your own host (the sovereign path): scripts/distribution/frame-up.py stands up the full stack (SPIRE, the mTLS gateways, the ledger, the kit) under your own DNS name with no step on the hub, and frame-federate.py joins the network through the invitation rail (their admin approves, your licence is applied). The README section "Run a federated frame on your own host" is the short form; this section describes the single-registry operator that joins an existing frame.

5.1 What the nginx federation sidecar actually does

It is a TLS-terminating reverse proxy that fronts your registry + TEG for cross-frame federation traffic, and it does mutual TLS:

• Listens on :8443 (mTLS) and a plain :8080 health server. The federated operator compose publishes the mTLS port (e.g. 12443:8443).
• Server certificate = a SPIRE-issued X.509 SVID (never Let's Encrypt, never self-signed by you).
• ssl_verify_client on — it rejects any peer whose client cert does not chain to the SPIRE trust bundle. (Note: nginx verifies the chain to a trusted CA; the actual SPIFFE-ID/SAN check is done in the backend via the forwarded client PEM, core/mtls_auth.py. nginx alone only enforces "signed by a trusted CA.")
• Upstreams: federation/cross-TEG/well-known paths → registry:8000; TEG paths → teg-layer:8080.

The SVID is delivered to nginx by a cert-writer sidecar, not by nginx talking to SPIRE directly: cert-writer runs spire-agent api fetch x509 against the agent's Unix socket, writes /certs/svid.0.pem, /certs/svid.0.key, /certs/bundle.0.pem, appends federated peer trust bundles so you trust cross-frame peers, chmod 644s the key (so non-root nginx can read it), and refreshes every ~300 s.

5.2 What you configure on the nginx sidecar (very little)

Per-instance variation is env + the shipped SPIRE cert files, not hand-edited nginx. The shipped operator nginx-federation.conf has no variables (literal registry / teg-layer upstream aliases and fixed server_name). The only substituted value is TEG_UPSTREAM_HOST (= ${REGISTRY_NAME}-teg) in the optional nginx-teg.conf, rendered by envsubst in nginx-entrypoint.sh. Cert paths are always /certs/svid.0.* + /certs/bundle.0.pem. You do not edit server_name or cert paths. Your published mTLS host port (e.g. 12443) is the one knob you may remap.

5.3 SPIRE — x509pop attestation (the docs in identity-fabric/docs/ are WRONG)

The identity-fabric/docs/SPIRE_SETUP_GUIDE*.md files are MVP-demo docs and mislead operators. The real production scheme:

• Node attestation is x509pop, NOT join_token. Ignore every "generate a join token / export SPIRE_AGENT_JOIN_TOKEN" instruction in those guides — there is no join-token wiring in the real agent.
• A federated operator runs only a spire-agent (one per host). It does not run a spire-server — your agent x509pop-attests across to the hub's spire-server (spire.theprotocol.cloud:8444, trust domain of the parent frame, e.g. agentvault.com).
• The agent presents an x509pop cert that was CA-signed on the hub (the private attestation-ca.key). This is why federation is not self-serve — see §5.4.
• The SPIRE topology that does exist self-signs its SVID-signing CA (KeyManager "disk", no UpstreamAuthority); the generate_dummy_ca.sh / ca/ directory in the repo are vestigial.

Env that wires SPIRE into your registry/TEG (set in the federated compose, not by hand): SPIRE_TRUST_DOMAIN, TRUSTED_SPIFFE_DOMAINS (inbound peer acceptance), SPIRE_SERVER_ADDRESS/PORT, and the shared spire-agent-socket volume mounted read-only into registry/TEG/nginx at /opt/spire/sockets.

5.4 The onboarding handshake (one-time, runs on the hub)

To federate, the network operator runs scripts/distribution/onboard-operator.sh <your-name> on the hub. It:

1. generates your x509pop attestation cert (signs with the hub's private CA),
2. exports the hub's SPIRE trust bundle,
3. registers your SPIRE workload entries on the hub server (register-operator-entries.sh, keyed on your cert fingerprint + a docker-label selector),
4. writes your agent.conf, and hands you a spire/ package + the nginx-federation.conf/entrypoint to drop in.

You then place the spire/ and nginx/ files, run generate-operator-env.sh, and docker compose -f docker-compose.operator.yml --env-file .env.operator up -d.

Implication for the release: a third party with only the GitHub repo cannot join the federation unattended — it requires this hub-side step. Standalone has no such dependency. (See §9 for the design decision this raises.)

---

6. First-run verification

# containers healthy (standalone = 9)
docker compose -f docker-compose.standalone.yml --env-file .env.operator ps

# core health
curl -s http://localhost:8000/health
curl -s http://localhost:8000/api/v1/network/stats        # registry up + DB reachable

# supply invariant must read delta 0 (your sovereign ledger is conserved)
curl -s http://localhost:8000/api/v1/projections/supply-audit
#   -> {"delta": 0, "status": "OK", ...}

# log in as the first admin (password from the first-boot log banner, §3 step 5)
#   then immediately rotate it and create your real dev/agent accounts.

For federated mode, also confirm the federation health endpoint over mTLS succeeds and that your peer row was activated on the hub (the hub-side onboarding does this; you cannot self-activate).

---

7. Operations

• Backups: snapshot the Postgres named volumes (registry-data, teg-data, es-data) + Redis. The TEG DB is the source of truth for balances; the Event Store is the audit ledger. Never hand-edit balance columns — all token movement goes through the TEG API.
• Upgrades: pull the new image tag and up -d; migrations run automatically on boot. Keep the previous image tag to roll back. There is no git-based deploy — you deploy by image tag.
• Secret rotation: rotate API_KEY_SECRET and TEG_REGISTRY_JWT_SECRET together (they must stay equal). Rotate the first-admin password immediately after first boot.

---

8. Troubleshooting & known gotchas

• Browser shows a "select a certificate" prompt → you enabled ssl_verify_client on a browser-facing vhost. Set it to off on every public/registry/API vhost; keep mTLS only on the federation sidecar (:8443) and the Event Store federation vhost.
• nginx returns 502 after a backend container restarts → add resolver 127.0.0.11 valid=10s; to any proxy_pass $variable; block. Without it nginx caches the upstream IP at config-load forever.
• Federation TEG calls 403/Host errors → the Frame-A pattern forces proxy_set_header Host "teg-layer:8080" because the TEG enforces a trusted-host check; the operator nginx-teg.conf carries the same override. Don't remove it.
• SVID key unreadable / nginx won't start → cert-writer chmod 644s the key; if you tightened perms, nginx (non-root) can't read it. Also: nginx blocks up to 120 s waiting for /certs/.ready; if the SPIRE agent socket never appears, cert-writer exits after 60 s and nginx never starts — fix the SPIRE agent first.

---

9. Known issues / release-prep items (not operator-fixable)

These are tracked for the maintainer and listed here so operators aren't surprised:

1. Older internal documents carried credentials and host details; they are not part of this repository, and scripts/distribution/DEPLOYMENT_GUIDE.md no longer names a host.
2. Mainframe first-admin seeding is env-driven — resolved 2026-06. prod_seed.py reads FIRST_ADMIN_NAME / FIRST_ADMIN_EMAIL / FIRST_ADMIN_PASSWORD (password unset → a random one is generated and printed once to the logs). The operator path (operator_seed.py) was always correct. No hardcoded credentials remain in either seed path.
3. Image-host drift: resolved 2026-09-14. docker-compose.standalone.yml runs the images build.sh tags locally (theprotocol/...:stable); only the federated compose names the network's image host, whose pulls need the credentials issued at onboarding (§2a).
4. Standalone compose bugs: resolved (checked 2026-09-21): pgbouncer sets AUTH_TYPE and IGNORE_STARTUP_PARAMETERS, and the TEG runs under gunicorn. Still open: a harmless NameError warning path in operator_seed.py.
5. README quickstart: resolved 2026-09-14. It builds the images, generates the env, launches, mints the initial supply, covers the no-SMTP path and points operators here.
6. identity-fabric/docs/SPIRE_SETUP_GUIDE*.md document join-token attestation — the real scheme is x509pop (§5.3). These should be corrected or marked MVP-only.
7. Federation is not self-serve (§5.4): joining requires a hub-side onboard-operator.sh against a private CA. Decide whether to (a) document this as "apply to federate", (b) provide a self-serve enrollment, or (c) lead with Standalone and treat Federation as invite-only.
8. From-source vs pre-built-image: decided 2026-09-14. Building from source (§2b) is the operator path; there is no public image registry.

---

End of Operator Setup Manual.