Run canaries on your frame

A canary is a synthetic transfer that your frame makes to itself, on a fixed cadence, so that a broken lane tells you rather than a user. Each fire is a real transfer by a real agent over the rail your users use: a canary that passes is proof that minting a token, moving money, settling it and recording it all still work, end to end, right now.

Canaries move real currency on every fire. Read "What it costs, and how to keep it from draining your frame" before you pick amounts, especially on a frame whose supply is small.

What you need

• A frame stood up with frame-up.py, and its admin login (in the frame's secrets directory). A cluster of a frame plus its cloud operators works the same way, with one extra step for the operators.
• Your frame's database, for four INSERTs. There is no admin rail that creates canary agents or paths; the runner reads them from the registry's own database.
• The registry image V0424 or newer if you run more than one registry. From V0424 an unlisted agent federates (a peer holds an unlisted mirror of its card), which is how a cross-registry path's sender resolves the agent it pays. On an older image, canaries between registries cannot resolve each other.

The shape

Each frame is the cockpit for itself and for its own cloud operators:

• the frame schedules its own paths and dispatches its operators' paths to them over the federation channel (CANARY_RUNNER_MODE=master),
• an operator never schedules anything itself; it fires only what its frame dispatches (CANARY_RUNNER_MODE=follower), with its own agents and its own credentials, which never leave it.

An operator trusts its parent frame's identity and nobody else's, so a frame cannot be the cockpit for another frame's operators. Two federated frames with two operators each means two masters, not one.

Step by step

1. A service account owns the canaries, never your admin account

Create one developer per registry, called svc-canary, and give it exactly two flags:

# on each registry, as its admin
curl -sX POST "$REG/api/v1/auth/register" -H 'Content-Type: application/json' \
  -d '{"name":"svc-canary","email":"svc-canary@<your-frame>.dev","password":"<generated>"}'
curl -sX POST "$REG/api/v1/admin/developers/$DEV_ID/force-email-verify" -H "Authorization: Bearer $ADMIN_JWT"
curl -sX PUT  "$REG/api/v1/admin/developers/$DEV_ID/admin-flags" -H "Authorization: Bearer $ADMIN_JWT" \
  -H 'Content-Type: application/json' \
  -d '{"admin_treasury": true, "agents_unlisted_by_default": true}'

admin_treasury lets it fund its own agents from the frame's treasury. agents_unlisted_by_default makes every agent it mints unlisted: dark for discovery, absent from the public list, the explore page and the globe, while still resolvable by DID and still federated. A canary is a service agent; nobody should be able to find and hire it.

Ask the server for the developer id rather than reading it from the register response, which does not carry one.

2. Two agents per registry

Mint them as svc-canary, through the normal onboarding rail:

BT=$(curl -sX POST "$REG/api/v1/onboard/bootstrap/request-token" -H "Authorization: Bearer $SVC_JWT" \
     -H 'Content-Type: application/json' -d '{"agent_type_hint":"canary_agent"}' | jq -r .bootstrap_token)
curl -sX POST "$REG/api/v1/onboard/create_agent" -H "Bootstrap-Token: $BT" -H 'Content-Type: application/json' \
  -d '{"agent_did_method":"theprotocol","agent_host":"https://canary-<reg>-sender.agents.local"}'

.agents.local is a reserved name the card pull-sync never dials, which is what you want for an agent that serves nothing. Keep the client_secret from the response (it is shown once) in a file only root can read, and read the card back: if unlisted is not true, stop and fix step 1 rather than continuing with a listed canary.

Do this twice per registry: a sender and a receiver.

3. Prefund both agents

curl -sX POST "$REG/api/v1/teg/treasury/fund-agent" -H "Authorization: Bearer $SVC_JWT" \
  -H "Idempotency-Key: canary-prefund-<reg>-<short did>" -H 'Content-Type: application/json' \
  -d '{"target_agent_did":"<did>","amount":"25","reason":"canary_prefund"}'

Both, not just the sender: the local path runs in both directions.

4. The rows

In each registry's own database (canary_agents, then canary_paths), written inactive so nothing fires yet:

-- this registry's own two agents: real credentials, and localhost as the URL the runner dials
INSERT INTO canary_agents (agent_did, client_id, client_secret, registry_id, registry_url, frame, is_active, prefund_amount, notes)
VALUES ('<sender did>', '<client id>', '<client secret>', 'myframe', 'http://localhost:8000', 'myframe', TRUE, 25, 'svc-canary sender (dark)'),
       ('<receiver did>', '<client id>', '<client secret>', 'myframe', 'http://localhost:8000', 'myframe', TRUE, 25, 'svc-canary receiver (dark)');

-- a reference row for every agent on ANOTHER registry that a path of this one pays: no login happens as it
INSERT INTO canary_agents (agent_did, client_id, client_secret, registry_id, registry_url, frame, is_active, prefund_amount, notes)
VALUES ('<far sender did>', 'receiver-no-login', 'receiver-no-login', 'myoperator', 'myoperator', 'myoperator', TRUE, 0, 'payee on myoperator');

-- the paths: intra stays on this registry, 2pc crosses to another one
INSERT INTO canary_paths (path_id, sender_agent_id, receiver_agent_id, backend, amount, interval_sec, slo_settled_sec, is_active, notes, transport)
VALUES ('myframe-local',      <sender id>,   <receiver id>, 'intra', 0.1, 1800,  60, FALSE, 'canary: local intra', 'bearer'),
       ('myframe-local-back', <receiver id>, <sender id>,   'intra', 0.1, 1800,  60, FALSE, 'canary: the return leg', 'bearer'),
       ('myframe-to-myoperator', <sender id>, <far sender id>, '2pc', 0.1, 1800, 180, FALSE, 'canary: myframe->myoperator', 'bearer');

registry_url = http://localhost:8000 is load bearing: it is how the rebalancer recognises an agent homed on this node and funds it from this node's treasury. A cross-registry path should pay the far registry's sender (see the money section). Give a cross-registry path a longer settle SLO than an intra one.

5. The dispatch rows, on the frame only

For every path that lives on one of your operators, tell the frame to schedule it:

curl -sX POST "$FRAME/api/v1/admin/canary/fleet/remote-paths" -H "Authorization: Bearer $ADMIN_JWT" \
  -H 'Content-Type: application/json' -d '{"paths":[
    {"home_registry":"myoperator","remote_path_id":<the id in the OPERATOR database>,
     "path_label":"myoperator-local","interval_sec":1800,"is_active":false}]}'

home_registry must match the operator's name in the frame's peer registries: that is how the dispatcher finds it.

6. Probe every lane before you arm anything

As each path's sender, fire exactly what the runner fires, and watch the payee's balance on its own registry:

JWT=$(curl -sX POST "$REG/api/v1/auth/agent/token" -d "username=<client id>&password=<client secret>" | jq -r .access_token)
# intra
curl -sX POST "$REG/api/v1/teg/transfer" -H "Authorization: Bearer $JWT" -H 'Content-Type: application/json' \
  -d '{"receiver_agent_id":"<payee did>","amount":"0.1","message":"probe"}'
# cross-registry
curl -sX POST "$REG/api/v1/teg/cross-registry-transfer?backend=2pc" -H "Authorization: Bearer $JWT" \
  -H 'Content-Type: application/json' -d '{"receiver_did":"<payee did>","amount":"0.1","message":"probe"}'

A 200 that moves no money is a failure. Fix a lane here, where nothing is scheduled yet, rather than after it has become an alert that fires every half hour.

7. Arm, with the runners still off

curl -sX POST "$FRAME/api/v1/admin/canary/fleet/pause" -H "Authorization: Bearer $ADMIN_JWT"
# set is_active = TRUE on every canary_paths row, and on the frame's dispatch rows
curl -sX POST "$FRAME/api/v1/admin/canary/fleet/resume" -H "Authorization: Bearer $ADMIN_JWT"

8. Turn the switches on, as their own recreate

Put these in the registry's env file and recreate the registry alone (docker compose up -d --no-deps --no-build --force-recreate <service>). Roll your operators first, so an operator is judging results before its frame starts dispatching to it.

On a frame (a master):

FT_REG_CANARY_ENABLED=true
FT_REG_CANARY_RUNNER_ENABLED=true
FT_REG_CANARY_JUDGE_ENABLED=true
FT_REG_CANARY_VALIDATOR_ENABLED=true
FT_REG_CANARY_REBALANCER_ENABLED=true
FT_REG_CANARY_REFILL_ENABLED=true
CANARY_RUNNER_MODE=master
CANARY_REBALANCER_INTERVAL_SEC=86400
CANARY_REBALANCER_FLOOR_AVT=25
CANARY_REBALANCER_TOPUP_AVT=25
CANARY_REBALANCER_CEILING_AVT=150

On a cloud operator (a follower), the same with CANARY_RUNNER_MODE=follower and CANARY_REBALANCER_ENABLED=false: its frame asks it to refill its own senders, so it needs no rebalancer of its own.

A kit frame's compose maps the first six from their FT_REG_ names, which is why they carry that prefix and the rest do not. On any registry whose compose does not map them (a cloud operator, for instance), write the plain names. A value written in the compose environment: block always beats the env file.

9. Watch the first cycle

docker logs --since 5m <registry> | grep -E "CanaryRunner|CanaryJudge|CanaryRebalancer"
curl -s -H "Authorization: Bearer $ADMIN_JWT" "$REG/api/v1/admin/canary/status" | jq '.summary'
curl -s -H "Authorization: Bearer $ADMIN_JWT" "$REG/api/v1/admin/canary/fleet/status" | jq '{mode, fleet_paused}'

You want FIRED/DISPATCH lines, cycle done: fired=N errors=0, the judge's → passed, and your supply delta still 0.0. The dashboard is at /ui#/admin/canaries.

What it costs, and how to keep it from draining your frame

Every fire moves currency, and on a frame with a small supply a careless matrix bleeds the treasury into agents that never spend. Two rules keep it flat:

1. Make the money circulate. A cross-registry path pays the far registry's sender, and a local path runs both ways (-local and -local-back). Then every canary agent both sends and receives, each pair nets out, and only fees leave the loop. Pay a dedicated receiver on every leg instead, and that receiver grows forever.
2. Let the rebalancer hold a band. Once a day is plenty. Below CANARY_REBALANCER_FLOOR_AVT it tops an agent up by CANARY_REBALANCER_TOPUP_AVT from its own registry's treasury; above CANARY_REBALANCER_CEILING_AVT it returns everything beyond floor + top-up to that treasury (fee free, supply neutral, at most one sweep per agent per day). CANARY_REBALANCER_CEILING_AVT=0, the default, means it only ever pays out.

Sizing: make the floor cover more than one day of an agent's worst case, which is every outgoing path firing while every incoming one fails: outgoing paths × 86400 / interval × amount. Put the ceiling well above floor + top-up (a ceiling at or below it is refused, because it would sweep an agent to where the next refill pays it again). At 0.1 a fire every 1800 s, 25 / 25 / 150 is comfortable; at 1.0 a fire, 200 / 300 / 2000.

The first sweep usually returns the grant the canary was born with, which is the point: that money belongs in the treasury, not parked in a monitor.

Traps worth knowing

• The pause flag is per registry. POST /admin/canary/fleet/pause sets a key in that registry's own Redis and it outlives whatever set it. A stale pause on an operator makes its refill answer paused while dispatched fires still execute, which looks like a rebalancer that does not work. Ask fleet/status on every node, not just the frame.
• An empty maintenance scope means everything. POST /admin/canary/maintenance with {"scope": [], "enable": false} switches off every path on that node.
• /admin/canary/fire talks to a scheduler. On a follower it does nothing; dispatch that path from its frame.
• Canaries are dark by design. If one shows up in discovery, its owner lost agents_unlisted_by_default; fix the developer rather than the agent.

See also: 06_MONITORING.md for health and metrics, 04_TOKEN_ECONOMY.md for what the transfers cost in fees.