Migrating a Tenant to Synced Config

This is the procedure for taking one tenant from inherit-on-read (config merged in from the control plane on every request) to synced (the tenant reads only its own database). Do it for a pilot tenant first, then use fleet rollouts for the rest.

Background: per-tenant sync mode, syncing a resolved parent chain and shadow mode.

Before you start

Pick the pilot. A tenant you can afford to look at closely: real traffic, but a tenant whose owner you can reach. Prefer one with a Workers for Platforms worker if that is where you are heading, since that exercises the dispatch transport.

Checklist. Every item must be true before step 1.

• The control plane has the tenant operations adapters (GET /api/v2/tenants/{id}/operations answers) and a sync-defaults executor. For a tenant with its own worker and database, syncDefaults: createDispatchSyncConfig({...}) is passed to initMultiTenant. If the control plane encrypts at rest, it also passes the layer that returns secrets as stored as storedDataAdapter; without it step 3's status shows drifted: true for a tenant that is in sync. The Workers for Platforms guide sets all of this up.
• WFP_INTERNAL_SYNC_SECRET is set to the same value on the control plane and the tenant worker. A mismatch answers 401.
• The tenant worker has TENANT_ID set (a push for another tenant is refused with 403 sync_config_tenant_mismatch).
• The tenant worker has CONTROL_PLANE_ENCRYPTION_KEY, so it can decrypt secrets the control plane encrypted. Without it the push stops with 422 sync_config_key_ring and nothing is written.
• Shadow mode (inheritanceShadow) is on for the host that serves the pilot's traffic. It runs where the fallback adapter runs. A pilot served only by its own worker needs it in the worker: createWfpTenantApp({ inheritanceShadow: true, ... }), whose cfgdiff logs land in the tenant's own logs and are read through the same API. Either way, Verify in step 3 is the check that does not depend on traffic.
• The tenant is on the parent you intend: check parent_tenant_id (empty means the control plane).
• You hold a control-plane token. Only the control plane can change config_sync.

Know what is not synced. Look at these before the dry run, because the sync will not change them:

• Email provider and branding have no id to mark a synced copy by, so a row the tenant already has is only replaced when the parent locks it (override: none or enforce). Otherwise it stays, and a later edit on the parent does not reach the tenant. Prompt settings follow the same rule, and are applied only when locked.
• A connection's enabled_clients is not copied (it holds the parent's client ids). When a synced connection is first added, it is enabled on the tenant's clients that list their connections explicitly; clients without a list already use every connection. Later syncs leave the tenant's choice alone, so enable or disable it per client afterwards if you want something else.
• A row the tenant already has with the same strategy and name (connections), identifier (resource servers) or name (actions) wins: no synced copy is added next to it, and the dry run lists skip entity:key (the tenant has its own). Rename or remove one side if you want the parent's row instead.
• inherit: true action stubs only resolve against the control plane while the tenant is inherit-on-read, and client URLs from the control plane's client are no longer merged once it is synced. Copy what the tenant needs into the tenant itself.
• Platform-scoped rows are copied so they can run, but are hidden from the tenant's management API.

1. Dry run

POST /api/v2/tenants/{id}/operations
{ "kind": "sync-defaults", "input": { "dry_run": true } }

Read result: status (dry-run), per-entity counts (added, changed, unchanged, deleted, errors) and changes, the list of entity:key that would change. It shows which rows change, never their content or secrets, so there is no field-level diff; compare content on the control plane if a row looks surprising.

Expect added for everything the tenant inherited. Investigate:

• any errors;
• skip ... (the tenant has its own) lines (a name collision from the list above);
• anything you did not expect to be published (check its scope on the control plane).

The same preview is in the admin UI on the tenant's Operations page (Preview sync).

2. Sync with inheritance still on

Run the operation without dry_run. The tenant is still inherit-on-read, so nothing changes for its users yet: the synced rows exist, and read-time inheritance still serves requests.

The operation must end succeeded. If it ends failed, read error and the failed event in GET /operations/{id}; the tenant's stored hash is only updated by a fully successful run, and retrying is safe. A partial run leaves the rows it applied in place.

3. Verify

Do all of these before flipping.

1. Status. GET /api/v2/tenants/{id}/config-sync shows drifted: false and equal config_sync_hash and expected_hash (the admin UI shows this as In sync). The hash only says the parents have not moved since the last sync, not that the tenant's own rows still match.

2. Verify. The hard check, on demand and read-only:

   GET /api/v2/tenants/{id}/config-sync/verify

   ok: true (no differences and no errors) is the pass criterion (Verify on the tenant's sync status card in the admin UI). A tenant that inherits on read is compared the way shadow mode does, without the rows only the sync model copies. A synced tenant is checked with a sync dry run against its own rows, so a copy someone deleted (missing), edited (changed) or that the parents no longer publish (stale) shows up although the hash matches. For a tenant with its own database the dry run goes through the same transport as sync-defaults (syncDefaults), and the call answers 502 if the worker cannot be reached. A row the tenant owns that wins over the parent's is not a difference. Only ids and field names are returned, never content or secrets.

3. Shadow. Let the pilot take normal traffic, then look for differences between what read-time inheritance would give and what the synced config gives:

   GET /api/v2/logs?q=type:cfgdiff

   with the pilot's tenant-id. An empty result after a representative period is supporting evidence. Each entry names the kind (missing, extra, changed), entity and key, never content. Shadow is sampled and rate-limited, so absence is evidence over time rather than proof (use Verify above for that), and extra entries for hooks can be noise from the synced copies: confirm them against the hook list.

4. Smoke tests. Log in with each connection the tenant uses, request a token for each resource server, trigger each inherited hook, and confirm each hook runs once.

4. Flip

curl -X PATCH https://auth.example.com/api/v2/tenants/settings \
  -H "authorization: Bearer $CONTROL_PLANE_TOKEN" \
  -H "tenant-id: $TENANT" -H "content-type: application/json" \
  -d '{"config_sync": "synced"}'

Or Settings → Advanced in the admin UI. The instance that handles the request drops its remembered mode at once; every other instance notices within the mode cache time: 10 seconds by default, or the configSyncCacheTtlMs you set in initMultiTenant. Wait that time plus a margin before judging the result. If the auth server runs with a cache, logins can take longer: the client bundle (the tenant's client, connections and settings) is kept for 5 minutes and may be served up to 10 minutes more while it refreshes, and purging it on a tenant update is best effort. On such a deployment give the flip up to 15 minutes, or purge the cache, before the smoke tests.

Then repeat the smoke tests from step 3. The tenant now reads only its own database.

5. Watch, and keep the rollback window

For at least a few days (pick the length with the tenant's owner), watch:

• GET /logs?q=type:cfgdrift for the tenant: a drift entry means the parent changed and the tenant has not been brought up to date. With the scheduled handler wired, a rollout fixes it; otherwise run a sync-defaults operation.
• The tenant's operations history (GET /tenants/{id}/operations?kind=sync-defaults) for failed runs.
• Login and hook error rates for the tenant, as you would for any release.

Rollback. Set config_sync back to inherit-on-read (same call as the flip). Read-time inheritance resumes. Know what stays behind:

• The synced rows (sy ids) remain in the tenant. They are not removed.
• An inheritable hook is not run twice: the original is skipped when the tenant holds its synced copy.
• Changes made on the control plane while the tenant was synced are not applied to the copies. Run sync-defaults again (it still works in inherit-on-read) if you want them current.
• The copies stay read-only to the tenant's admin, according to their override and enforce.

Do not delete the synced rows by hand during the window; it makes a later re-sync add them back.

Success criteria

The pilot is migrated when: the sync ended succeeded, drifted is false, verify answers ok, the shadow comparison showed no unexplained difference, the smoke tests pass in synced mode, and the watch period ended with no regression. Record the date, the operation id and what you found, then move on to a rollout for the next tenants, starting with a canary.

Not supported yet

• Re-parenting. Changing parent_tenant_id of a synced tenant changes the hash, and the next sync adds the new parent's rows and updates shared ones. Rows only the old parent published stay behind until you run a sync with prune (preview it first). To leave a parent for good, detach the tenant instead.
• Automatic delete propagation. A row removed or unshared on the parent is removed from a synced tenant by a sync with prune (Remove unshared on the tenant's Operations page), never by a plain sync. A preview always lists what would go.

These are tracked as follow-up issues of the config inheritance epic.

Detaching a tenant

A synced tenant leaves its parent with a detach tenant operation (Detach on the tenant's Operations page). The tenant has to be config_sync: "synced": detaching keeps what the tenant holds, and a tenant that inherits on read holds none of it, so it would lose what it inherits. The operation refuses such a tenant, and the admin does not offer it. Run Preview detach first, with the same options you will use (Remove inherited for clean): it lists every row as convert, remove or skip and writes nothing.

• Keep (the default). The copies a sync delivered become the tenant's own rows: the content is kept, the ids are normal, and scope, override and enforce are dropped, so nothing locks them. Hooks run the converted code, and the tenant's clients and organizations that used a copied connection use the new row.
• Clean (clean: true). The copies are removed instead, platform rows included, and the clients' connection lists are cleaned up. Connections, resource servers, actions, hooks and hook code are covered; see the next point for what is not.
• Email provider, branding and prompt settings have no id to mark a synced copy by, so detach cannot tell a copy from the tenant's own row. Neither mode touches them: a copy a sync delivered stays as it is, with its scope, override and enforce, and clean does not remove it. A platform email provider stays hidden from the tenant's admins.
• Platform rows (scope: "platform") run for the tenant but are hidden from its admins. Converting them would expose their code and secrets, so a plain detach keeps them as copies and reports skip ... (platform: stays a copy). Only clean removes them.
• The tenant gets config_sync: "detached", no parent and no sync state. It no longer inherits, and sync-defaults, rollouts and the drift check leave it alone. To attach it again, set config_sync and parent_tenant_id back and run a sync.

Rows are created before the copies they replace are removed, but the operation is not atomic: if it fails part-way the tenant is not marked detached, and a retry can leave a duplicate of a row that was already converted, so review the list afterwards. SCIM configurations that point at a copied connection are not moved. Connection order on a client's login page is not kept for converted connections.

Tenants with their own database

For a tenant with its own worker and database (deployment_type: "wfp"), pass detachTenant: createDispatchDetach({ dispatcher, internalSecret, controlPlaneAdapters }) (from @authhero/cloudflare-adapter/wfp) to initMultiTenant, or use it in a Workflows detach operation (runDetachOperation). The control plane checks that the tenant can be detached from its own tenant row, pushes to the tenant worker's /internal/detach (the same WFP_INTERNAL_SYNC_SECRET and TENANT_ID checks as /internal/sync-config), and marks its own row detached only when the worker reports that every row went through. A preview or a partial result leaves the control-plane row as it was, so a retry can finish the job. The worker converts the rows in its own database and marks its own copy of the tenant row.

Route by deployment_type in the host, as for syncDefaults: a custom detachTenant replaces the in-process detach for every tenant, so shared-database tenants must still go through createInProcessDetach. The worker needs a detach route as well when it is not built with createWfpTenantApp.