cmux IROH interactions
Verified main behavior and a separate /v2 proposal. Select a flow, then click a row to expand its payload and operation details. Click the row again to collapse it. This document changes the design only. Accepted decisions and registration occasions.
Observability for all backend activity
Accepted: record every logical backend operation and major event, including successes, response codes and rate-limit decisions. Use compact records; exclude credentials, raw request/response bodies and terminal content. Aggregate metrics from all outcomes; sampling is reserved for optional debug detail.
| Activity | Coverage |
|---|---|
| Every HTTP / socket operation | Success or failure, actual HTTP status or socket error, schema/method, user/team, request ID and duration. Includes auth failures before DO routing. |
| Every rate-limit decision | Allowed/limited, user policy, operation and retry delay. Counters cover all checks; rejected operations are searchable events. |
| Major changes | Enrollment/recovery, credentials, permission removals, directory updates, socket lifecycle, slow-client buffer hits, jobs and deployments. |
| Database and relay work | Operation timings, reads/writes and storage errors; relay auth/limits/lifecycle and aggregate bytes. No payload or encrypted-packet logging. |
| Collection health | Export failures/timeouts, queue pressure and dropped-record counts. Durable security history accompanies state changes. |
Collection plan: Cloudflare platform logs/metrics, structured searchable events through Axiom, and Sentry for unexpected exceptions. This collection stack and the completed operation coverage are accepted. Exact deployment, dashboards, retention and alert thresholds remain work to implement.
Use one shared operation wrapper for HTTP and application socket messages. Correlate Worker, DO and database phases without double-counting a routed request. Batch background exports with bounded memory, concurrency and time. Do not add a DO wake timer just for logs.
Ordinary logging is best effort, with measurable export losses; authority changes have durable history. Runtime ping/pong bypasses our handler. Direct IROH success and terminal responsiveness require app-side telemetry; the backend cannot infer them from its own socket. Full coverage and cost rules.
Accepted client compatibility
New Mac supports old and new iOS. New iOS only needs new Mac. The new Mac publishes the same IROH endpoint to both the older account directory and the v2 team directory. It accepts the older iOS connection messages alongside the new protocol.
Older iOS keeps its existing authorization rules. Modern peers require v2 permission and cannot fall back to older authorization after a denial. Pairing disable stops both services. Existing production storage and relay compatibility remain available.
New iOS uses v2 exclusively. Saved computer names and preferences survive the local storage move, but old records never grant v2 access. Verify new iOS with new Mac and old iOS with new Mac separately. Complete accepted scope.
Accepted decisions from Lawrence’s PR
Adopted into v2. These are implementation requirements, adapted to team-owned state and user allowances. PR by Lawrence Chen, inspected at a96093de.
| Accepted decision | v2 requirement |
|---|---|
| Shared local broker | HTTP and socket adapters call the same team-local broker. Keep transport/session handling separate from operation and storage rules. Ordinary IROH operations do not proxy to Vercel. |
| Zod at both boundaries | Validate every server request and response using the versioned Zod contract. Keep JSON Schema export, quicktype Swift/TS generation, and compatibility checks. |
| Safe object activation | Check and apply Drizzle durable-sqlite migrations inside blockConcurrencyWhile before serving requests. Use supported synchronous storage transactions; no partial schema is visible. |
| Immutable migration history | Never edit a shipped migration. Verify migration history and refuse a schema the worker cannot safely serve. Roll back code only when it supports the already-applied schema. |
| Database-enforced quotas | Use constraints/triggers to update usage and reject writes in the same transaction. Attribute product usage to users; size the shared physical database guard separately. |
| Bounded temporary state | Keep one pending challenge slot per device/build identity. Reject expired challenges when used, replace the slot on a new challenge request, and clear it on successful consumption. No challenge sweep or scheduled deletion. Any future retained-history cleanup needs an explicit retention policy and bounded work. |
| Real runtime verification | Require Miniflare/workerd tests for persistence, reactivation, isolation, migrations and rollback/failure. Test real alarms only for features that require them. Add team/user separation, supported v2 client compatibility and interrupted-renewal coverage. |
| Enforced architectural boundary | Required build checks must invoke the no-Vercel/no-Hyperdrive dependency check. Permit the deliberate PlanetScale global ownership service; keep ordinary team-local reads and renewal local. |
| Staged deployment | Run generation, schema checks, runtime tests and a Worker dry run in development. Canary one team, then promote the same immutable worker build. Separate large backfills from traffic cutover. |
| Complete, bounded observability | Use Cloudflare logs/metrics, Axiom structured operation events and Sentry exceptions. Cover all HTTP/socket outcomes and major events. Bound exports and keep telemetry delivery off the response path. |
Flow 6 shows schema readiness, shared request handling, transactional guards and deadline cleanup with expandable payloads. Required verification includes the architectural boundary check, closing the wiring gap observed in the inspected PR.
Our choices remain explicit: team DOs, user allowances, 30-minute challenge and relay credentials, one pending challenge per identity, no scheduled presence heartbeat, and durable device identity across offline periods. Numeric storage guards are sized for our model. Accepted implementation rules · Source notes and adaptations.
Team-owned state, user-owned limits
Decided: use Stack Auth teams for shared IROH state. The Worker verifies the user and selected team before routing to that team’s DO. A valid client-supplied team ID alone grants nothing. Known membership or permission removal invalidates affected sessions and future minting.
| Area | Scope |
|---|---|
| Durable Object and SQLite | One per Stack Auth team in each environment; include Stack project in routing identity. |
| Directory and device records | Team-owned directory, filtered by the caller’s permissions; each device keeps its owner user and unique EndpointID. |
| Permissions and revocations | Team membership plus explicit device/connect/manage permission. Membership alone never grants terminal access. |
| Control sockets and live updates | Join the verified team object; team revisions and permission-filtered broadcasts. |
| Tickets, challenges and signed peer permissions | Bind environment, Stack project, team, user, device/build, generation and allowed actions. |
| Relay settings, caches and Dashboard | Team-scoped shared settings, cached directory and Dashboard selection. User UI preferences and local pairing opt-in stay individual. |
| Global EndpointID map and reports | PlanetScale stores team ownership plus owner user. Global uniqueness remains global. |
| Request and product usage limits | Per verified user and operation, shared across their devices and teams within an environment; no IP quota. |
| Physical safety bounds | Payload/socket limits and finite DO memory still apply. These protect resources; they are not a shared team request allowance. |
Team changes must isolate cached credentials, directory revisions, pending work, and local keys. An ordinary team selection must not silently publish a Mac to a new team or broaden terminal access. Exact sharing/transfer UI and membership-change delivery remain open.
Rate key: (environment, Stack project, verified userId, operation). Separate team DOs must coordinate this user budget. The counter owner/batching design remains open and is excluded from current request-cost estimates; separate counters per team would allow a team switch to reset the allowance.
Accepted generous starting output buffers: 2 MiB / 1,024 messages per connection and 8 MiB / 4,096 messages total per user. Bound only unsent data, preserve ordered permission changes, and disconnect/resume a slow control connection when necessary. Healthy peer traffic continues.
Pairing implementation is running independently in iOS Pairing Opt-in (workspace:51). Agent task · Lawrence PR lessons
Pairing is off until the user enables it
Decided: each Mac must have Enable iOS pairing turned on in cmux Settings before its IROH features run. Check that local setting before initializing the endpoint or starting any IROH backend task. While off: no IROH listener, relay connection, control socket, enrollment, directory fetch, or credential-refresh loop. Other cmux features keep their own independent network behavior.
Once enabled, a returning Mac immediately starts IROH from usable cached data while restoring backend control and refreshing credentials in parallel. First enrollment still obtains the missing prerequisites. Re-enabling an intact trusted identity resumes it. Disabling stops IROH listeners, sessions, and renewal/reconnect tasks; delayed callbacks must not start them again. Detailed migration and cleanup behavior remains an implementation decision.
Proposed empty computer list: “No Macs found. Open cmux Settings on your Mac, enable iOS pairing, and select the same team on both devices.”
iOS shows setup instructions before starting pairing. The user’s Continue action begins phone setup and discovery. A phone cannot verify a remote Mac’s local setting without communicating; the Mac’s own gate provides the enforcement. An empty list does not prove the setting is off. Exact settings location, buttons, and diagnostic copy remain design details.
Seven devices fit the registration allowance
Accepted: 300 registrations/hour, burst 50 per user. Two iPhones plus five Macs need seven initial registrations, using 7 of 50 immediately available requests. Normal launches and renewals need none. Distinct app channels or dev tags add identities; the shared allowance refills at one request every 12 seconds.
| Condition | Call device.register? |
|---|---|
| First setup of an enabled team/user/device/build/environment identity | Yes, once per identity. |
| Approved key replacement or recovery | Yes, after new-key proof. |
| Reinstall, restore, sign-in, team change, or pairing re-enabled | Only if the matching identity/trust is missing or new. Resume an intact identity. |
| Interrupted enrollment or lost response | Retry safely using the same retry ID; refresh a replaced/expired challenge as needed. |
| Restart, wake, foreground, reconnect, renewal, relay change, new terminal, ordinary app update | No. These reuse the identity or use separate update operations. |
No scheduled DO presence heartbeat
Accepted removal. A usable authorized IROH session determines whether this client is connected. A saved Mac without an active session has unknown reachability. Never hide it or block cached dialing based on a backend online flag. Keep control sockets for credentials, directory updates, and revocation; their status describes backend connectivity only.
Keepalives exchange traffic in both directions. In IROH’s QUIC transport, either endpoint can send a probe and the peer acknowledges it. This does not require duplicate application timers. Keep the library’s transport liveness and application request deadlines. On the backend WebSocket, a native client can ping and Cloudflare automatically returns pong without waking the DO. Exact intervals, sender configuration, and failure policy remain open. QUIC PING/ACK · WebSocket ping/pong.
Keep a bound on unsent outgoing data
Accepted generous starting bounds: A slow phone must not make the server accumulate unlimited unsent messages. Bound queued and transport-buffered bytes, pause production where possible, and resync the control socket when required. This does not limit total messages delivered or terminal output. Allow up to 1,024 unsent messages or 2 MiB per connection, with a total of 4,096 messages or 8 MiB per user, whichever is reached first. Allocate memory only as needed. These starting sizes are increased eightfold; tune them after measuring real payloads. A shared DO also needs a physical memory safeguard that handles slow consumers fairly.
IROH has one connection coordinator per peer to avoid duplicate callers, while permitting parallel direct/relay paths, independent peers, and a planned replacement beside a healthy session. No local attempts-per-minute quota is added. Local cache reads, credential installation, server replies, and database calls behind an accepted request have no additional request counter.
User limits by endpoint and message
Decided: each client endpoint or message type gets its own generous per-user allowance, with no IP quotas. Registration is accepted at 300/hour, burst 50. Other numerical allowances remain recommendations. All devices, builds, and Dashboard sessions belonging to the same user share that user’s operation allowance across teams within an environment. A socket message and its equivalent HTTP fallback share that same allowance. Development and production have separate budgets.
Each allowance refills at the stated rate. “Burst” is how many requests can arrive together after idle time; unused allowance cannot accumulate beyond that burst. There are no additional per-device request quotas or daily lockouts. One connection owner per peer prevents duplicate callers while allowing parallel paths and make-before-break replacement. Message sizes remain bounded.
Room for many builds: Seven continuously active app instances need about 7.6 API-ticket renewals/hour and 16.8 relay renewals/hour, with no scheduled DO presence messages. Proposed user allowances are 1,200/hour for each credential type. Generous limits do not increase the normal renewal frequency.
Separate allowance for each client operation. Directory reads, metadata changes, preferences, and revocation each have their own user counter. The key is (environment, Stack project, verified user, operation). Switching teams does not reset it. Versions of the same operation and its HTTP fallback share one policy. Route parameters such as a device ID never create fresh allowances.
Local work, server responses, and database calls behind an accepted request have no separate request quota. Message sizes, queued bytes, and open-connection counts remain resource bounds. Message names and unmapped fallback routes are still working design details.
Scheduled application presence is removed. Protocol ping/pong needs no application counter: Cloudflare handles it without waking the DO. Actual socket-close cleanup and enforcement of known revocations must always run. Cloudflare ping/pong behavior.
| Operation | Who shares the limit | Allowance / rule | Handling / purpose |
|---|
How limits preserve connections and control costs
- Return HTTP
429withRetry-After, or a socket error withrequestId, code: rate_limited, retryAfterMs, scope. Report the affected team operation and retry time. Clients share that cooldown for that operation; unrelated operations and healthy IROH sessions continue. - Enforce each operation’s allowance in the team DO with coordinated user rate state. The server maps recognized endpoints and message types to a fixed policy ID; it never uses arbitrary client strings as new quota keys. Count each operation inside a batch. Setup that performs ticket issuance or challenge creation consumes those operations once, without charging their responses or internal database calls.
- Before authentication, reject malformed credentials cheaply and bound shared authentication work. Never charge a user named in an unverified request, since that would let someone exhaust another user’s allowance. No application IP quota is introduced.
- Rate state must survive reconnects and hibernation. Keep bounded counters and measure persistence work. Do not reset a budget when a device, build, socket, or key changes. Exact storage remains an implementation decision.
- Retry a mutation with its original request ID. Return the prior result where safe, while still counting incoming traffic against abuse protection. Coalesce unchanged metadata before doing work.
- No scheduled presence messages, presence-only alarms, or presence checkpoints. Keep other required state and deadlines correct through hibernation.
- Preserve ordered updates and revocations. Resync or close a slow control socket when its output buffer fills; never silently discard access removals. Known revocations take priority over renewal and rate limits.
- Measure high user usage and throttled operations before adjusting these proposals. Cloudflare’s Worker limiter is local to a location and approximate, so it cannot alone represent one exact shared user allowance. Cloudflare limiter behavior.
Expected /v2 usage
Updated for accepted heartbeat removal. Scheduled DO presence traffic is zero. This model includes only enabled app instances; Macs with pairing off create no IROH traffic.
A planning model, not production measurements. Defaults represent 100,000 app instances active for 24 hours a day, one control-socket opening and two metadata changes per day, over 30 days. A user with a Mac and an iPhone has two instances; each simultaneous build adds another. Suspended iOS apps do not run continuous renewal loops.
Scheduled assumptions: ticket renewal around minute 55 of its one-hour lifetime, relay renewal around minute 25 of its 30-minute lifetime; no application presence timer. These timings are proposed. Counts are long-run averages; renewal requests replace due startup issuance rather than adding duplicate mints. Extra expiry checks after suspension, retries, enrollments, and user actions add to this model.
| Operation / unit counted | Per app / day | Selected fleet / 30 days | When it happens / backend work |
|---|
Request counts do not give the complete bill. Measure DO active duration, alarms, rate-counter checkpoints, SQLite rows and indexes touched, stored data, Stack usage, PlanetScale usage, relay bandwidth, and logs, team fanout, and coordination of user quotas across team DOs. Billing allowances are shared across the Cloudflare billing account. Cloudflare DO pricing, checked 10 September 2026.
For 100,000 users with two iPhones and five Macs all enabled and always active, set app instances to 700,000. No scheduled DO presence messages are generated. Real active hours, especially on phones, need measurement.
The client owns credential renewal
One current cached value: each device/build keeps an API ticket per team authorization scope and a relay credential per relay audience (the relay or fleet that accepts it). Keep the previous valid credential while installing its replacement. Replacing the cache does not invalidate previously issued signed tokens; they can remain valid until expiry, subject to applicable revocation checks. No database row per issued credential and no credential-deletion job.
One client coordinator schedules API-ticket and relay-credential renewal. It requests a replacement over the control socket before expiry. The DO checks current permission and issues the replacement; it does not run a second renewal timer for each client.
The client installs the new credential, switches when it is usable, and briefly retains the still-valid old one for ongoing work. The server permits that bounded overlap. Failed renewal backs off and retries without restarting a healthy IROH session. Neither side extends expired authority; explicit revocation takes effect independently.
IROH starts immediately with cached data
Returning Mac and iOS clients start IROH and backend refresh in parallel. Use the saved identity, usable cached relay credentials, relay URLs, and still-valid peer permissions without waiting for the control socket, ticket renewal, or directory refresh. A healthy peer connection continues while refresh runs.
Fresh credentials update the shared credential state and promptly unblock a retry that needed them. One coordinator owns retries. First launch, or loss of required saved identity data, must complete the missing enrollment and credential steps first. Network and peer availability can still affect success.
What are “hints”?
For v2, a hint means the reachable relay server URL. We may keep more than one relay URL for fallback. A relay hint helps find the app; it does not grant terminal access or prove EndpointID ownership.
Direct mode addresses, direct ports, and optional direct IROH paths stay local or move directly between peers. They are not stored in the server registration.
A relay credential is permission to use the relay service. It is separate from the hint and from terminal permission.
When the Mac registers today
The code also calls updates to an existing device record “registration.” Those updates normally keep the same registered identity.
- When the connection system starts after launch/sign-in, register before getting relay credentials.
- After IROH starts, update the record with the actual relay and connection ports.
- After a credential renewal, check the connection details. Publish again if they changed, or the previous publication is at least 15 minutes old.
- Switching teams, restarting the connection system, or re-enabling networking repeats setup. An incomplete setup can retry.
Separate “still online” messages run every 15 seconds. Silence for 45 seconds marks the Mac offline. A clean shutdown says “going offline” immediately. These messages do not create a new device identity.
Yes, iOS registers too
The Mac needs to know that the connecting phone belongs to the team, even though the phone does not advertise itself as a Mac to connect to.
- First setup after sign-in: register before obtaining relay credentials and the device list.
- On a later app launch: refresh registration during setup. With usable saved data, that refresh happens in the background.
- Missing saved identity or device-list information, or unusable saved relay credentials, makes the current code register before finishing setup.
- Account changes or rebuilding the connection system repeat setup. Incomplete setup is retried.
Returning to the foreground with the connection system still alive checks credentials and reconnects. It does not by itself call registration. Connecting to another Mac or opening another terminal reuses the phone identity.
iOS setup code · Foreground behavior
Accepted v2 registration behavior
Both apps register once per team/user/device/build, then resume on restart. Changes to a name or published relay update the same record. Direct ports stay off the server. A lost key or approved key replacement requires registration again. A revoked device must not silently register again to bypass its removal.
Chosen: Durable Object SQLite
Use Durable Object SQLite for IROH persistent storage. One object per Stack Auth team and environment is decided: it stores that team’s devices, access removals, pending challenges, and connection preferences alongside its live connections.
PlanetScale is the primary general database choice for data better suited to a shared database. Specific table assignments are still recommendations.
| Durable Object SQLite | PlanetScale |
|---|---|
| One team’s devices, pending challenges, connection settings, and team device metadata. | Records shared with other product services or involving multiple teams. |
| Fast changes that should be pushed to that team’s connected apps. | Admin searches and reports across teams. |
| Local permission decisions for that team. | The global EndpointID ownership map. |
Give each record one authoritative home. A later reporting copy should not independently decide access.
A Durable Object’s SQLite database is private to it. PlanetScale supplies a shared relational database; we have not selected its Postgres/MySQL option here. Cloudflare storage · PlanetScale
Shared types and server validation
Updated decision: use Zod instead of Ajv. Author versioned request and response schemas in Zod. Zod validates server inputs and outputs; export JSON Schema at build time and use quicktype to generate Swift and TypeScript models. Clients decode generated types and handle server errors. They do not run schema validators.
Keep a portable JSON shape for the shared contract. Request and response schemas are separate; avoid coercions and transforms that change the wire type. Keep custom server checks, such as team permission, signature, or cross-field rules, explicitly documented. Fail generation for an unrepresentable shape instead of silently exporting unrestricted data.
Each method/frame carries its schema ID. Compatible additions preserve old behavior; breaking request/response changes or intentional behavior differences for newer clients get a new handler version. Keep supported App Store versions working. No mandatory startup version exchange; app version can accompany an ordinary request. Retired versions receive client_upgrade_required with the agreed long backoff.
Build checks regenerate models, compare the generated TypeScript shape with Zod’s inferred types, compile Swift, and exercise shared request/response examples. Quicktype code generation and compatibility tests are additions we still need beyond Lawrence’s PR.
Version and team rules · Zod JSON Schema export · quicktype · Lessons from Lawrence’s PR
Heartbeat removal and hibernation
v2 sends no scheduled application presence heartbeat to the DO. This removes the previously modeled 4.32 billion monthly presence messages at 100,000 continuously active app instances, plus their handler work and presence-only deadlines. Protocol ping/pong and IROH transport keepalives remain. Credential, directory, security, and storage work still have costs.
Updated usage model and remaining costs · Hibernation behavior
EndpointID ownership
Every unique identity tuple owns one EndpointID. The active map is keyed by EndpointID and stores its environment, Stack project, team, owner user, device, app namespace, build tag, platform, and generation.
Registration and key replacement reserve this key atomically. No other device, build, or environment may reuse it. Team-local Durable Object SQLite stores the full device record; PlanetScale stores the strongly consistent global map.
Development without another DB server
| Environment | Resources | Source |
|---|---|---|
| Production | Own Worker, DO namespace and SQLite data, signing keys, auth configuration | Explicit release deployment |
| Shared development | Own Worker, DO namespace and SQLite data, development secrets | Latest main |
| Branch development | Own suffixed Worker + DO namespace; clean synthetic seed | Requested branch, on demand |
Both apps in a test pair resolve the same environment. Bind API tickets and relay credentials to that environment; separate trust keys/relay audiences prevent production acceptance. Branch resources expire after a configurable inactivity window and are explicitly cleaned up.
Use hibernating WebSockets and push only changed facts. Avoid copying production data and avoid a permanent database server per branch. Hibernating objects retain sockets without duration charges while asleep; stored data still costs money. Hibernation behavior
Security and cost tradeoffs
- One-hour API tickets remove Stack calls from ordinary backend operations. The team DO still checks the current device generation and revocation state before allowing protected actions.
- 30-minute relay credentials reduce routine mint traffic. A disconnected relay verifier can accept an already issued token until its expiry. Stopping future minting does not erase that token.
- Relay access grants transport service, not terminal authority. The Mac must independently authorize the peer against its trusted device list.
- Online revocation pushes can close sessions immediately after receipt. A disconnected Mac has only its cached list, so the offline authorization lease determines the stale-access window. The main value of 24 hours is a separate v2 decision.
- Use one pending challenge field set per identity, bounded user rate counters and audit retention. An enrollment challenge request is rate limited before allocating a pending-only device row.
- Preserve cached state through recoverable outages; never reconnect all IROH sessions just because a backend credential refreshed.
Proposed client response contract
Codes below are proposed for v2. Clients branch on stable error codes as well as HTTP status. A server failure or corrupt response must not become “wrong password.”
| Response | Client action |
|---|---|
| 200 / 201 | Decode generated types; check user, team, environment, endpoint and revision. Apply only if still relevant to the current operation. |
| 204 | Complete the mutation without parsing a JSON body. |
| 101 / WebSocket frames | Decode the versioned frame and check its team and revision. Resume from a known revision, otherwise request a full snapshot. |
| 400 / 405 / 413 / 415 / 422 | Report invalid request or unsupported format. No automatic unchanged retry. |
| 401 ticket_expired | Single shared ticket refresh, then retry once if safe. If Stack refresh fails transiently, preserve local state and back off; only a definitive authentication rejection requires sign-in. |
| 403 device_revoked / team_access_revoked / permission_denied / team_mismatch / user_mismatch / environment_mismatch | Stop affected protected operations and close sessions when access is revoked. Clear only that team’s permissions when team access changes; preserve unrelated teams. Explain the denial and do not repeatedly refresh Stack. |
| 404 challenge_missing / 409 challenge_replaced / 410 challenge_expired | For enrollment, request one new challenge and retry the transaction within a bounded attempt count. Other 404s are handled according to the missing resource. |
| 409 revision_conflict | Fetch the latest revision and reconcile. Retry only a mutation whose intent still applies. |
| 426 client_upgrade_required | Present the update action, stop automatic retries of that retired version, and use a 1h, 6h, then 24h jittered backoff. Reset after success or app update; allow a user-triggered retry. |
| 429 | Honor Retry-After, share cooldown across callers, and add bounded jitter. Do not turn a mint limit into a re-enrollment loop. |
| 5xx / timeout / connection loss | Back off with jitter. Preserve a healthy peer session while its independent authorization remains valid. Reconcile uncertain mutations before replay. |
| Redirect / unexpected status / malformed body | Never forward credentials to another origin automatically. Record a protocol/configuration failure; no success assumption or unbounded retry. |
| IROH or WebSocket close | Map the application close code to revoked, superseded, unavailable, or retryable transport loss. Keep lifecycle ownership in one reconnect coordinator. |