DRQ measurements and delivery roadmap
Issue #184. Consumers take verified stock; measurement, qualification and replenishment belong to DRQ. The approved roadmap is a delivery plan, not a claim that its pending capabilities are live.
Replay an existing measurement
From the platform repository root:
pnpm exec tsx infra/r2-replay.ts --input /absolute/path/to/saved-export.json --format text
The input is { "pages": [{ "body": { "events": [...] } }] }, containing unmodified saved ledger events. Omit --format text or select --format json for the machine-readable report with raw trials and reference identities. The command does not contact the provider, rerun a measurement or change admission.
The text report separates the recorded outcome from the replayed native qualification. It shows the six current protocol origins, measurement windows, sample counts, separate candidate/control PUT/HEAD/GET distributions, failed instrument bounds, missing evidence and rejected reference observations. Distribution percentiles use the existing lower order statistic at floor((n - 1) * fraction); JSON's historical median field is this lower P50, not interpolation between the middle pair. Instrument health uses the third-smallest duration and the qualifier's unchanged limits. A complete window or healthy instrument alone does not qualify its target.
Native bindings and external endpoints are different paths. DO reference latency is not a native R2 lower bound. Latency indicates operation behaviour; it does not locate every stored copy or establish physical network adjacency.
Ordered delivery
| Stage | Scope and acceptance | State |
|---|---|---|
| 1 | Preserve clean consumption, record PR #315's actual outcomes, inspect stock, demand and stopped jobs without mutation. | Baseline recorded in readiness. |
| 2 | Extend existing replay with explicit origin coverage, operation distributions and actionable instrument failures. | Offline report and wider capture implemented. |
| 3 | Attempt all 17 agreed Cloudflare origins, verify actual execution and report unavailable/wrong-city origins explicitly. Retain partial observations. | Deployed. New captures retain measurements from misplaced instruments under their actual cities; they do not claim 17 distinct execution cities. |
| 4 | Price and deploy useful AWS regional and Local Zone origins. Independent scheduling and retained results must survive loss of Cloudflare measurement Workers. | Eleven origins deployed under the wider coverage and cost assessment, replacing the unapproved two-city pilot proposal. All eleven supplied repeated measurements for the five buckets and retained canary on 23 September. |
| 5 | Validate a versioned rule using independent controls, withheld measurements, geographic/provider coverage, contradictions, reference changes and provider/origin outages. Sufficient validated surviving evidence may qualify; insufficient evidence stays inconclusive. | ADR-0045 merged in #371 and deployed on 23 September at 08:01 AEST. All five previously inconclusive buckets passed fresh combined checks; both ledger copies and deterministic replay agree. |
| 6 | Compare new and old evaluation before promotion. Restore replenishment and periodic checks using preserved identities/cursors after recovery acceptance. | Geographic v3 is deployed. Consumer allocation/restock replay passed, the replenisher resumed its original cursor, and the updated periodic checker completed all 683 members: 681 stock passes and two healthy controls. A real first-draw stock addition also passed under v3. See the activation evidence. Both Mac jobs later stopped when their worktree was deleted and were retired (#466): restock happens only in take --restock, and the pool check runs in the ledger for R2 buckets and both controls. |
| 7 | Extend operation-specific Worker, DO, D1 and other service qualification without claiming untested configurations. | Pending; additional providers and old-resource cleanup deferred by the owner to #329. |
Cloudflare origin target: Sydney, Melbourne, Brisbane, Perth, Adelaide, Canberra, Hobart; Auckland, Singapore, Jakarta, Hyderabad, Nouméa, Suva, Hong Kong, Tokyo, Los Angeles, Honolulu. Every full round attempts the Australian set; missing origins are evidence of incomplete coverage, never fabricated locations. A requested placement is not an observed execution city. A difficult origin must not prevent collecting the rest.
External measurements distinguish AWS-to-target, Cloudflare-to-AWS references and selected AWS-to-AWS reference paths. External request time and operations inside a Worker remain separate. Multiple providers sharing a Worker or route do not become fully independent votes. DRQ retrieves authenticated evidence from approved sources; consumers cannot upload qualification evidence.
Start with repeated bounded trials, measure reference changes over time and increase sampling only when useful. Price compute, requests, scheduling, logs, retained evidence and traffic without assuming free-tier availability. Record the approved monthly budget and incident limits before provisioning. No always-on global fleet or provider-wide abstraction is required for the first useful delivery.
Wider observational capture
Run from the platform repository root after deploying the measurement stack:
pnpm exec tsx infra/r2-capture.ts --bucket canary-r2-o74bamza --output /absolute/new-round.jsonl
pnpm exec tsx infra/r2-capture.ts --input /absolute/new-round.jsonl
The first command reads the selected references from the ledger, then attempts all 17 origins sequentially. It uses the existing machine-local ledger and measurement credentials, with RESIDENCY_LEDGER_TOKEN and R2POOL_TOKEN overrides. It creates a private file without overwriting an earlier round and syncs each origin's result before starting the next; a retention failure stops collection. Exit zero means the attempts were saved, not that the target qualifies. The second command is offline and reports absent attempts explicitly. An unreadable final fragment without a newline is reported as incomplete or invalid while earlier synced rows remain readable; invalid interior or schema-invalid records refuse. Preserve interrupted rounds; another invocation is another observation, never a replacement for an inconvenient result.
The r2-shadow-v1 reply retains requested/observed origins, start/end execution checks, selected reference identities, each trial's actual reference observation, native PUT/HEAD/full-GET durations, operation failures and cleanup failures. A failed reference does not suppress native measurements; a failed PUT suppresses only that target's HEAD/GET for that trial. Candidate/control order alternates. The exact synthetic keys are cleaned from both buckets, with bounded cleanup and failures retained; timeouts cannot prove a native operation was cancelled by Cloudflare. Historical qualification is unchanged. Geographic v3 validates and compacts these observations through its own bounded evidence schema. Raw capture files remain operational evidence outside Git, never consumer-supplied admission evidence.
| Source/mode | Capture | Qualification |
|---|---|---|
| Cloudflare native binding, historical six-origin protocol | Existing ledger checks and offline replay | Existing evaluator only |
| Cloudflare native binding, 17-origin capture | Operator capture and fixed authenticated collector; actual cities and partial results retained | Geographic v3 validates source identity, reference health and time before assessment; deployed |
| AWS external S3 requests | Eleven independent scheduled collectors, private raw records and fixed-slot authenticated reader | Separate geographic assessment under v3; deployed |
Cloudflare placement hints select a nearby Cloudflare site; they do not create resources in the named provider or guarantee that site's city. Added regional hints use Azure's Canberra region, AWS Hyderabad/Hong Kong/Tokyo regions, and GCP Los Angeles. Brisbane keeps its existing selected-reference relay. Perth and Nouméa use public RIPE Atlas measurement targets as TCP placement hints, but the first deployed capture observed Singapore and Sydney respectively: neither added usable city coverage. Adelaide, Hobart, Suva and Honolulu have no verified placement anchor; their Workers use incoming execution placement and report a wrong-city result when appropriate. All six remain missing origins. Establishing actual execution there remains follow-up work; a deployed hint alone never counts as coverage.
Decisions and retained work
Preserve exact-ID admission, mirrored history, original canaries, candidate budgets and request identities under verified admission. A changed qualification basis requires an explicit successor decision and a new evaluator version; historical rules and events retain their original interpretation. No repeated sampling or control replacement merely to obtain a pass.
Source composition follows ADR-0021; executable evidence rules follow ADR-0025; infrastructure ownership remains with ADR-0029. Effect remains internal under ADR-0006/0022; new constructors and layers follow the owner's role-based naming convention without a repository-wide rename.
Additional provider expansion and legacy-resource work remain tracked in #329. The current wider-measurement goal includes dependency-checked cleanup of its obsolete resources while preserving useful controls and evidence. The restricted consumer credential is merged and deployed in #328; local default CLI acceptance passed. Other agents own their application changes and deployments. Independent ledger failover is outside this probe-resilience scope: independent probes cannot allocate stock without the authoritative ledger or make an unavailable target operate.