Phase M2: Remote fetch (3-node cluster)

USE_REMOTE_FETCH (getMappedRange) against SCAN_AND_FETCH on the final tree (267a41434a81723b07e7150869d702636a1b91a9) across a 4-host VPC topology: one dedicated client VM and three dedicated FoundationDB 7.3.77 server VMs (double redundancy, ssd-2 storage engine, 9 storage servers). This is phase M2 (M2rl and M2skip) of hcloud-multinode-vpc-20261014.

Shard placement and qualification (Gate P0)

• Before any M2 timing, placementprobe_cgo (H5, reading \xff/dataDistributionMode, \xff/moveKeysLock/Owner, \xff/moveKeysLock/Write, \xff/keyServers/, and transactional storage addresses via libfdb_c at snapshot read version V1 and rechecking stability at V2) and the ballast tool (H6) were qualified on the live cluster:
  • Probing with data distribution on was refused.
  • The unmodified scratch control store passed integrity_ok = true and all 30 Python provenance/key mutation controls (ok = true).
  • All three physical on-cluster defects (delete-chunk, stray-suffix, version-key) failed integrity_ok with the expected error messages.
• Three placement attempts (256 MiB, 512 MiB, 1024 MiB ballast) were run following the preregistered rule (protocol.json placement.mechanism): settle for 120 s with moving_data in-flight and queued bytes at 0, freeze data distribution (datadistribution off), clear only the ballast range and verify it empty, and take the canonical CGo placement probe and receipt.
• Per protocol, attempt 3 was sealed before timing (plans/testingb/manifest.json SHA 4ada3920f92da4489fe4037ad5dd3440cec59957fc4ccdcba81b792b7a45f54a):
  • benchfetch/small: integrity_ok = true, passed = true, 100% cross-server (server-disjoint fraction 1.0000) on every cell.
  • benchfetch/planskip: integrity_ok = true, passed = true, 100% cross-server (server-disjoint fraction 1.0000) on every cell.
  • benchfetch/split: integrity_ok = true, 1 index shard and 7 record shards, shared_shards = 0: limit1-forward and limit1-reverse are cross-server (1/1), rows10 is partially colocated (f = 0.0000) (0/10), and rows1000 is partially colocated (f = 0.6960) (696/1000 rows on storage-server teams disjoint from the index shard’s team).
  • With double replication across three machines, any two 2-replica storage teams share at least one physical machine by pigeonhole; cross-server therefore means storage-process (address) disjoint, not machine disjoint.
• Before the first process and after the last process of every M2 case, R2 (bench_runner_mn.py) took a fresh CGo placement receipt and verified via placement_receipt.compare that shard boundaries, storage teams, and disjointness fractions remained identical (equal = true).
• R4 telemetry reconciliation (reconcile_telemetry.py) across node1..node3 and client reported 12/12 clean cases in M2rl (144 processes, 432 samples) and 4/4 clean cases in M2skip (48 processes, 144 samples), with 0 contaminated and 0 unverifiable cases.

Results

M2rl: BenchmarkIndexFetch (final-remote / final-ordinary)

Geometric mean of six paired block ratios (median of 3 repeats per process, 3s benchtime), with nominal 95% block-bootstrap intervals (20,000 resamples, preregistered seed 492817; initial exploratory analysis with seed 391706 is retained in the campaign archive). Below 1 means USE_REMOTE_FETCH is faster or allocates less.

M2skip: BenchmarkPlanSkip (final-remote / final-ordinary)

All rows/op and prefix-B ratios in both M2rl and M2skip were 1.000 [1.000, 1.000] (prefix-B = 19 in M2rl, 22 in M2skip).

Absolute values

Medians of the six block medians per arm.

Interpretation

• 1,000-row scans on the 3-node double ssd-2 cluster:
  • On this 3-node ssd-2 cluster (which differs from the single-node campaign simultaneously in host topology, network, ssd-2 vs memory storage engine, double vs single redundancy, and storage-server count, so differences between the two campaigns are descriptive and cannot be attributed to one factor alone), every 1,000-row scan — both the cross-server cases (small-rows1000, skip3-rows1000) and the partially colocated (f = 0.6960) case (split-rows1000) — was 23% to 133% slower with USE_REMOTE_FETCH than with SCAN_AND_FETCH.
  • Untested hypothesis: in SCAN_AND_FETCH, the client pipelines GetRange point-range reads directly to the storage servers holding the primary records (with client-side load balancing across the two double replicas). In USE_REMOTE_FETCH, the storage server holding the index shard issues secondary-to-primary lookups to the storage processes holding the primary records (which for process-disjoint teams may be on another process on the same host or across the VPC, as per-lookup routing was not traced) and serializes the combined MappedKeyValue batches back through the index storage server; on 1,000-row and 100 MB split-record scans on ssd-2, that path may bottleneck compared with direct client-pipelined fetches.
• Where USE_REMOTE_FETCH won:
  • skip3-rows10-reverse (0.158 [0.155, 0.160], 4.87 ms vs 31.13 ms): dramatically faster with remote fetch.
  • split-rows10 (0.873 forward, 0.854 reverse, where the 10 target records happened to be colocated on the same storage team as the index shard, f = 0.0000) and small-limit1 (0.979 forward).
  • Client-side Go heap allocations (B/op and allocs/op) were lower with USE_REMOTE_FETCH on every multi-row case (B/op 0.527–0.759, allocs/op 0.398–0.790).
• Implication for defaults: these multi-node VPC measurements show that USE_REMOTE_FETCH is not a universal win and regressed 1,000-row scans in the measured configurations (1.226–2.326×), supporting retaining SCAN_AND_FETCH as the default for the measured workloads and conditions.