Phase C1: Remote fetch (forward)

Part of the hcloud-rebench-20261014 campaign.

BenchmarkIndexFetch in two trees and two fetch methods, counterbalanced. USE_REMOTE_FETCH asks the storage server to resolve each index entry to its record (a mapped range read); SCAN_AND_FETCH scans the index and loads each record from the client. The question: what does remote fetch buy over the ordinary path, both in the tree that introduced it and in the final tree?

Arms

Both trees ran byte-identical benchmark source (pkg/recordlayer/benchfetch, harness d94b4c61377d173b7c6884fde426a227dab0f0bb; see campaign-harness.md).

Environment and design

• Host, CPU groups and network path as in phase A and phase B: Hetzner ccx43, one fdbserver 7.3.77 process on CPUs 2–7 with the memory engine and single redundancy, the benchmark process on CPUs 8–15 in a network namespace reaching the server over a veth pair.
• Fixture: 1,000 records behind a value index, seeded and read back before any timer. small records are one key-value each; split records carry a 100,000-byte payload and are stored as two chunks. Every measured operation’s result (IDs, order, fields, payload checksum, terminal reason) is checked with the timer stopped, and an instrumented run before and after the loop asserts the path: remote fetch with no index scan or record load, or an index scan with record loads for SCAN_AND_FETCH.
• Cases (all forward scans):
  • limit-1-broad: a scan over the whole index that stops after one row;
  • exact-10: IDs 100–109;
  • all-1000: all 1,000 records.
• Per case: 8 blocks over the four arms in a Williams-square order (seed 641201), so each arm precedes each other arm equally often. One fresh process per arm and block, three repeats each (-test.count=3, 3 s per repeat): 24 samples per arm and case, 192 processes and 576 samples in total.
• Estimand and bootstrap as in phase B: geometric mean over blocks of each block’s ratio of process medians, nominal 95% percentile bootstrap over blocks (20,000 resamples, seed 391706). Absolute values are the median over blocks of the process medians. ns/op is testing.B’s mean, not a percentile.

Results

Ratios with nominal 95% intervals. Below 1 means the numerator arm is faster or allocates less.

Remote fetch vs ordinary fetch, final tree

final-remote / final-ordinary.

Remote fetch vs ordinary fetch, first remote-fetch tree

pr2-remote / pr2-ordinary.

Final vs first remote-fetch tree, per method

final-remote / pr2-remote:

final-ordinary / pr2-ordinary (control: the ordinary path):

Absolute values

Median over blocks of each process’s median of its three repeats.

Final tree, allocations per operation:

Interpretation

Measured:

• Remote fetch is faster in five of six cases, in both trees. In the final tree it takes 0.71–0.87 of the ordinary path’s time for the one-row and ten-row scans and 0.34 for the 1,000 small records (44.0 → 15.3 ms).
• One retained negative result: 1,000 split records. Remote fetch is 15% slower in the final tree (1.150 [1.129, 1.181]; 163 → 188 ms) and 21% slower in the first tree (1.213), although it allocates 31% fewer bytes and 60% fewer objects. Every one of the 8 block ratios is above 1.11.
• The final tree improved remote fetch itself. final-remote/pr2-remote has allocs/op 0.62–0.95 in every case and time 0.80 [0.77, 0.82] for 1,000 small records and 0.95 [0.93, 0.97] for 1,000 split records. In the one-row and ten-row cases time is 0.94–1.01; only split-limit-1-broad (0.944 [0.891, 0.996]) and small-exact-10 (0.975 [0.962, 0.987]) have intervals below 1.
• The ordinary path did not change between the trees in bytes or allocations (all ratios within 0.1% of 1). Its time ratios are 1.002–1.037; split-exact-10 (1.037 [1.013, 1.060]) and small-all-1000 (1.024 [1.010, 1.040]) have intervals above 1. With identical allocations, this run does not show where those 2–4% come from.

Hypotheses, not tested here:

• For 1,000 split records, the storage server reads and returns about 100 MB per operation itself; on one fdbserver process that serial server-side work may cost more than the client-side pipelined loads it replaces. The allocation ratios show the client does less work; server CPU was not profiled.
• The benefit measured here excludes inter-host and physical-network latency: client and server share a host and talk over a veth pair. Remote fetch saves client round trips, so its effect over a real network is likely different. The multi-node campaign (tools/benchcampaign/multinode) is designed to measure that.

Limits

• Single host, one fdbserver process, single redundancy, memory engine: no replication, no network hop, no disk. One operation at a time.
• testing.B reports means; no latency percentiles.
• Eight blocks give limited resolution; intervals are nominal, per case and metric, with no multiplicity correction and no aggregate across cases. Numerator-first and denominator-first blocks agree in direction for every remote/ordinary time ratio; the largest spread is pr2 split-exact-10 (0.947 vs 0.855).
• Case labels in the aggregate CSV were reconstructed from the pinned plan and the unchanged per-run logs, for the same runner bug described in phase B; 576 labels were restored in a separate working copy and no measurement was repeated or changed.