Background verification of data at rest (a "scrub"), and repair of what it finds.

A sealed segment is immutable, and nothing re-reads it: Malachi.Log.recover/2 trusts sealed files by name, so a checksum is only ever confirmed when a consumer happens to read that exact frame. Bit rot there is therefore silent, and worse than an error: the read bound comes from the control plane's record count rather than from the file, so a read past the damaged frame returns no records, which the broker reads as "this source is drained". The consumer then stalls at that offset or skips the rest of the range, with nothing logged anywhere.

This worker closes that by walking the node's own sealed segments on a slow cadence, verifying every frame's checksum (Malachi.Log.verify/2), and repairing a damaged copy from an intact replica. NorthGuard's rule that you are on the clock to re-replicate a lost replica applies here too: a copy that will not decode is a lost replica, whatever the metadata says about it.

Why it runs on every node, not just the leader

Unlike the healing and retention coordinators, this one is deliberately not leader-gated: each node is the only one that can read its own disk, and corruption is a per-copy fact, not a cluster-wide one. Every node scrubs what it stores.

Repair, and its one safety rule

Damage is repaired by fetching the segment again from a replica that still has it intact, which means deleting the local copy first. The rule is that the local copy is never destroyed before a peer has confirmed it holds the whole segment, intact: swapping a partially readable copy for a shorter one, or for no copy at all, would be worse than the damage. Both halves of that sentence carry weight, and the second is the one easy to lose: a peer whose checksums all pass can still be missing whole frames at the end, and repairing from it deletes the local copy and then catches up to an offset the source cannot reach. So a peer qualifies only when its scan matches what the control plane recorded at seal time, which is the same test cross_check/4 applies to this node's own copy. Peers are asked through their own Scrubber (never the replication server), so the verification scan stays off the replication hot loop.

When the damaged copy is the segment's primary, the repair first submits a :set_segment_replicas that moves this node to the end of the replica set. Reads follow the head of that set, so they move to an intact replica immediately instead of hitting the hole (or the gap left while the copy is being refetched).

Every pass returns its outcome; logging is only the default :on_result callback, so tests assert on data rather than on log lines.

Options

• :metadata_source - (-> Malachi.Metadata.t()), the current metadata (required);
• :local_ref - this node's replication server reference, or a (-> ref) resolved per pass (required). A single-node broker owns an unnamed replication server, so its reference is a pid that a broker restart replaces: passing a function keeps the scrub from silently matching nothing after such a restart;
• :directory - the data directory whose segments are scrubbed (required);
• :apply_command - (Metadata.command() -> any), applies the demote command (required);
• :peer_scrubber - (node() -> GenServer.server()), how to reach a peer's scrubber. Defaults to this process's own registered name on that node, since every node runs the same worker under the same name;
• :interval - ms between ticks (default 60s). A value that is not a positive integer is refused with a warning and the default used instead, since it arrives from the environment;
• :segments_per_tick - segments verified per tick (default 1). With the default 64MB segment size a full cycle takes segments * interval / segments_per_tick, so a node holding 10k sealed segments revisits each one about weekly;
• :on_result - (result -> any) for each pass (default: log damage and repairs).