Malachi.Cluster.SelfHealing (malachi v0.8.13)

View Source

Closes the self-healing loop for sealed segments: the decision comes from Malachi.Cluster.Placement and the execution from Malachi.Cluster.Catchup.

Given the current Malachi.Metadata, the set of live brokers, and the replication factor, heal_sealed/4 finds the sealed segments that are under-replicated (a replica left), picks the healed replica set with Placement.place/3, and backfills each newly added replica with the segment's records (copied from a surviving replica via Catchup.run/6). It returns the :set_segment_replicas commands whose backfill succeeded, for the caller to apply through the control plane (Malachi.Metadata / the replicated DS-RSM), plus any segments it could not heal.

Only sealed segments are handled here: their offset range [start_offset, start_offset + length) is fixed, so a backfill is a well-defined copy. The active segment grows as it is written, so a follower that falls behind on it rejoins via the write-path catch-up trigger (a separate slice) rather than a one-shot backfill.

Brokers are Malachi.Cluster.ReplicationServer references. The live-broker set is supplied by the caller (membership is a separate concern); a segment whose every replica is dead cannot be backfilled and is reported as failed rather than silently dropped.

Besides the metadata-level pass, heal_sealed/4 also runs a physical integrity pass over sealed segments whose replica set looks healthy: a live node can lose a sealed copy without the metadata noticing (a deleted or truncated file, a swapped disk), leaving the cluster silently below its replication factor while every replica is "alive". The pass probes each live replica's stored bytes (Malachi.Cluster.ReplicationServer.stored_bytes/3, a read-only stat that never opens the log, so steady state costs no descriptors); only a replica whose bytes fall short of the sealed byte_size is then opened for its exact durable end (Malachi.Cluster.ReplicationServer.durable_end/4) and re-backfilled from an intact replica via Malachi.Cluster.Catchup, resuming at the truncation point rather than recopying the segment. In-place corruption that keeps the byte size is out of this probe's reach by construction, since it compares sizes: that is what Malachi.Cluster.Scrubber verifies checksums for.

Summary

Types

result()

Functions

heal_sealed(metadata, live_brokers, replication_factor, opts \\ [])

Backfills and heals every under-replicated sealed segment. Returns %{applied: commands, failed: [{segment_id, reason}]}, applied are :set_segment_replicas commands to apply through the control plane; failed segments could not be backfilled (e.g. :no_live_source).

Types

result()

@type result() :: %{
  applied: [Malachi.Metadata.command()],
  failed: [{Malachi.Metadata.segment_id(), term()}],
  repaired: [{Malachi.Metadata.segment_id(), Malachi.Metadata.broker()}]
}

Functions

heal_sealed(metadata, live_brokers, replication_factor, opts \\ [])

@spec heal_sealed(
  Malachi.Metadata.t(),
  [Malachi.Metadata.broker()],
  pos_integer(),
  keyword()
) ::
  result()

Backfills and heals every under-replicated sealed segment. Returns %{applied: commands, failed: [{segment_id, reason}]}, applied are :set_segment_replicas commands to apply through the control plane; failed segments could not be backfilled (e.g. :no_live_source).

Options

  • :batch_size - forwarded to Malachi.Cluster.Catchup.run/6.
  • :spread - {attribute_key, attributes} forwarded to Placement.place/4 so re-replication stays rack/DC-aware. Best-effort only: any :min_domains/:policy is intentionally not forwarded: healing prioritises durability and never fails a re-replication for domain diversity.