Malachi.Cluster.RingTopology (malachi v0.8.13)

View Source

The cluster's routing topology as a versioned, gossip-disseminable value: a HashRing (which vnode owns which hash arc) plus each vnode's node placement (%{vnode_id => [node]}), tagged with a monotonic version.

This is how a vnode split propagates. NorthGuard keeps "very minimal global states" and spreads them over its SWIM dissemination path (the meetup transcript: "we also use this dissemination for spreading some minimal global like cluster metadata"); the ring is exactly such minimal global state. So rather than a dedicated Raft cluster for the ring (strongly consistent but heavier and less faithful), the ring rides the existing gossip and every node converges on the latest version.

Convergence is a CRDT join: merge/2 keeps the higher version (last-version-wins), with a deterministic term-order tiebreak on the rare same-version clash, so merge is commutative, associative and idempotent, and every node reaches the same topology regardless of gossip order. Only the rebalancing leader advance/3s the version (one writer), so a same-version clash should not arise in normal operation; the tiebreak just keeps the math total.

The topology also carries an optional pending-split intent (begin_split/4): the durable record of a split that is migrating but not yet complete. It rides the same gossip, so if the coordinator driving a split crashes, whichever node takes over the lease reads the intent and reconciles the interrupted split (see pending/0). advance/3 (complete) and clear_pending/1 (abort) both clear it.

Summary

Types

pending()

A split in flight: the durable intent recorded before a split migrates, so a coordinator that takes over the lease after the previous one crashed mid-split can reconcile it (nil when none is pending). Carries what a reconciler needs to identify and undo the split: the new vnode, its ring token, and its placement nodes (the old ring/placements are the topology's own, since a pending split has not yet advanced the ring).

placements()
t()

Functions

advance(ring_topology, ring, placements)

A new topology at version + 1 with ring/placements: the single-writer bump the rebalancing leader applies to complete a split (or any ring change). The monotonic version is what makes merge/2 converge. Completing a ring change also clears any pending-split intent: the split it recorded is now reflected in the ring, so there is nothing left to reconcile.

begin_split(topology, new_vnode_id, token, nodes)

Records a pending split at version + 1 without changing the ring: the intent that a split of new_vnode_id at token onto nodes is now in flight. Written (and gossiped) before the migration so a coordinator that takes over the lease mid-split can find and reconcile it. The ring still routes by the pre-split placement until advance/3 completes the split (or clear_pending/1 aborts it).

clear_pending(topology)

Drops a pending-split intent at version + 1, keeping the ring unchanged, how an aborted split is published (the reconciler rolled the migration back, so the ring stays as it was and the intent is gone). Contrast advance/3, which clears the intent by moving the ring forward to the completed split.

merge(a, b)

CRDT join of two observations of the topology: the higher version wins; a same-version clash is broken deterministically by Erlang term order (so the result is independent of argument order). Idempotent (merge(a, a) == a), commutative and associative.

new(ring, placements)

The initial (version 0) topology for ring and its vnode placements; no split pending.

servers(ring_topology)

The %{vnode_id => server_id} routing map derived from the placements: each vnode's server id is {vnode_id, a_member} (any member of its placement: ra resolves the live leader). A vnode with an empty placement is skipped (it has no cluster to route to). Used to point routing at the topology.

Types

pending()

@type pending() ::
  %{
    new_vnode: Malachi.Cluster.HashRing.vnode_id(),
    token: Malachi.Cluster.HashRing.token(),
    nodes: [node()]
  }
  | nil

A split in flight: the durable intent recorded before a split migrates, so a coordinator that takes over the lease after the previous one crashed mid-split can reconcile it (nil when none is pending). Carries what a reconciler needs to identify and undo the split: the new vnode, its ring token, and its placement nodes (the old ring/placements are the topology's own, since a pending split has not yet advanced the ring).

placements()

@type placements() :: %{required(term()) => [node()]}

t()

@type t() :: %Malachi.Cluster.RingTopology{
  pending: pending(),
  placements: placements(),
  ring: Malachi.Cluster.HashRing.t() | nil,
  version: non_neg_integer()
}

Functions

advance(ring_topology, ring, placements)

@spec advance(t(), Malachi.Cluster.HashRing.t(), placements()) :: t()

A new topology at version + 1 with ring/placements: the single-writer bump the rebalancing leader applies to complete a split (or any ring change). The monotonic version is what makes merge/2 converge. Completing a ring change also clears any pending-split intent: the split it recorded is now reflected in the ring, so there is nothing left to reconcile.

begin_split(topology, new_vnode_id, token, nodes)

@spec begin_split(
  t(),
  Malachi.Cluster.HashRing.vnode_id(),
  Malachi.Cluster.HashRing.token(),
  [node()]
) ::
  t()

Records a pending split at version + 1 without changing the ring: the intent that a split of new_vnode_id at token onto nodes is now in flight. Written (and gossiped) before the migration so a coordinator that takes over the lease mid-split can find and reconcile it. The ring still routes by the pre-split placement until advance/3 completes the split (or clear_pending/1 aborts it).

clear_pending(topology)

@spec clear_pending(t()) :: t()

Drops a pending-split intent at version + 1, keeping the ring unchanged, how an aborted split is published (the reconciler rolled the migration back, so the ring stays as it was and the intent is gone). Contrast advance/3, which clears the intent by moving the ring forward to the completed split.

merge(a, b)

@spec merge(t(), t()) :: t()

CRDT join of two observations of the topology: the higher version wins; a same-version clash is broken deterministically by Erlang term order (so the result is independent of argument order). Idempotent (merge(a, a) == a), commutative and associative.

new(ring, placements)

@spec new(Malachi.Cluster.HashRing.t(), placements()) :: t()

The initial (version 0) topology for ring and its vnode placements; no split pending.

servers(ring_topology)

@spec servers(t()) :: %{required(term()) => {term(), node()}}

The %{vnode_id => server_id} routing map derived from the placements: each vnode's server id is {vnode_id, a_member} (any member of its placement: ra resolves the live leader). A vnode with an empty placement is skipped (it has no cluster to route to). Used to point routing at the topology.