Malachi.Cluster.Rebalance (malachi v0.8.13)

View Source

Executes a rebalancing plan (from Malachi.Application.rebalance_plan/2) against the vnodes' ra clusters. For each change it adds the joining members before removing the leaving ones (add-before-remove, so a vnode never drops below quorum mid-move) through injected add_member / remove_member seams, so the executor is testable without ra, and R3-b supplies the real ra ops.

It is idempotent: the seams must treat an already-present add / already-gone remove as :ok, so an interrupted commit can be re-run. It is fail-fast: within a change the first failing add stops it (the removes are not attempted, protecting quorum); across a plan the first failing change stops the run, returning what was applied so a later commit resumes. Between changes it re-checks the leader? seam and stops if leadership was lost mid-commit (the lease holder dropped the lease).

Summary

Types

change()
failure()
member_op()

Functions

apply_change(map, add_member, remove_member)

Applies one change: adds every add member, then removes every remove member (never the reverse), stopping at the first failure. Returns :ok or {:error, {step, vnode_id, node, reason}}.

apply_plan(plan, add_member, remove_member, leader? \\ fn -> true end)

Applies a whole plan one change at a time, fail-fast. Before each change it checks leader? (default always) and stops if leadership was lost. Returns {:ok, applied_vnode_ids} or {:error, {applied_vnode_ids, failure}}; idempotent, so re-running resumes.

ra_add_member(vnode_id, new_node, current_members)

Adds new_node to vnode vnode_id's ra cluster (the real add_member seam apply_plan/4 uses in production): starts the ra server on new_node (via :erpc, as a member of the existing cluster) then adds it to the consensus, routing the change through current_members (any of them reaches the leader). Idempotent. An already-running server or already-present member counts as :ok. On success ra replicates the vnode's state (log/snapshot) to the new member automatically.

ra_remove_member(vnode_id, leaving_node, current_members)

Removes leaving_node from vnode vnode_id's ra cluster: removes it from the consensus (routing through current_members), then stops its server. Idempotent. A non-member counts as :ok.

Types

change()

@type change() :: %{vnode_id: atom(), add: [node()], remove: [node()]}

failure()

@type failure() :: {:add | :remove, atom(), node(), term()} | :lost_leadership

member_op()

@type member_op() :: (atom(), node() -> :ok | {:error, term()})

Functions

apply_change(map, add_member, remove_member)

@spec apply_change(change(), member_op(), member_op()) :: :ok | {:error, failure()}

Applies one change: adds every add member, then removes every remove member (never the reverse), stopping at the first failure. Returns :ok or {:error, {step, vnode_id, node, reason}}.

apply_plan(plan, add_member, remove_member, leader? \\ fn -> true end)

@spec apply_plan([change()], member_op(), member_op(), (-> boolean())) ::
  {:ok, [atom()]} | {:error, {[atom()], failure()}}

Applies a whole plan one change at a time, fail-fast. Before each change it checks leader? (default always) and stops if leadership was lost. Returns {:ok, applied_vnode_ids} or {:error, {applied_vnode_ids, failure}}; idempotent, so re-running resumes.

ra_add_member(vnode_id, new_node, current_members)

@spec ra_add_member(atom(), node(), [node()]) :: :ok | {:error, term()}

Adds new_node to vnode vnode_id's ra cluster (the real add_member seam apply_plan/4 uses in production): starts the ra server on new_node (via :erpc, as a member of the existing cluster) then adds it to the consensus, routing the change through current_members (any of them reaches the leader). Idempotent. An already-running server or already-present member counts as :ok. On success ra replicates the vnode's state (log/snapshot) to the new member automatically.

ra_remove_member(vnode_id, leaving_node, current_members)

@spec ra_remove_member(atom(), node(), [node()]) :: :ok | {:error, term()}

Removes leaving_node from vnode vnode_id's ra cluster: removes it from the consensus (routing through current_members), then stops its server. Idempotent. A non-member counts as :ok.