Malachi. Telemetry
(malachi v0.8.13)
View Source
The telemetry events Malachi emits on its hot paths. Attach a handler with :telemetry.attach/4 (or
attach_many/4) to feed metrics, logs, or traces: Malachi ships a default handler that folds a few of
these into the ETS Malachi.Metrics (which the Prometheus endpoint exposes).
Events, event name - measurements / metadata:
[:malachi, :produce]:%{count, bytes}/%{topic}- records appended to a topic.[:malachi, :consume]:%{count}/%{topic}- records read from a topic.[:malachi, :auth].%{count: 1}/%{result: :ok | :error}- an authentication attempt.[:malachi, :replication, :commit].%{count}/%{result: :ok | :no_quorum}- a quorum replication of a batch (count= records in the batch;resultis whether a quorum stored it).[:malachi, :storage, :integrity].%{position, unreadable_bytes}/%{result, sealed, source, segment}- a stored segment failed verification.resultis what the verification found, and the list is open rather than closed, so a consumer should have a fallback branch: today it is:bad_crcor:bad_magic(a frame that does not decode),:incomplete(a frame cut short),:short_copy(every frame decodes, but there are fewer records or bytes than the control plane recorded at seal time),:bad_index(the sparse index sidecar does not describe the segment) or a POSIX reason when the device itself could not be read.positionis the byte where the damage starts,sealedwhether the segment was immutable (damage there is corruption at rest, not a crash mid-write), andsourcewhere the verdict came from (:recoverwhen a segment was opened,:scrubfrom the background pass).[:malachi, :storage, :scrub].%{verified, damaged, repaired, unrepairable}/%{}- one background verification pass finished, with how many segments it covered. Steady progress withdamaged: 0is what a healthy node looks like; no events at all means the scrub is not running.
Emitting is a no-op fast path when nothing is attached, so these are safe on the hot path.
Summary
Functions
An authentication attempt with its result (:ok or :error).
Records read from topic (count records).
Records appended to topic (count records, bytes total value bytes).
A quorum replication of a batch of count records with its result.
One background scrub pass finished, with the segments it verified, found damaged and repaired.
A stored segment failed verification: verdict is the storage layer's finding (:reason,
:position, :unreadable_bytes, :sealed?), segment_id names the segment and source says
whether it surfaced while opening the segment (:recover) or during the background scrub.
Functions
@spec auth(:ok | :error) :: :ok
An authentication attempt with its result (:ok or :error).
@spec consume(String.t(), non_neg_integer()) :: :ok
Records read from topic (count records).
@spec produce(String.t(), non_neg_integer(), non_neg_integer()) :: :ok
Records appended to topic (count records, bytes total value bytes).
@spec replication_commit(non_neg_integer(), :ok | :no_quorum) :: :ok
A quorum replication of a batch of count records with its result.
@spec scrub_pass( non_neg_integer(), non_neg_integer(), non_neg_integer(), non_neg_integer() ) :: :ok
One background scrub pass finished, with the segments it verified, found damaged and repaired.
A stored segment failed verification: verdict is the storage layer's finding (:reason,
:position, :unreadable_bytes, :sealed?), segment_id names the segment and source says
whether it surfaced while opening the segment (:recover) or during the background scrub.