Malachi. LogApi
(malachi v0.8.13)
View Source
The client-facing log API over a Malachi.BrokerServer: the NorthGuard model, not Kafka's.
A client deals in topic + key (produce) + an opaque cursor (consume). It never sees
partitions or offsets: those are internal (ranges and per-range offsets) and deliberately hidden,
so the system can split/merge/restripe underneath without breaking clients. That hiding is the
point: it is what lets malachi evolve its physical layout where Kafka leaks it to the client.
The cursor is just a token the client echoes back; today it encodes the consumer's position as
%{range_id => next_offset}, but its contents are not part of the contract. Because it comes from
an untrusted client, decode_cursor/1 bounds its size, uses binary_to_term(_, [:safe]) and
validates the shape.
Those steps answer different attacks, and only one of them is about code execution. On OTP 28 the
:safe option rejects both a fun and an atom the node has never seen, so a crafted token cannot
introduce code or exhaust the atom table.
What :safe does not bound is size, and that takes two guards rather than one. A byte ceiling
alone is not enough, because Erlang's external term format has a compressed variant whose header
declares an uncompressed size: measured, a 2.6KB token inflates into a million-element list, so a
token well under any sane ceiling can still ask for gigabytes. So decode_cursor/1 requires the
uncompressed map tag and a ceiling, both before decoding, and valid_positions?/1 then rejects
whatever decoded cleanly without being a cursor.
create_topic, produce (by key) and fetch/fetch_group (by opaque cursor) over a topic's
ranges, including consumer groups with server-side committed positions, cross-epoch consume across
ranges that split, and long-poll (fetch/fetch_group with wait_ms). The read orchestration
itself lives in Malachi.BrokerServer (one coherent call that can also park long-poll waiters).
Summary
Functions
Durably commits a consumer group's position for topic from cursor (a token from fetch/
fetch_group). Returns :ok, or {:error, :invalid_cursor} for a bad token.
Creates topic. The client does not specify partitions: the keyspace is internal.
Encodes internal consume positions into an opaque cursor. Public because the streaming push path
(the connection forwarding {:log_records, ...}) must turn the pushed positions into a client cursor,
the same token fetch/fetch_group return.
Fetches up to max records per current range of topic from the position in cursor (nil or
:start begins at the beginning). Returns {:ok, records, next_cursor}, advance by passing
next_cursor back. Records carry no client-visible offset.
Fetches for a consumer group, resuming from the group's durably committed position (or the
beginning if it never committed). Returns {:ok, records, next_cursor}; the client processes the
records and then commit/4s next_cursor to advance the durable position (at-least-once).
Fetches for member of consumer group on topic: consults the coordinator for the member's
assigned ranges (registering it if new: the fetch is the heartbeat) and consumes only those ranges
from the group's committed positions. Returns {:ok, records, next_cursor}; the opaque cursor covers
only the member's slice, so commit/4 advances just its ranges (the client never sees a range id).
Members of the same group thus consume disjoint records in parallel.
Produces records (each %{"value" => binary, optional "key" => binary, optional "headers" => map}) to topic, routed by key. Returns {:ok, produced_count}; the client gets no offsets.
Appends already-built Malachi.Log.Records (offset unassigned) to topic: the binary protocol path,
which decodes records off the wire directly, skipping the JSON map→record step of produce/3. Returns
{:ok, count} or {:error, reason}.
Acks count streamed records for group at cursor: durably commits the position (at-least-once) and
returns count records of window credit, unblocking further pushes. Returns :ok, or
{:error, :invalid_cursor} for a bad token.
Like stream_ack/5 but for a group member: re-polls the coordinator (a heartbeat that also
refreshes the member's ranges, so a rebalance is picked up on the next ack) before acking. Returns :ok
or {:error, :invalid_cursor}.
Opens a streaming subscription: registers the calling process as a push subscriber of topic for
consumer group, bounded by a credit window (max in-flight records) and a max per-push batch. The
broker resumes from the group's committed position, pushes an initial backlog, and then pushes new
records on produce as {:log_records, topic, records, positions} messages to the caller. Returns :ok.
Like subscribe/5 but for a consumer-group member: registers the member with the coordinator,
scopes the push stream to its assigned ranges (server-side: the client still only gets records + an
opaque cursor), and lets the broker leave the group when the calling process exits. The member stays
alive by acking (stream_ack_member/7); the coordinator does not run in the broker, so the broker never
calls it (deadlock-safe).
Types
@type cursor() :: String.t()
An opaque consumer position token (treat as a string; do not interpret).
Functions
@spec commit( GenServer.server(), Malachi.Metadata.topic_name(), Malachi.Metadata.group(), cursor() ) :: :ok | {:error, term()}
Durably commits a consumer group's position for topic from cursor (a token from fetch/
fetch_group). Returns :ok, or {:error, :invalid_cursor} for a bad token.
@spec create_topic(GenServer.server(), Malachi.Metadata.topic_name()) :: :ok | {:error, term()}
Creates topic. The client does not specify partitions: the keyspace is internal.
Encodes internal consume positions into an opaque cursor. Public because the streaming push path
(the connection forwarding {:log_records, ...}) must turn the pushed positions into a client cursor,
the same token fetch/fetch_group return.
@spec fetch( GenServer.server(), Malachi.Metadata.topic_name(), cursor() | nil | :start, pos_integer(), non_neg_integer() ) :: {:ok, [Malachi.Log.Record.t()], cursor()} | {:error, term()}
Fetches up to max records per current range of topic from the position in cursor (nil or
:start begins at the beginning). Returns {:ok, records, next_cursor}, advance by passing
next_cursor back. Records carry no client-visible offset.
@spec fetch_group( GenServer.server(), Malachi.Metadata.topic_name(), Malachi.Metadata.group(), pos_integer(), non_neg_integer() ) :: {:ok, [Malachi.Log.Record.t()], cursor()} | {:error, term()}
Fetches for a consumer group, resuming from the group's durably committed position (or the
beginning if it never committed). Returns {:ok, records, next_cursor}; the client processes the
records and then commit/4s next_cursor to advance the durable position (at-least-once).
@spec fetch_member( GenServer.server(), GenServer.server(), Malachi.Metadata.topic_name(), Malachi.Metadata.group(), term(), pos_integer(), non_neg_integer() ) :: {:ok, [Malachi.Log.Record.t()], cursor()} | {:error, term()}
Fetches for member of consumer group on topic: consults the coordinator for the member's
assigned ranges (registering it if new: the fetch is the heartbeat) and consumes only those ranges
from the group's committed positions. Returns {:ok, records, next_cursor}; the opaque cursor covers
only the member's slice, so commit/4 advances just its ranges (the client never sees a range id).
Members of the same group thus consume disjoint records in parallel.
@spec produce(GenServer.server(), Malachi.Metadata.topic_name(), [map()]) :: {:ok, non_neg_integer()} | {:error, term()}
Produces records (each %{"value" => binary, optional "key" => binary, optional "headers" => map}) to topic, routed by key. Returns {:ok, produced_count}; the client gets no offsets.
@spec produce_records(GenServer.server(), Malachi.Metadata.topic_name(), [ Malachi.Log.Record.t() ]) :: {:ok, non_neg_integer()} | {:error, term()}
Appends already-built Malachi.Log.Records (offset unassigned) to topic: the binary protocol path,
which decodes records off the wire directly, skipping the JSON map→record step of produce/3. Returns
{:ok, count} or {:error, reason}.
@spec stream_ack( GenServer.server(), Malachi.Metadata.topic_name(), Malachi.Metadata.group(), cursor(), non_neg_integer() ) :: :ok | {:error, term()}
Acks count streamed records for group at cursor: durably commits the position (at-least-once) and
returns count records of window credit, unblocking further pushes. Returns :ok, or
{:error, :invalid_cursor} for a bad token.
@spec stream_ack_member( GenServer.server(), GenServer.server(), Malachi.Metadata.topic_name(), Malachi.Metadata.group(), term(), cursor(), non_neg_integer() ) :: :ok | {:error, term()}
Like stream_ack/5 but for a group member: re-polls the coordinator (a heartbeat that also
refreshes the member's ranges, so a rebalance is picked up on the next ack) before acking. Returns :ok
or {:error, :invalid_cursor}.
@spec subscribe( GenServer.server(), Malachi.Metadata.topic_name(), Malachi.Metadata.group(), pos_integer(), pos_integer() ) :: :ok
Opens a streaming subscription: registers the calling process as a push subscriber of topic for
consumer group, bounded by a credit window (max in-flight records) and a max per-push batch. The
broker resumes from the group's committed position, pushes an initial backlog, and then pushes new
records on produce as {:log_records, topic, records, positions} messages to the caller. Returns :ok.
@spec subscribe_member( GenServer.server(), GenServer.server(), Malachi.Metadata.topic_name(), Malachi.Metadata.group(), term(), pos_integer(), pos_integer() ) :: :ok | {:error, term()}
Like subscribe/5 but for a consumer-group member: registers the member with the coordinator,
scopes the push stream to its assigned ranges (server-side: the client still only gets records + an
opaque cursor), and lets the broker leave the group when the calling process exits. The member stays
alive by acking (stream_ack_member/7); the coordinator does not run in the broker, so the broker never
calls it (deadlock-safe).