Malachi.LogApi (malachi v0.8.13)

View Source

The client-facing log API over a Malachi.BrokerServer: the NorthGuard model, not Kafka's.

A client deals in topic + key (produce) + an opaque cursor (consume). It never sees partitions or offsets: those are internal (ranges and per-range offsets) and deliberately hidden, so the system can split/merge/restripe underneath without breaking clients. That hiding is the point: it is what lets malachi evolve its physical layout where Kafka leaks it to the client.

The cursor is just a token the client echoes back; today it encodes the consumer's position as %{range_id => next_offset}, but its contents are not part of the contract. Because it comes from an untrusted client, decode_cursor/1 bounds its size, uses binary_to_term(_, [:safe]) and validates the shape.

Those steps answer different attacks, and only one of them is about code execution. On OTP 28 the :safe option rejects both a fun and an atom the node has never seen, so a crafted token cannot introduce code or exhaust the atom table.

What :safe does not bound is size, and that takes two guards rather than one. A byte ceiling alone is not enough, because Erlang's external term format has a compressed variant whose header declares an uncompressed size: measured, a 2.6KB token inflates into a million-element list, so a token well under any sane ceiling can still ask for gigabytes. So decode_cursor/1 requires the uncompressed map tag and a ceiling, both before decoding, and valid_positions?/1 then rejects whatever decoded cleanly without being a cursor.

create_topic, produce (by key) and fetch/fetch_group (by opaque cursor) over a topic's ranges, including consumer groups with server-side committed positions, cross-epoch consume across ranges that split, and long-poll (fetch/fetch_group with wait_ms). The read orchestration itself lives in Malachi.BrokerServer (one coherent call that can also park long-poll waiters).

Summary

Types

An opaque consumer position token (treat as a string; do not interpret).

Functions

Durably commits a consumer group's position for topic from cursor (a token from fetch/ fetch_group). Returns :ok, or {:error, :invalid_cursor} for a bad token.

Creates topic. The client does not specify partitions: the keyspace is internal.

Encodes internal consume positions into an opaque cursor. Public because the streaming push path (the connection forwarding {:log_records, ...}) must turn the pushed positions into a client cursor, the same token fetch/fetch_group return.

Fetches up to max records per current range of topic from the position in cursor (nil or :start begins at the beginning). Returns {:ok, records, next_cursor}, advance by passing next_cursor back. Records carry no client-visible offset.

Fetches for a consumer group, resuming from the group's durably committed position (or the beginning if it never committed). Returns {:ok, records, next_cursor}; the client processes the records and then commit/4s next_cursor to advance the durable position (at-least-once).

Fetches for member of consumer group on topic: consults the coordinator for the member's assigned ranges (registering it if new: the fetch is the heartbeat) and consumes only those ranges from the group's committed positions. Returns {:ok, records, next_cursor}; the opaque cursor covers only the member's slice, so commit/4 advances just its ranges (the client never sees a range id). Members of the same group thus consume disjoint records in parallel.

Produces records (each %{"value" => binary, optional "key" => binary, optional "headers" => map}) to topic, routed by key. Returns {:ok, produced_count}; the client gets no offsets.

Appends already-built Malachi.Log.Records (offset unassigned) to topic: the binary protocol path, which decodes records off the wire directly, skipping the JSON map→record step of produce/3. Returns {:ok, count} or {:error, reason}.

Acks count streamed records for group at cursor: durably commits the position (at-least-once) and returns count records of window credit, unblocking further pushes. Returns :ok, or {:error, :invalid_cursor} for a bad token.

Like stream_ack/5 but for a group member: re-polls the coordinator (a heartbeat that also refreshes the member's ranges, so a rebalance is picked up on the next ack) before acking. Returns :ok or {:error, :invalid_cursor}.

Opens a streaming subscription: registers the calling process as a push subscriber of topic for consumer group, bounded by a credit window (max in-flight records) and a max per-push batch. The broker resumes from the group's committed position, pushes an initial backlog, and then pushes new records on produce as {:log_records, topic, records, positions} messages to the caller. Returns :ok.

Like subscribe/5 but for a consumer-group member: registers the member with the coordinator, scopes the push stream to its assigned ranges (server-side: the client still only gets records + an opaque cursor), and lets the broker leave the group when the calling process exits. The member stays alive by acking (stream_ack_member/7); the coordinator does not run in the broker, so the broker never calls it (deadlock-safe).

Types

cursor()

@type cursor() :: String.t()

An opaque consumer position token (treat as a string; do not interpret).

Functions

commit(server, topic, group, cursor)

@spec commit(
  GenServer.server(),
  Malachi.Metadata.topic_name(),
  Malachi.Metadata.group(),
  cursor()
) ::
  :ok | {:error, term()}

Durably commits a consumer group's position for topic from cursor (a token from fetch/ fetch_group). Returns :ok, or {:error, :invalid_cursor} for a bad token.

create_topic(server, topic)

@spec create_topic(GenServer.server(), Malachi.Metadata.topic_name()) ::
  :ok | {:error, term()}

Creates topic. The client does not specify partitions: the keyspace is internal.

encode_cursor(positions)

@spec encode_cursor(map()) :: cursor()

Encodes internal consume positions into an opaque cursor. Public because the streaming push path (the connection forwarding {:log_records, ...}) must turn the pushed positions into a client cursor, the same token fetch/fetch_group return.

fetch(server, topic, cursor, max, wait_ms \\ 0)

@spec fetch(
  GenServer.server(),
  Malachi.Metadata.topic_name(),
  cursor() | nil | :start,
  pos_integer(),
  non_neg_integer()
) :: {:ok, [Malachi.Log.Record.t()], cursor()} | {:error, term()}

Fetches up to max records per current range of topic from the position in cursor (nil or :start begins at the beginning). Returns {:ok, records, next_cursor}, advance by passing next_cursor back. Records carry no client-visible offset.

fetch_group(server, topic, group, max, wait_ms \\ 0)

Fetches for a consumer group, resuming from the group's durably committed position (or the beginning if it never committed). Returns {:ok, records, next_cursor}; the client processes the records and then commit/4s next_cursor to advance the durable position (at-least-once).

fetch_member(server, coordinator, topic, group, member, max, wait_ms \\ 0)

Fetches for member of consumer group on topic: consults the coordinator for the member's assigned ranges (registering it if new: the fetch is the heartbeat) and consumes only those ranges from the group's committed positions. Returns {:ok, records, next_cursor}; the opaque cursor covers only the member's slice, so commit/4 advances just its ranges (the client never sees a range id). Members of the same group thus consume disjoint records in parallel.

produce(server, topic, records)

@spec produce(GenServer.server(), Malachi.Metadata.topic_name(), [map()]) ::
  {:ok, non_neg_integer()} | {:error, term()}

Produces records (each %{"value" => binary, optional "key" => binary, optional "headers" => map}) to topic, routed by key. Returns {:ok, produced_count}; the client gets no offsets.

produce_records(server, topic, records)

@spec produce_records(GenServer.server(), Malachi.Metadata.topic_name(), [
  Malachi.Log.Record.t()
]) ::
  {:ok, non_neg_integer()} | {:error, term()}

Appends already-built Malachi.Log.Records (offset unassigned) to topic: the binary protocol path, which decodes records off the wire directly, skipping the JSON map→record step of produce/3. Returns {:ok, count} or {:error, reason}.

stream_ack(server, topic, group, cursor, count)

@spec stream_ack(
  GenServer.server(),
  Malachi.Metadata.topic_name(),
  Malachi.Metadata.group(),
  cursor(),
  non_neg_integer()
) :: :ok | {:error, term()}

Acks count streamed records for group at cursor: durably commits the position (at-least-once) and returns count records of window credit, unblocking further pushes. Returns :ok, or {:error, :invalid_cursor} for a bad token.

stream_ack_member(server, coordinator, topic, group, member, cursor, count)

@spec stream_ack_member(
  GenServer.server(),
  GenServer.server(),
  Malachi.Metadata.topic_name(),
  Malachi.Metadata.group(),
  term(),
  cursor(),
  non_neg_integer()
) :: :ok | {:error, term()}

Like stream_ack/5 but for a group member: re-polls the coordinator (a heartbeat that also refreshes the member's ranges, so a rebalance is picked up on the next ack) before acking. Returns :ok or {:error, :invalid_cursor}.

subscribe(server, topic, group, window, max)

Opens a streaming subscription: registers the calling process as a push subscriber of topic for consumer group, bounded by a credit window (max in-flight records) and a max per-push batch. The broker resumes from the group's committed position, pushes an initial backlog, and then pushes new records on produce as {:log_records, topic, records, positions} messages to the caller. Returns :ok.

subscribe_member(server, coordinator, topic, group, member, window, max)

@spec subscribe_member(
  GenServer.server(),
  GenServer.server(),
  Malachi.Metadata.topic_name(),
  Malachi.Metadata.group(),
  term(),
  pos_integer(),
  pos_integer()
) :: :ok | {:error, term()}

Like subscribe/5 but for a consumer-group member: registers the member with the coordinator, scopes the push stream to its assigned ranges (server-side: the client still only gets records + an opaque cursor), and lets the broker leave the group when the calling process exits. The member stays alive by acking (stream_ack_member/7); the coordinator does not run in the broker, so the broker never calls it (deadlock-safe).