G
YouTube TikTok Games
Skip to content

perf(server): retire sessions at 1M partitions - #4904

Open
hubcio wants to merge 1 commit into
perf/bulk-primary-movesfrom
perf/retire-sessions-at-scale
Open

hubcio wants to merge 1 commit into
perf/bulk-primary-movesfrom
perf/retire-sessions-at-scale

Conversation

@hubcio

@hubcio hubcio commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Each node retired ended consumer sessions one at a time. A walk took
the ended session with the lowest watermark and proved every stored
partition of the node free of it. Then the metadata primary
committed one FinalizeSession for that session, and a topic create
or delete reset the walk. At 30,000 partitions, a batch of 32
sessions took about 15 minutes to retire. At 1M partitions, a client
that touched every stream would leave about 22 minutes of retirement
per node. A partition that owes the op refuses new sessions.

A walk now proves a batch. Its watermark is the highest end op in
the client table when the walk starts, and the batch is every ended
session at or below it. Each node lists its stored partitions once
per walk, so a create or delete no longer restarts it. A partition
read gets the whole batch. It answers retired only when its frontier
is at or above the watermark and no session of the batch holds a
slice entry or a pending request. A barrier names at most 128
sessions and raises the frontier to the newest end op. So a backup
whose walk started later can still prove its copy.

A new partition starts with its required metadata frontier at its
create op (Partition::created_op), so it refuses every request of
the batch. The walk skips each partition created above its
watermark, instead of stalling on a partition that no session of the
batch can reach. A retried namespace becomes unprovable only when
the whole walk proved nothing for 2 s. Before, a backup that read
ahead of the barrier of the primary gave up after one retry and
recorded a debt.

Reports carry the watermark, and the metadata primary finalizes
through the lowest watermark of the reports it uses. FinalizeSession
(op 70) changes in place. Its body is retired_through plus the
optional debt section, and the apply forgets every ended session at
or below it. A node whose metadata WAL still holds an old entry
fails at boot until a checkpoint trims it. The namespace_revision
field of the consumer session heartbeat header becomes
retired_through, at the same offset and size. REPLICA_WIRE_REVISION
goes to 3.

On one machine with 3 nodes and 30,000 partitions, the cluster
finalized the 32 sessions of a create pass 118 s after the pass,
against 891 s before. It finalized the 32 sessions of a write pass
after 89 s, against 903 s. That took 4 FinalizeSession ops instead
of 64. The peak of partitions owing retirement fell from 388, 159
and 244 to 19, 11 and 2 per node.

Each node retired ended consumer sessions one at a time. A walk took
the ended session with the lowest watermark and proved every stored
partition of the node free of it. Then the metadata primary
committed one FinalizeSession for that session, and a topic create
or delete reset the walk. At 30,000 partitions, a batch of 32
sessions took about 15 minutes to retire. At 1M partitions, a client
that touched every stream would leave about 22 minutes of retirement
per node. A partition that owes the op refuses new sessions.

A walk now proves a batch. Its watermark is the highest end op in
the client table when the walk starts, and the batch is every ended
session at or below it. Each node lists its stored partitions once
per walk, so a create or delete no longer restarts it. A partition
read gets the whole batch. It answers retired only when its frontier
is at or above the watermark and no session of the batch holds a
slice entry or a pending request. A barrier names at most 128
sessions and raises the frontier to the newest end op. So a backup
whose walk started later can still prove its copy.

A new partition starts with its required metadata frontier at its
create op (Partition::created_op), so it refuses every request of
the batch. The walk skips each partition created above its
watermark, instead of stalling on a partition that no session of the
batch can reach. A retried namespace becomes unprovable only when
the whole walk proved nothing for 2 s. Before, a backup that read
ahead of the barrier of the primary gave up after one retry and
recorded a debt.

Reports carry the watermark, and the metadata primary finalizes
through the lowest watermark of the reports it uses. FinalizeSession
(op 70) changes in place. Its body is retired_through plus the
optional debt section, and the apply forgets every ended session at
or below it. A node whose metadata WAL still holds an old entry
fails at boot until a checkpoint trims it. The namespace_revision
field of the consumer session heartbeat header becomes
retired_through, at the same offset and size. REPLICA_WIRE_REVISION
goes to 3.

On one machine with 3 nodes and 30,000 partitions, the cluster
finalized the 32 sessions of a create pass 118 s after the pass,
against 891 s before. It finalized the 32 sessions of a write pass
after 89 s, against 903 s. That took 4 FinalizeSession ops instead
of 64. The peak of partitions owing retirement fell from 388, 159
and 244 to 19, 11 and 2 per node.
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Nov 14, 2026
@hubcio
hubcio added this pull request to stack #4757 November 14, 2026 10:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant