Repository navigation
Conversation
Each node retired ended consumer sessions one at a time. A walk took the ended session with the lowest watermark and proved every stored partition of the node free of it. Then the metadata primary committed one FinalizeSession for that session, and a topic create or delete reset the walk. At 30,000 partitions, a batch of 32 sessions took about 15 minutes to retire. At 1M partitions, a client that touched every stream would leave about 22 minutes of retirement per node. A partition that owes the op refuses new sessions. A walk now proves a batch. Its watermark is the highest end op in the client table when the walk starts, and the batch is every ended session at or below it. Each node lists its stored partitions once per walk, so a create or delete no longer restarts it. A partition read gets the whole batch. It answers retired only when its frontier is at or above the watermark and no session of the batch holds a slice entry or a pending request. A barrier names at most 128 sessions and raises the frontier to the newest end op. So a backup whose walk started later can still prove its copy. A new partition starts with its required metadata frontier at its create op (Partition::created_op), so it refuses every request of the batch. The walk skips each partition created above its watermark, instead of stalling on a partition that no session of the batch can reach. A retried namespace becomes unprovable only when the whole walk proved nothing for 2 s. Before, a backup that read ahead of the barrier of the primary gave up after one retry and recorded a debt. Reports carry the watermark, and the metadata primary finalizes through the lowest watermark of the reports it uses. FinalizeSession (op 70) changes in place. Its body is retired_through plus the optional debt section, and the apply forgets every ended session at or below it. A node whose metadata WAL still holds an old entry fails at boot until a checkpoint trims it. The namespace_revision field of the consumer session heartbeat header becomes retired_through, at the same offset and size. REPLICA_WIRE_REVISION goes to 3. On one machine with 3 nodes and 30,000 partitions, the cluster finalized the 32 sessions of a create pass 118 s after the pass, against 891 s before. It finalized the 32 sessions of a write pass after 89 s, against 903 s. That took 4 FinalizeSession ops instead of 64. The peak of partitions owing retirement fell from 388, 159 and 244 to 19, 11 and 2 per node.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each node retired ended consumer sessions one at a time. A walk took
the ended session with the lowest watermark and proved every stored
partition of the node free of it. Then the metadata primary
committed one FinalizeSession for that session, and a topic create
or delete reset the walk. At 30,000 partitions, a batch of 32
sessions took about 15 minutes to retire. At 1M partitions, a client
that touched every stream would leave about 22 minutes of retirement
per node. A partition that owes the op refuses new sessions.
A walk now proves a batch. Its watermark is the highest end op in
the client table when the walk starts, and the batch is every ended
session at or below it. Each node lists its stored partitions once
per walk, so a create or delete no longer restarts it. A partition
read gets the whole batch. It answers retired only when its frontier
is at or above the watermark and no session of the batch holds a
slice entry or a pending request. A barrier names at most 128
sessions and raises the frontier to the newest end op. So a backup
whose walk started later can still prove its copy.
A new partition starts with its required metadata frontier at its
create op (Partition::created_op), so it refuses every request of
the batch. The walk skips each partition created above its
watermark, instead of stalling on a partition that no session of the
batch can reach. A retried namespace becomes unprovable only when
the whole walk proved nothing for 2 s. Before, a backup that read
ahead of the barrier of the primary gave up after one retry and
recorded a debt.
Reports carry the watermark, and the metadata primary finalizes
through the lowest watermark of the reports it uses. FinalizeSession
(op 70) changes in place. Its body is retired_through plus the
optional debt section, and the apply forgets every ended session at
or below it. A node whose metadata WAL still holds an old entry
fails at boot until a checkpoint trims it. The namespace_revision
field of the consumer session heartbeat header becomes
retired_through, at the same offset and size. REPLICA_WIRE_REVISION
goes to 3.
On one machine with 3 nodes and 30,000 partitions, the cluster
finalized the 32 sessions of a create pass 118 s after the pass,
against 891 s before. It finalized the 32 sessions of a write pass
after 89 s, against 903 s. That took 4 FinalizeSession ops instead
of 64. The peak of partitions owing retirement fell from 388, 159
and 244 to 19, 11 and 2 per node.