G
YouTube TikTok Games
Skip to content

perf(shard): tick only partitions with due work - #4905

Open
hubcio wants to merge 1 commit into
perf/retire-sessions-at-scalefrom
perf/partition-deadline-scheduler
Open

hubcio wants to merge 1 commit into
perf/retire-sessions-at-scalefrom
perf/partition-deadline-scheduler

Conversation

@hubcio

@hubcio hubcio commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

The consensus tick visited every partition of a shard every 10 ms,
and a visit of an idle partition found nothing to do. Idle CPU
grew with the partition count: about 350% of one core per node at
30,000 partitions.

A partition now sleeps while it is quiet: Normal status, nothing
to commit, replicate, persist, repair or transfer, and no report
or gauge that waits for a visit. The partition table keeps one
wake tick per slot, the earliest timer deadline of the partition.
Every accessor that hands out a partition wakes it, so a frame, an
I/O turn or a reconcile step needs no call of its own. A partition
that is not plainly idle stays due on every tick, so the visit
countdowns of repair, transfer and the view fence run as before.

The WAL and repair ring gauges count sleeping partitions too, so
they are now sums of what each partition showed at its last visit.
A debug build tests every sleeping partition on every tick and
fails on one with work. A release build tests eight per tick,
counts them in partition_missed_wakeups and wakes them. The
simulator and its fuzzer can make every tick visit every
partition, and tests require the same executor schedule in both
modes, also across crashes, restarts and state transfers.

Three idle nodes with 30,000 partitions each: 350% -> 13% of one
core per node. A node visited 30,000 partitions on each of its 100
ticks per second, and partition_tick_visits now counts 50,000
visits per second.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Nov 14, 2026
@hubcio
hubcio added this pull request to stack #4757 November 14, 2026 13:54
The consensus tick visited every partition of a shard every 10 ms,
and a visit of an idle partition found nothing to do. Idle CPU
grew with the partition count: about 350% of one core per node at
30,000 partitions.

A partition now sleeps while it is quiet: Normal status, nothing
to commit, replicate, persist, repair or transfer, and no report
or gauge that waits for a visit. The partition table keeps one
wake tick per slot, the earliest timer deadline of the partition.
Every accessor that hands out a partition wakes it, so a frame, an
I/O turn or a reconcile step needs no call of its own. A partition
that is not plainly idle stays due on every tick, so the visit
countdowns of repair, transfer and the view fence run as before.

The WAL and repair ring gauges count sleeping partitions too, so
they are now sums of what each partition showed at its last visit.
A debug build tests every sleeping partition on every tick and
fails on one with work. A release build tests eight per tick,
counts them in partition_missed_wakeups and wakes them. The
simulator and its fuzzer can make every tick visit every
partition, and tests require the same executor schedule in both
modes, also across crashes, restarts and state transfers.

Three idle nodes with 30,000 partitions each: 350% -> 13% of one
core per node. A node visited 30,000 partitions on each of its 100
ticks per second, and partition_tick_visits now counts 50,000
visits per second.
@hubcio
hubcio force-pushed the perf/partition-deadline-scheduler branch from 3d5a0c5 to c0cd463 Compare November 14, 2026 17:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant