[hubcio]

Four limits kept a cluster with many partitions from moving its
primaries in time. Each part of this change lifts one of them.

A node that dies takes the primary of every group it led, and the
heartbeat timers of those groups fire within one interval. Each view
change writes two superblocks per replica through the partition I/O
lane. A status timer votes for the next view after 5 s. So the view
changes that the lane does not end in time start a second round.

A shard now casts at most 4 heartbeat votes per tick, and none while
128 of its partitions are in a view change. A held vote keeps its
timer fired, and a backup whose vote is due joins the vote of a peer.
The vote of a replica already in a view change, a primary transfer
and the election of a recovering replica are not held. Two gauges
report the partitions in a view change and the held votes.

A node was not a source or a target of moves while any of its
partitions repaired, transferred state or held an untrusted log. So
one long repair kept every other primary off the node. Each change
of that flag also restarted the 5 s wait of the coordinator. The
node flag is gone. The source capture and the target probe refuse
the partition that moves if it has a transfer session, a scheduled
transfer re-arm or an untrusted log. A node is ready for primaries
once its metadata consensus settles.

The coordinator held one move for the whole cluster and planned one
per second. So the 333,000 primaries of a node that came back needed
about 4 days to return. The coordinator now plans until 512 moves
are in flight, the number of new views that one status reply reports
per second. Each pick counts the moves in flight as done, so a round
ends in the balance of single picks. One round of 512 picks over
300,000 partitions takes 0.7 ms.

A change of eligibility drops every planned move, but a grant that
left can still run. So the worst case of extra moves grows from 1 to
512.

A node that stopped took its primaries down with it. Each of those
groups waited for a heartbeat timeout, 5 s when busy and 12 s when
idle. On the first stop signal, each shard now spreads the
partitions it leads over the in-sync members whose link is up. A tie
goes to the member that an election picks. The limits on view
changes pace the pass, and the node refuses readiness while it
leaves.

The shutdown starts when the last shard ends its handoff, or after
sharding.shutdown_primary_handoff_timeout (15 s). "0 s" skips it, and
so does "unlimited", which reads as zero like "disabled" and "none".
A second signal ends the handoff. The handoff runs before the shutdown
flag, so the drain and the join budget stay as they were. The
metadata group has no primary transfer, so its view change still
waits for a timeout.

Nodes that stop at once can take each other as targets. So while a
shard hands off, a primary that votes to give a group to this node
gets no partition from it. A peer that just restarted votes in no
election, so the pass waits for each handoff for 2 s at most.

On one machine with 3 nodes and 30,000 partitions, a stop of one
node took 6.0 s with no election by timeout. The write pass after it
took 3.4 s, against 20.9 s without the handoff. The node got its
10,000 primaries back in 20 s after its restart.