Repository navigation
CKS: running clusters cannot be scaled live on a default install, because the KVM SystemVM template is not dynamically scalable #14704
Description
In short
#13226 added live offering changes for running CKS clusters on KVM. On a default install that feature never works: CKS nodes use the SystemVM template, which is registered as not dynamically scalable, so every node is created as not scalable and the scale fails. Proposal: register KVM SystemVM templates as dynamically scalable by default. We tested the side effects and found nothing that should stop this.
How to reproduce
On a default KVM zone with enable.dynamic.scale.vm=true, create a CKS cluster without choosing a node template, then change its service offering while it is running. The scale fails with Unable to scale VM instance ... as it does not have tools to support dynamic scaling.
Why it matters
The CKS documentation says "A running or stopped Kubernetes cluster can be scaled ... to a desired cluster size and service offering". On a default install, only stopped clusters can. To use the live feature, an operator has to find out on their own that the SystemVM template's flag must be turned on. They also have to turn it on for each existing node while the cluster is stopped, because a VM's flag can only be enabled when its template allows it: dynamic scaling cannot be enabled for instances the template does not have scaling enabled.
Cause
vm_template.dynamically_scalabledefaults to 0, andSystemVmTemplateRegistrationnever sets it, so the KVM SystemVM template is alwaysfalse. The column dates from 2013, when it meant "the template has XenServer/VMware tools". KVM needs no guest tools for hot-plug.- A user VM, including a CKS node, is scalable only when its template, its offering and
enable.dynamic.scale.vmall allow it. So the template flag blocks every CKS node.
What we tested (standard x86_64 KVM hosts)
| Situation | Result |
|---|---|
| Default install: live offering change of a running CKS cluster | Fails: "does not have tools to support dynamic scaling" |
| Template flag on, node flags on, same change | Works: all nodes resized live, no reboot, guests see the added CPUs |
| Template flag on, global setting on: newly created router, SSVM, console proxy | They become dynamically scalable too, and lose about 67 MB of usable memory each (a 64 MB kernel I/O buffer for the hot-plug memory range): 457 → 390 MiB on 512 MiB routers and SSVM, 960 → 894 MiB on the 1 GiB console proxy |
| Template flag on, global setting off | No change for system VMs or CKS nodes; nothing becomes scalable (checked in the code) |
| Existing VMs after the template flag is changed | Unchanged; each VM keeps the flag it was created with, so nothing breaks |
| VM whose vCPU maximum exceeds the machine type limit | Refused by libvirt on every host; CloudStack tries the other hosts and returns an error, no crash |
The memory cost applies only where an operator has turned on enable.dynamic.scale.vm, which is already required to use live scaling. Anyone who has made the SystemVM template scalable by hand, to get live CKS scaling, already has it today.
vCPU maximum limits seen on the hosts we checked
A dynamically scalable VM gets the host's CPU count as its vCPU maximum. These are the machine type limits libvirt reports:
| Host | QEMU / libvirt | pc (i440fx) |
Q35 |
|---|---|---|---|
| AlmaLinux 9.1 | 7.0.0 / 8.5.0 | 240 | 710 |
| Ubuntu 24.04.4 | 8.2.2 / 10.0.0 | 255 | 1024 |
| Ubuntu 26.04.1 | 10.2.1 / 12.0.0 | 255 | 4096 |
| Debian 13 | 11.0.2 / 11.3.0 | 255 | 4096 |
Only hosts with more threads than these limits would be affected, and kvm.cpu.dynamic.scaling.capacity caps the maximum per cluster. On Q35, more than 255 vCPUs also needs an IOMMU in extended interrupt mode (<ioapic driver='qemu'/> plus <iommu model='intel'><driver intremap='on' eim='on'/></iommu>).
Proposal
Make KVM SystemVM templates dynamically scalable by default, from the next release on:
- In
SystemVmTemplateRegistration, set the flag when a KVM SystemVM template is registered, for fresh installs and new template versions. - In the upgrade SQL, set it on all existing KVM SystemVM templates, since installs upgraded across several releases keep several of them:
UPDATE cloud.vm_template SET dynamically_scalable = 1 WHERE type = 'SYSTEM' AND hypervisor_type = 'KVM' AND removed IS NULL;
The SQL runs before the new template row is created, so both 1 and 2 are needed. - Mention in the release notes and the CKS docs that existing CKS nodes keep their old flag until they are updated or recreated.
Side note: system VMs ignore their offering's dynamic-scaling flag
Routers, SSVM and console proxy copy the template's flag and never look at their service offering's dynamic-scaling setting (NetworkHelperImpl, SecondaryStorageManagerImpl, ConsoleProxyManagerImpl). We set it to off on those system offerings, recreated the VMs, and they still came up scalable. User VMs check template, offering and the global setting together since #4643; system VMs were not updated then. @harikrishna-patnala, as the author of #4643 (and of the original system VM scaling change, CLOUDSTACK-3042), could you confirm whether leaving system VMs out was an omission, or deliberate for a reason we're missing? If it's an omission, it's a separate small bug, not part of this proposal.
Suggested testing for the fix
- Fresh install: the KVM SystemVM template is registered with
dynamically_scalable = 1. - Upgrade from 4.20 / 4.22 / 4.23: every existing KVM SystemVM template row is set to 1, and the new one too.
- A new CKS cluster on the default template scales live, with
enable.dynamic.scale.vmon. - Recreated routers, SSVM and console proxy start normally, with the global setting on and off.
- Existing VMs and CKS nodes are unchanged after the upgrade.
Activity