llmctl

Local LLM orchestration for Linux and macOS. llmctl probes your hardware, plans which models fit, downloads them with cryptographic verification, builds the inference engines from pinned source, and runs multiple models concurrently with strict memory budgets — as systemd user services (Linux) or launchd agents (macOS).

Engines (llama.cpp and colibri vendored as git submodules, pinned to stable tags; the onnx runner ships in-tree):

• llama.cpp tag b11379 (unchanged since 3.1.0; it was b10969 before 3.1.0) — GGUF models, OpenAI-compatible llama-server (CUDA / ROCm / Metal / CPU backends).
• colibri v1.11.0 — pure-C engines for very large MoE models memory-mapped from NVMe (no GPU required); OpenAI- and Anthropic-compatible API.
• onnx — a pure-python3 runner (lib/onnx_server.py, no build step, no submodule) for encoder-class decision models (NLI/sequence-classification ONNX exports such as the decide-nli profile); CPU-only inference via guarded optional pip deps (onnxruntime, numpy, sentencepiece, tokenizers). See docs/decision-models.md.

git clone --recursive <this-repository-url> llmctl
cd llmctl
./scripts/install.sh      # setup -> download 'small' -> install -> enable
                           # -> verify reboot/logout survival -> status

This builds the engines, downloads the default small profile, writes and enables its persistent service (systemd --user on Linux, launchd on macOS), confirms the host is actually configured to survive logout/reboot (and tells you exactly what to run if it isn't — it never guesses), and prints final status. Install a different profile, or several, with --profile:

./scripts/install.sh --profile fast --profile vision

See docs/scripts/install.md for every flag, idempotency guarantees on an already-partially-installed host, and troubleshooting.

git clone --recursive <this-repository-url> llmctl
cd llmctl
./bin/llmctl setup        # doctor -> build engines -> hardware plan
./bin/llmctl models download fast
./bin/llmctl start fast   # serves an OpenAI-compatible API on 127.0.0.1:8080

start runs the profile only for this session — it stops at logout/reboot. For a service that survives logout/reboot, follow up with ./bin/llmctl install (writes the persistent-service templates) then ./bin/llmctl enable fast (enable + start persistently) — or just use ./scripts/install.sh above, which chains all of this for you.

Forgot --recursive? git submodule update --init (llama.cpp and colibri). llmctl build llama|colibri|all initializes those two itself; llmctl setup and llmctl build decide|all also link containers or stop with the resolver's message. constitution, containers and superspec are not submodules of llmctl: they are resolved from the constitution hub (or LLMCTL_*_DIR) - run make deps, see docs/dependency-layout.md.

Release 3.4.1 is a fix release of the 3.4.0 host reserve: the start-time admission race is closed (admitted-but-not-yet-running units are counted), the admission store is host-wide, validated and per-uid, and duplicate or forged admission records can only add counted budget. Details: CHANGELOG.md, docs/capacity.md.

Release 3.4.0 keeps resources for the host when models run locally: by default max(2 CPUs, 25%) (whole cores) and max(4 GiB, 20%) of the RAM stay reserved, every model gets a thread cap, a CPU quota and a CPU affinity, a start is refused (exit 75) under CPU pressure or when the model does not fit next to the current workload, and llmctl capacity --json reports per-profile verdicts, the largest decision model that fits and a local|cloud|none fallback. Upgrade note: run llmctl install and re-validate (--threads changes the launch fingerprint). Details: CHANGELOG.md, docs/capacity.md.

Release 3.3.2 is a patch release: a killed llmctl validate / validate-after-start / scheduler run no longer leaves a stray flock -w 900 waiter running for up to 15 minutes (lock waits are now bounded 1-second slices). Details: CHANGELOG.md.

Release 3.3.1 is a patch release: the replication append handler stores a whole request in one WAL transaction (a 500-entry append was 500 fsyncs and timed out on failover), the registry watch no longer misses a same-size rewrite, and several tests no longer leave processes running. Details: CHANGELOG.md.

Release 3.3.0 makes the model validation gate enforced (strict by default; for systemd units this holds once llmctl install has regenerated them - a legacy, not re-installed unit still keeps an UNDETERMINED profile running, NOT validated), resolves constitution / containers / superspec from the constitution hub instead of vendoring them, and documents llmctl's governance role (below). Typed local decisions (noul / choice / score) behind a secured HTTPS gateway arrived in 3.1.0. What changed and what you must do when upgrading: CHANGELOG.md (migration table). What it does not do, and what was not verified live: docs/limitations.md. Building the decision binary needs Go >= 1.25; everything else still needs only bash, python3 and curl. Manual QA was waived by the operator on 2026-10-08 (the constitution gate is operator-waived, not satisfied); see docs/validation.md.

3.2.0 (host safety and model validation), status stated plainly. Services now run under per-profile memory caps with no swap, a bounded restart policy (a SIGKILL/oom-kill is not restarted automatically) and a memory-pressure start gate (start, restart, enable, switch can exit 75). A deterministic model validation gate was added (llmctl validate, vision validate, validate-all, require-validated). Real-model runs exist (vision twice, small once, one host; see below and docs/qa/small-live-2026-10-14/): the guarded live test of profile vision PASSED on the author's host on 2026-10-13 (21 pass / 0 fail / 0 undetermined, first of two runs, this host only; model sha256 882e8d2d...0863, mmproj sha256 8c0fb064...40cb; evidence docs/qa/vision-live-2026-10-13/). A second run on the same host (docs/qa/vision-live-2026-10-13-run2/, PASS 21/0/0) measured the unit MemoryPeak at 3005 MiB. Profile small has also run live once (2026-10-14, one host: PASS, suite chat-v1 6 pass / 0 fail / 0 undetermined, model sha256 6c99cc00...f74d, unit MemoryPeak 925 MiB; evidence docs/qa/small-live-2026-10-14/). Not shown by these three runs: the process VmHWM (UNKNOWN), variation across hosts, and every other profile (fast, vision-pro and all the rest have NOT run live; a fast attempt on 2026-10-14 ended UNDETERMINED because its port was held by another program, and no model ran). The gate is otherwise proven against a fixture server only, and the restart/cap/gate behaviour against hermetic tests, not a real systemd under real pressure. Since the 2026-10-14 fix round the gate is ENFORCED and strict by default (an unvalidated profile is started unpublished, validated, and served only on PASS; see docs/validation.md); those runs predate it and their records are no longer current - every profile must be validated again. Details and upgrade steps: CHANGELOG.md; boundaries: docs/limitations.md; gates and limits: docs/host-safety.md.

What the verification did and did not cover: docs/limitations.md.

Every profile has a computed footprint (see docs/architecture.md for the memory model). Reservations are tracked in $XDG_STATE_HOME/llmctl/run/.

• llmctl start fast small coder starts all three only if their combined RAM+VRAM reservations fit the live budgets (probed free memory minus 4 GiB RAM / 15% VRAM headroom, minus what already runs). Otherwise it refuses, names the alternative that does fit, and reminds you about switch.
• llmctl auto coder vision picks the best-ranked fitting profile per capability. If they don't fit alongside what is running, llmctl evicts non-enabled services in LRU order (oldest first) until they do. Enabled services are never evicted by auto.
• llmctl switch <p> is the always-works escape hatch: stop all, start one.

Ports are fixed per profile: fast 8080, coder 8081, vision 8082, vision-pro 8083, moe-fast 8084, small 8085, ws-dense-32b 8086, ws-moe-30b 8087, colibri-glm 8090, colibri-qwen36 8091, decide-tiny 8092, decide 8093, decide-pro 8094, decide-nli 8096, decide-2b 8098, decide-max 8097 and further decision profiles listed in docs/decision-models.md (the decide gateway, llmctl decide serve, listens on 8095). By default every profile binds to 0.0.0.0 (LAN-accessible) — see "Safety guarantees" below for the security trade-off and how to lock a profile (or the whole host) back to 127.0.0.1.

The planner classifies the host from the live probe (no hardcoded assumptions): below-minimum, baseline, workstation, datacenter. See docs/hardware-tiers.md. The baseline reference machine (Ryzen 7 2700X, 32 GB RAM, RTX 3060 12 GB, NVMe) runs fast, coder, vision, moe-fast, small — including co-resident combinations. Workstations (64-core Threadripper, 32 GB VRAM) unlock the ws-* profiles and full co-residency; colibri-glm (a 744B MoE that streams weights from ~380 GB of NVMe) is gated to datacenter. The catalog also carries decision profiles (capability decide) for typed noul/choice/score decisions, from decide-kev-08b and the other native profiles (decide-kev-08b, decide-kev-4b and decide-lev verified live; decide-julia ran live but clears no baseline (G-140); decide-kev-9b only on the unpinned nezha snapshot, not on a commit-pinned tree), decide-nli (encoder; use it for choice - its noul and score output does not beat the majority baselines), and the letter-logit profiles (per-profile status, anton CPU, 2026-10-09, with the thinking-mode fix: decide-2b 128/132 well-formed and good on noul/choice, decide-pro 112/132 with HTTP 502 deadline expiries on a slow CPU engine (G-156), decide not usable at its current mass_threshold (G-157), decide-max not exercised; the earlier nezha numbers 0/132, 0/132 and 68/132 are PRE-fix, from an unpinned snapshot; see docs/decision-models.md); decide-tiny is catalogued but not servable yet. decide-nli is a DeBERTa-v3-large zero-shot NLI encoder on the onnx engine (any tier). The catalogue runs up to workstation-tier profiles. The authoritative, current profile table (tiers, sizes, ports, source-labelled benchmarks) is docs/decision-models.md.

• Verified downloads: every file is checksummed against sha256 values captured from the Hugging Face API at catalog-generation time (or fetched live when a catalog entry is null). Mismatched content never lands at the final path — verification happens before the atomic rename. Evidence (command, exit code, output) is appended to ~/.local/state/llmctl/verify/<profile>.log.

• Smoke tests: GGUF profiles are booted in a real llama-server (ctx 512, CPU layers only) and must answer the deterministic prompt "Reply with exactly: OK" before the download is accepted. Decision GGUF profiles are then asked fixed typed questions through the production Go driver (llmctl decide smoke: the answer must be a valid typed choice with finite probabilities, and the invoice-routing probe must pick billing). onnx-engine decision profiles are booted in the real lib/onnx_server.py encoder runtime and must pass entail / contradict / batch probes against its /v1/score; when the optional inference deps are missing the smoke test SKIPs with a recorded reason and the download is NOT reported as fully verified.

• Budget refusals: the scheduler never overcommits; it refuses with exact numbers and a suggested alternative.

• OS-level protection: systemd units carry a per-profile MemoryHigh/MemoryMax (1.25x / 1.5x of the plan footprint, from the 05-memory.conf drop-in; your own later drop-in wins), MemorySwapMax=0, lower Nice/IO/CPU weight and a higher OOM score, a bounded stop (TimeoutStopSec=20) and a bounded restart policy (Restart=on-failure, 3 starts per 900 s) that does not restart after SIGKILL (oom-kill), SIGTERM or "port already in use" (exit 98). A permanently broken model fails visibly instead of crash-looping — llmctl status reports it as failed (crash-loop) with its last log line. launchd agents use KeepAlive + a widened ThrottleInterval=60 (launchd has no native give-up-after-N-restarts primitive, so this bounds the restart rate, not the total attempts; the Linux caps and the pressure gate's unit hook are systemd-only).

• Memory-pressure start gate: start, restart, enable and switch are refused with exit 75 when memory PSI or MemAvailable says the host cannot take the model; switch refuses before it stops anything. llmctl doctor FAILs on two enabled engine units for one profile. See docs/host-safety.md.

• Model validation gate: llmctl validate <profile> runs deterministic known-answer suites judged by plain code; UNDETERMINED (exit 2) is never a pass; the API token goes only to loopback hosts and no proxy is used. Real-model runs are recorded for vision (two runs, 2026-10-13) and small (one run, 2026-10-14), all PASS, one host; every other profile is not validated (docs/validation.md).

• LAN-accessible by default, with an explicit trade-off: every engine server binds to 0.0.0.0 by default, so any device on the local network can reach it (not just the host itself). This has no built-in authentication — llama-server's and colibri's OpenAI-compatible APIs accept unauthenticated requests from anyone who can reach the bound address, so on an untrusted or shared network segment (a coworking space, a guest Wi-Fi, a corporate LAN with unknown peers) this means anyone on that network can consume your GPU/model resources or read chat completions with zero auth. This is an intentional, explicit choice — llmctl is designed to be reachable from other devices you own on your own trusted LAN (a phone, a laptop, a second workstation) without extra setup. If your network is not fully trusted, scope reachability at your firewall/router (block the port from outside your LAN, or put the host on its own VLAN) or lock llmctl itself back to localhost-only:

  • LLMCTL_BIND_HOST=127.0.0.1 (env var, before start/enable) reverts every profile to localhost-only.
  • LLMCTL_BIND_HOST_<PROFILE>=127.0.0.1 (e.g. LLMCTL_BIND_HOST_FAST) reverts just that profile — the rest of the fleet stays LAN-accessible. Profile names are upper-cased with - → _ (ws-dense-32b → LLMCTL_BIND_HOST_WS_DENSE_32B).

  Both are read at start/enable/switch/auto time (lib/catalog.sh's catalog_bind_host()), so they take effect the next time a profile is (re)started — restart an already-running profile after setting either variable for it to apply. lib/download.sh's one-shot model-verification smoke test is unaffected either way: it always uses a throwaway, localhost-only port during the download step, never reachable from the LAN.

  colibri-glm/colibri-qwen36 need one extra step for the LAN-accessible default to actually work. The colibri engine has its own, independent fail-closed bind guard: it refuses to start on any non-loopback host unless it is given an API key or COLI_ALLOW_INSECURE_BIND=1 is set in its environment — confirmed live: with neither set, coli serve --host 0.0.0.0 ... prints refusing to bind 0.0.0.0 beyond localhost without COLI_API_KEY set (set COLI_ALLOW_INSECURE_BIND=1 to override) and exits immediately, which under enable/install shows up as a crash-loop. llmctl does not set this for you automatically — silently overriding a component's own explicit security control is a bigger, separate decision from choosing llmctl's own bind-host default, and it is yours to make, not llmctl's. To actually get a LAN-accessible colibri profile, add Environment=COLI_ALLOW_INSECURE_BIND=1 to that unit (systemctl --user edit llmctl-colibri@colibri-qwen36.service on Linux) or export it before a manual coli serve invocation — or just leave that one profile on LLMCTL_BIND_HOST_<PROFILE>=127.0.0.1, which needs no such change.

Copy .env.example to .env, fill in real values, and lock it down:

cp .env.example .env
chmod 600 .env

.env is gitignored and never printed or logged (Constitution §11.4.10). Single-host usage needs no credentials by default; HF_TOKEN is only required for gated/private Hugging Face model repos, and the LLMCTLD_* variables are only read when the opt-in cluster daemon (llmctld, see docs/cluster-architecture.md) is enabled.

make test       # deterministic test harness (fixtures, dry-run, local HTTP)
make lint       # shellcheck (skipped with a message when not installed)
make validate   # json-check + lint + test
make archive    # ../llmctl.tar.gz + ../llmctl.zip

llmctl is a hub engine of the Helix constitution: it lives at constitution/submodules/llmctl, is consumed by reference (never copied into a product, initialised one level only, recursive_init: false), and declares itself in a tracked helix-deps.yaml (Constitution §11.4.31). It must not nest the constitution or any other own-org repository: constitution, containers and superspec are found by one resolver (lib/resolve_deps.sh) instead of nested copies (Constitution §11.4.28(C); gate CM-HUB-ENGINE-NO-OWNORG-NESTING). Layout and the resolver order: docs/dependency-layout.md.

claude_toolkit (JEV) integration. claude_toolkit (also a hub engine) owns the consumer side of the decision layer: it detects the local decision gateway of llmctl, registers decision endpoints with classification: "decision" (they never become aliases), and exposes a permission gate, an MCP tool and launch-time environment to its agents (jev-setup.sh, jev-gate.sh, jev_mcp.py). llmctl supplies the endpoint and the validation gate; the toolkit's own guides are in docs/jev/ of that repository. The diagrams are indexed in docs/diagrams/README.md.

Every doc in this project is reachable from this table (Constitution §11.4.212 — no orphan docs; scripts/check_doc_reachability.sh and tests/test_doc_reachability.sh enforce it). Start with the Quickstart above for the fastest path to a running model.

MIT (see LICENSE).

Dependency manifest (§11.4.31): see helix-deps.yaml — own-org dependencies are declared there and resolved at run time by lib/resolve_deps.sh; they are never vendored or nested as submodules.