This directory is the complete deploy unit for running the public ECHIDNA API on a single Hetzner server behind Caddy (auto-TLS via Let’s Encrypt), orchestrated with podman systemd quadlets.

• Quadlets over podman-compose: systemd-native restart-on-boot with no extra daemon, unit-level dependency ordering, and first-class AutoUpdate=registry integration with podman-auto-update.timer.

• Rootless: the API executes user-supplied prover input; run both units as an unprivileged deploy user with linger.

• Caddy over svalinn: the estate’s svalinn/vordr stack is not yet CI-proven; Caddy is the low-risk choice for a public endpoint. The container/compose.toml selur stack remains for local development.

• No API auth at launch: the service is deliberately public but caged (rate limit, body cap, memory/pids limits, tmpfs, unpublished port). If abuse appears, add a bearer-token matcher in the Caddyfile for POST routes. A server-side per-request prover timeout is tracked as a follow-up issue in the main repo. Superseded 2026-11-05: see Containment stopgap (owner ruling 2026-11-05).

Until bearer authentication lands on every execution route, this host serves read-only routes only, and it no longer rolls out new images by itself.

• The Caddyfile forwards only GET/HEAD on /api/health and /api/provers. Every other request is answered 403 by Caddy and never reaches the container. That includes /api/prove, /api/verify*, /api/session/, /api/agent/ and /api/search.

• echidna.container drops AutoUpdate=registry and sets Pull=never. The unit runs the image already on the host, so a newly published :latest is not deployed. This freezes the running image. It does not verify it.

Apply it on the host as the deploy user with stopgap-apply.sh, from a copy of this directory:

The script fails closed. It stops before restarting anything when a step cannot be confirmed, and it ends by printing stopgap: PASS. In order, it:

1. Checks that echidna.container is the stopgap version. It reads the directives as systemd does: indentation and spaces around = are ignored, and so are # and ; comment lines. The [Container] section must set exactly one Image=ghcr.io/hyperpolymath/echidna:latest and exactly one Pull=never. No AutoUpdate= may appear in any section, and no io.containers.autoupdate label either.

2. Checks that the Caddyfile has the stopgap shape. Comments are ignored. There must be exactly one @public matcher, holding exactly method GET HEAD and path /api/health /api/provers, and no other named matcher. There must be exactly one reverse_proxy, and it must be the only thing inside handle @public. Exactly one catch-all handle, next to it, must hold only respond …​ 403. No other handle is allowed, and neither is import, invoke, handle_path, rewrite, uri, php_fastcgi or forward_auth. method is allowed only inside @public. This is a check of the text, not of caddy adapt output: the proxy image carries the rate_limit plugin, which a stock caddy does not know, and the host has no caddy outside that image.

3. Detects the install layout, and refuses unless exactly one is present. A quadlet host (§2) runs echidna.service and caddy.service; an appendix host runs container-echidna and container-caddy.

4. Disables podman-auto-update.timer. That does not end a run the timer has already started, so it also stops podman-auto-update.service. It then refuses unless systemctl --user is-active reports that service inactive or failed.

5. Records the image ID the echidna container is running, in ~/.local/state/echidna/stopgap-running-image. If the local :latest tag has moved since the container started, it points the tag back at the running image, because a restart under Pull=never starts whatever :latest resolves to.

6. Installs the Caddyfile into /opt/echidna/. On a quadlet host it also installs echidna.container, reloads systemd and restarts echidna and caddy. On an appendix host it restarts only container-caddy: those containers carry no auto-update label, and podman generate systemd --new units do not pull an image that is already present.

7. Verifies that the container still runs the recorded image, that the timer is disabled and no auto-update run is active, and that GET /api/health answers 200. It then sends each of these and requires 403 from every one: POST to /api/prove, /api/verify, /api/verify_raw, /api/verify_parallel, /api/suggest, /api/session/create and /api/agent/plan, GET /api/search, POST /api/health, and GET on a route the server does not have. Any other answer, a 429 from the rate limiter included, fails the run. One run sends at least 11 requests, and more while it waits for /api/health. The limit is 30 a minute for each client, so wait a minute before running verify again after a failure.

To check the host again later, from any shell:

Do not proceed if it prints a FAIL line. The script’s behaviour is tested with stub commands in tests/deploy/stopgap-apply.test.sh. Neither the script nor its test has been run against the live host.

The stopgap ends in the change that enables authentication. That change re-pins Image= to the signed image by digest and removes the @public matcher. Do not re-enable podman-auto-update.timer before then.

Gates — stop and reassess if any fail:

• Ports 80/443 unbound (ss -tlnp shows no listener). If something else owns them, this plan does not apply as-is.

• At least 2 GB RAM. The API container is capped at 1500 MB (PodmanArgs=--memory=1500m), deliberately below the host total so Caddy and the OS keep headroom; raise both together if you want more. The :full image (Isabelle/Agda/Julia) needs >= 4 GB and is not deployed here.

• Quadlet keys are all understood by the installed podman. Do not trust a version number for this — measure it. Copy the three unit files to the server and run the generator in dry-run mode:

  scp echidna.network echidna.container caddy.container <user>@<SERVER_IP>:/tmp/qcheck/
  ssh <user>@<SERVER_IP> 'QUADLET_UNIT_DIRS=/tmp/qcheck /usr/libexec/podman/quadlet -dryrun'

  It must print generated [Service] sections for all three units with no warnings. These units were authored and dry-run verified against podman 5.4; podman 5.x is recommended and 4.x is likely to reject at least one key (HealthOnFailure= in particular). If the dry-run complains, use the Debian-12 fallback in the appendix instead.

• Image pullable anonymously:

  podman manifest inspect ghcr.io/hyperpolymath/echidna:latest

  On 401: make the GHCR package public (GitHub → Packages → echidna → Package settings → visibility) or use a read-only PAT with podman login ghcr.io as the deploy user.

Create the quadlet directory before copying anything into it — as the deploy user on the server:

Then, from your workstation in this directory:

Back on the server, as deploy:

Create in the Cloudflare zone for nesy-prover.dev:

Grey cloud is required at launch so Caddy’s Let’s Encrypt HTTP-01 challenge reaches the server directly. (Optional later hardening: orange-cloud + a Cloudflare Origin CA cert in the Caddyfile.)

Wait until dig +short api.nesy-prover.dev @1.1.1.1 returns the server IP before the first Caddy start — ACME then succeeds first try.

End-to-end, from anywhere:

Controls:

Reboot survival (validates linger + [Install] + Restart in one shot):

ghcr-publish.yml pushes a new :latest on every GitHub release. Enable unattended rollout:

How the safety net actually works: podman auto-update pulls the new image and restarts the unit. HealthOnFailure=kill in echidna.container means a container that goes unhealthy is killed, which fails the systemd unit — and auto-update rolls the unit back to the image it was running before when the restarted unit does not come up healthy. Without that key an unhealthy-but-running container would satisfy systemd and no rollback would occur, so do not remove it.

Manual rollback to a recorded digest. Stop the timer first — otherwise its next run pulls registry :latest again and silently undoes the rollback:

Re-enabling the timer does nothing on its own: podman auto-update acts only on containers labelled by AutoUpdate=registry, and the stopgap unit removes that key. Restoring unattended rollout therefore means putting AutoUpdate=registry back in echidna.container (and dropping Pull=never), reloading systemd, and only then enabling the timer. That ends the containment stopgap, so it belongs to the change that lands authentication, not to an operator step. The intended end state is pinned mode: Image= set to the signed image’s digest, no AutoUpdate, and updates by PR.