G
YouTube TikTok Games
Skip to content

Retry and the next start try an unfinished backend update again when it left OpenBot running but not working - #1503

Open
guidovizoso wants to merge 1 commit into
mainfrom
guido/retry-update-on-degraded-stack
Open

guidovizoso wants to merge 1 commit into
mainfrom
guido/retry-update-on-degraded-stack

Conversation

@guidovizoso

Copy link
Copy Markdown
Collaborator

What was wrong

A dogfooding report: OpenBot Desktop 1.0.1 updated to 1.0.2 on a Mac. On first start, the backend update (Intelligence 0.16.0-rc.10 to rc.11) paused the services (update-quiesce) and then failed once at dependencies-install. That left the stack running with its checks failing (readiness degraded) and the update on record (unfinishedUpdate).

The window then said OpenBot is stopped, with "OpenBot tries again the next time it starts" on the update row, but every Retry asked for local status once and gave up in 0.0s. The update was never tried again, so the person was stuck.

The wait loop in desktop/src/LocalIntelligence.tsx re-ran an unfinished update only on a stack that was ready, stopped, or still on the release it had with every check passing (kept). A degraded stack with failing checks matched none of these. start is skipped while an update is on record (the CLI refuses it), and degraded is not starting, so the wait stopped at once.

The fix

  • A stack that is stopped or degraded with an update on record now runs local update again. That is the only call that gets it going, since the CLI refuses start. The old kept case is one of these degraded stacks, so the condition now covers it without listing it separately.
  • This still runs at most once per wait (updatedIntelligence). If the update fails again, the wait stops at once on the failed row ("It stopped while updating where OpenBot keeps your data"). It does not wait the way it does after waking a stopped stack, because nothing on a degraded stack comes back by itself. Each Retry is one more attempt.
  • The comments ("AN UPDATE THAT DID NOT FINISH" and woken) now explain this.
  • No user-facing text changed.

The repair before an update (local_update.rs)

The repair rolls back only a release that Helm left pending-*, and puts back Learning Deployments that are missing. A release that a failed dependencies-install left failed is not touched. Helm 3 upgrades a failed release from its last deployed revision, so it needs no repair. A new fake-CLI case pins this down: the repair does not roll back or re-apply anything, and the next update runs to the end.

Tests

  • LocalIntelligence.test.tsx, "a backend an update left not working runs the update again, and opens once it finishes": status degraded with dependencies-install on record and updateTo set, then local update completes, status is ready and OpenBot starts. Before the fix it failed (no update, nothing started). After the fix it passes.
  • LocalIntelligence.test.tsx, "an update that stops again is tried once per Start, and Retry tries it once more": the first update fails, the wait stops after status, update, status with the failed row and Retry, and there is no second update. Retry runs status, update, status again, that update completes, and OpenBot starts. Before the fix it failed (no update, so no update row). After the fix it passes.
  • App.test.tsx, "a backend whose check fails is still not healthy, with an update on record or not": this test now expects the update to be tried once (status, update, status). Its point is unchanged: a failing check is still not treated as running on the release the stack had, nothing is started and Retry is offered.
  • local_eval.rs, a_release_an_update_left_failed_is_not_repaired_and_the_next_update_finishes: this uses the fake CLI and Helm. The first helm upgrade -f stopped.yaml fails and leaves the release failed. The repair checks the release and does nothing, the update is not retried in the same call, and the next local update completes with no rollback.

Run locally: bun test desktop/src (407 pass, 0 fail), bun run typecheck in desktop, bun run typecheck:tests, Biome format and lint, cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test --locked --lib --bins (1087 + 304 pass).

Not reproduced in the real product

I did not reproduce this in a real Desktop build. The CLI keeps every local stack in one shared config folder, and the machine I worked on also runs an installed OpenBot and other people's stacks. To reproduce, I would have had to make a real rc.10 to rc.11 helm upgrade fail on a live stack. Verification is the tests above, including the fake-CLI test of the native update and repair path.

…it left OpenBot running but not working, instead of giving up at once

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant