Skip to content

service: healthy background service is killed after host sleep because probe timeouts count as unresponsive #58111

Open

Description

@AlexeyShirshov

Description

After the host resumes from sleep, the client replaces a healthy background service: it sends SIGTERM (then SIGKILL) to the running opencode serve --service, clears the PTY handoff, and spawns a replacement. In-flight work is interrupted and persistent terminals are lost even though the old process was alive and answering.

Root cause (both clients, @opencode/client Effect and Promise implementations):

  • service-probe.ts probes /api/info with the global fetch and a 2 s AbortSignal.timeout, so the probe can sit on a stale keep-alive socket from before the suspend and never reach the server.
  • effect/service.ts:74-88 (and promise/service.ts:56-70) counts three consecutive timeouts as "unresponsive" and runs destructive recovery: PtyHandoff.clear() → terminate() (SIGTERM, then SIGKILL) → recruitNow().
  • A resume also trips the event-stream watchdog (solid/connection.ts:91, idleTimeout 45 s), which calls the reconnect path (cli/.../default.ts:107) → Service.ensure(), so the probe loop runs exactly when the machine comes back from suspend.

There is no check that the process is still alive (kill(pid, 0)), no hysteresis beyond 3 × 2 s, and no awareness that the probe was in flight across a suspend.

Observed timeline (local time, log at ~/.local/share/opencode/log/opencode.log):

  • 16:50:28 MCP SSE transports drop in the server; 16:50:32 the TUI logs event stream disconnected → reconnect → Service.ensure()
  • 16:50:32 → 16:51:25 nothing is logged: host suspended. systemd-resolved logs Clock change detected. Flushing caches. at 16:51:25 on resume
  • three 2 s /api/info probes time out back-to-back (no /api/info request from them ever reaches the server, i.e. they were sent on dead pooled sockets)
  • 16:51:31.8 the old server starts its teardown (watcher stopped, failed to terminate MCP session) — it was alive the whole time
  • ~16:51:36.8 SIGKILL (graceful shutdown does not fit the 5 s wait), 16:51:37.5 four serve --service contenders spawn, 16:51:38 the TUI reconnects to the new service

Plugins

opencode-roslyn, wsl-notify, superpowers, plus local btw-context.js and zen-failover

OpenCode version

2.1.0 (latest channel, local=false)

Steps to reproduce

  1. Run the TUI on a laptop with the managed service (opencode serve --service) and keep the TUI connected for a while.
  2. Suspend the host (lid close / sleep) for at least a minute, so a client request is in flight or the event stream goes idle.
  3. Resume and watch ~/.local/share/opencode/log/opencode.log (or opencode service status): the registered PID changes, and the previous server logs a graceful shutdown.

Seen once with this exact signature (2026-10-22 16:51 local, WSL2 on Windows 11); the same service replacement also appears at 2026-10-22 06:07 and several times on 2026-10-21 in the same log.

Screenshot

No response

Operating System

Windows 11 26300.9457 with WSL2, Linux 6.18.33.2-microsoft-standard-WSL2

Terminal

Windows Terminal via WSL2, TERM=xterm-256color (no TERM_PROGRAM)

Additional context

What the user sees: the TUI briefly reconnects, the running turn is gone, persistent terminal sessions are lost, and in some reports (see #58000, #57864) the tab loses its directory/session after sleep.

Related but distinct: #57038 covers the same destructive recovery branch when the service really is hung (unbounded respawn storm). This report is about a false positive: a slow/unreachable probe right after resume kills a healthy service. The two share a fix.

Suggested direction (happy to send a PR):

  1. Probe /api/info over a fresh connection (no keep-alive reuse) and re-probe once before counting a timeout.
  2. Do not count probes whose deadline spans a wall-clock jump; reset the timeout counter and the contender pool on resume.
  3. Before destructive recovery, check kill(pid, 0): if the process is alive, keep waiting (15–30 s) instead of 3 × 2 s.
  4. Use pty: "handoff", not PtyHandoff.clear(), whenever a restart is unavoidable.
  5. Keep the policy in the shared probe/ensure code so the Effect and Promise clients (TUI and Desktop) cannot drift.

Activity

  1. opencode-agent commented on Oct 22, 2026

    @opencode-agent
    Contributor

    Thanks for the detailed timeline and code pointers. I couldn't reproduce the false positive here, because it needs a real Windows host sleep with OpenCode running inside WSL2, which this sandbox can't do.

    What I tried on 2.1.0 with a real opencode serve --service:

    So on Linux loopback I couldn't trigger the stale keep-alive socket theory. Whether it happens likely depends on what WSL2 does on resume (VM pause, clock jump, virtual network re-initialisation), or on the service being briefly busy after resume (your log shows MCP connections dropping just before), so it answers /api/info too slowly. This needs a maintainer with a WSL2 laptop to check.

    If you can, these would help narrow it down:

    • Do you see any /api/info request in the server log between resume and the SIGTERM? That would show whether the probes reached the server at all.
    • Does it still happen without the MCP servers or plugins that reconnect on resume?

    Related issues:

  2. AlexeyShirshov commented on Oct 22, 2026

    @AlexeyShirshov
    Author

    Confirmed on 2.1.2 (latest, local=false), and it no longer needs a real suspend to reproduce.

    The shipped bundle still contains the same recovery policy — requestTimeout: 2000, if (count >= 3) → PtyHandoff.clear() → SIGTERM → SIGKILL → recruitNow(), and the probe is still a plain fetch("/api/info") with no control over connection reuse.

    Deterministic repro against the installed 2.1.2 binary, in an isolated XDG_* sandbox on port 49400 (a second serve --service with its own state dir, so the real service is untouched). Instead of suspending the host, the test process is frozen with SIGSTOP: it is alive, accepts TCP, but answers nothing — the same shape as a service that just came back from suspend.

    registered pid=178973
    froze pid=178973 at 17:34:17
    WARN ... message="Background service is unresponsive; recovery cannot preserve persistent terminals"
    RESULT: frozen pid was KILLED
    api exit=0 after 13.6s   -> registration now points at a new pid on the same port
    

    Timing matches the live incident exactly:

    • 6 s after the freeze → 3 × 2 s probes counted, "unresponsive" verdict
    • +5 s → SIGTERM cannot be handled by the frozen process, so SIGKILL, then a replacement serve --service
    • the calling client (opencode api get /api/info) exits 0 and returns the new pid, so nothing surfaces to the user — only the running turn and persistent terminals are lost

    Ran it three times: 11.8 s / 11.1 s / 13.6 s, the frozen process was killed every time. The live occurrence (16:51 local, WSL2) had the same 6 s to verdict and the same SIGKILL-then-respawn sequence.

    Also still true: the recovery branch calls PtyHandoff.clear(), so a restart triggered this way can never preserve persistent terminals, even when the old server is merely slow rather than dead.

  3. cevr commented on Nov 12, 2026

    @cevr

    Upstream text (not posted yet)

    Upstream still has this: Service.ensure evicts the registered service after three timed-out probes. It came in with
    PR #40896 ("fix(client): evict unresponsive service", merged 2026-08-06), and upstream/dev was unchanged on
    2026-11-12 (git diff HEAD upstream/dev -- packages/client/src/effect/service.ts shows only our fork's onWait
    lines). There is no newer upstream code to take.

    Issue #58111 ("healthy background service is killed after host sleep because probe timeouts count as unresponsive",
    open) reports the same root cause, through host sleep. Post this as a comment on #58111, not as a new
    issue. Also link it from #54810 (slow startup turns into a spawn/retry storm), which overlaps.


    The same eviction also fires without sleep: a loaded host is enough. On a Linux box at load 20–60, a service that is
    booting or busy misses three 2 s /api/info probes. Any opencode api … call then runs Service.ensure. It
    SIGTERMs the live service, then spawns a replacement, and the replacement is slow too. So the next CLI call kills
    that one as well.

    From one host's opencode.log on 2026-11-12, 00:00–19:03 UTC:

    • 19 server runs started.
    • 6 of them came 3–6 s after a Background service is unresponsive; recovery cannot preserve persistent terminals
      warning that a client logged:
      • five from opencode api POST /api/rpc/… calls made by health and status scripts;
      • one from a TUI opening a session (opencode -s <id>).
    • In the 18:34–18:41 window the service restarted six times in seven minutes.

    What I'd expect. A client never stops a service it did not start just because probes are slow. A slow probe
    does not prove the service is hung: it can be booting, under load, or behind a stale socket after sleep (#58111). The
    client could say that it is waiting (onWait), keep waiting up to its existing deadline, then fail with an error
    that says the service was left running and names opencode service restart. A user who wants a hung service gone
    can still stop or restart it explicitly; that path is Service.stop, which this change leaves as it is. A contender
    that this same ensure call spawned and that then stops answering can still be evicted as today.

    What we run in our fork.

    • packages/client/src/effect/service.ts and promise/service.ts: when a probe times out and the registered pid is
      not a live contender of this call (contenderPool().owns(pid)), ensure reports it, keeps polling until
      promiseTimeout, and recruits no contender against it.
    • If the service never answers, ensure fails with: "Background service at (pid N) did not answer within 120s.
      It was left running; if it is stuck, restart it with opencode service restart."
    • Tests in test/service.test.ts and test/promise-service.test.ts:
      • a service whose first five probes answer after the timeout survives, and is reused;
      • a hanging service is reported and left running, and no contender is spawned.
      • These replace "evicts an unresponsive registered service before starting its replacement".

    Fork commits: f0c6c66 (tests), 05cdc54 (fix) on cevr/opencode branch api-probe-kill. Happy to open a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions