testing/synctest: bubble not terminating

[alecthomas]

Go version

go version go1.26-devel_a4d9977 Wed Jul 30 14:00:16 2025 -0700 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/alec/dev/zero/.hermit/go/bin'
GOCACHE='/Users/alec/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/alec/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT='synctest'
GOFIPS140='off'
GOFLAGS='--tags=mysql,postgres,sqlite'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/xk/0xh6fbnx3wx2_rkxr4w9vx140000gn/T/go-build3867150125=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/alec/dev/zero/go.mod'
GOMODCACHE='/Users/alec/go/pkg/mod'
GONOPROXY='*.sqcorp.co,github.com/squareup'
GONOSUMDB='*.sqcorp.co,github.com/squareup'
GOOS='darwin'
GOPATH='/Users/alec/go'
GOPRIVATE='*.sqcorp.co,github.com/squareup'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/alec/Library/Caches/hermit/pkg/go@tip'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/Users/alec/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/Users/alec/Library/Caches/hermit/pkg/go@tip/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.26-devel_a4d9977 Wed Jul 30 14:00:16 2025 -0700'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I've written a test using testing/synctest that I believe should exit, but doesn't. The code to reproduce this is in this draft branch, but the stack dump is also attached.

Cloning that branch down and running GOEXPERIMENT=synctest go test -v ./providers/cron should reproduce the issue.

This occurs with latest stable Go and Go tip from a4d9977.

stack.txt

What did you see happen?

I believe it should exit because when I SIGQUIT the process, the only goroutines running are from the stdlib. It's also entirely possible I'm doing something spectacularly stupid, but if I am I can't figure it out.

As an aside, I think it would be very useful if SIGQUITting inside a bubble marked the goroutines preventing the bubble from terminating.

What did you expect to see?

I expect to either have synctest.Run() terminate, or for a goroutine to be in the stack trace that is obviously causing the issue.

[gabyhelp]

Related Issues

• testing: Example test with runtime.Goexit hangs #41084 (closed)
• testing/synctest: bug with httptest #71345 (closed)
• testing/synctest: Repeated sync.WaitGroup.Add appears flaky under synctest #74386 (closed)
• testing: stack unavailable when a parallel test times out #18042 (closed)
• testing: inconsistent behaviors between running tests directly and after compiling the code first #59879 (closed)
• cmd/go: test gets hung #27720 (closed)
• cmd/go: go test sometimes hangs if interrupted #60220 (closed)
• cmd/go: trim build cache incrementally #38939
• testing: "panic: Log in goroutine after Test..." is unreliable due to lack of synchronization on t.done #67701

Related Code Changes

• runtime: print stack traces for bubbled goroutines on synctest deadlock

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mrkfrmn]

CC @golang/runtime, @neild from owners.

[neild]

I'm on vacation for the next week, so I probably won't have a chance to look at this in detail until I'm back.

The stack traces don't show any goroutines in a synctest bubble, so I believe the bubble has exited at the time the SIGQUIT happened. I'm not sure what's going on from a cursory look, but goroutine 10 in the attached stack traces appears to be stuck trying to lock a context.Context mutex and there aren't any other goroutines that I see that would be holding that lock. That's pretty suspicious.

As an aside, I think it would be very useful if SIGQUITting inside a bubble marked the goroutines preventing the bubble from terminating.

On Go tip, the stack traces should include which bubble (if any) each goroutine is part of, and bubbled goroutines are marked with an indication of whether they're durably blocking or not.

[alecthomas]

On Go tip, the stack traces should include which bubble (if any) each goroutine is part of, and bubbled goroutines are marked with an indication of whether they're durably blocking or not.

Aha! That's fantastic news, thanks.

[alecthomas]

The stack traces don't show any goroutines in a synctest bubble, so I believe the bubble has exited at the time the SIGQUIT happened. I'm not sure what's going on from a cursory look, but goroutine 10 in the attached stack traces appears to be stuck trying to lock a context.Context mutex and there aren't any other goroutines that I see that would be holding that lock. That's pretty suspicious.

Based on this observation, I added a log line after the bubble in the main test and saw the output. So it looks like the bubble is terminating, but the test itself is not.