net: BenchmarkTCP{4,6}ConcurrentReadWrite failures on freebsd-amd64-race with "read: connection reset by peer"

[bcmills]

2021-03-23T05:12:12-cd26192/freebsd-amd64-race
2021-03-23T04:23:52-0265b64/freebsd-amd64-race

(See also #20960.)

CC @bradfitz @ianlancetaylor

These failures are both very recent, so I wonder if something has recently changed in the net package or the freebsd-amd64-race builders. (CC @golang/release)

[bcmills]

#19229 also features connection reset by fail errors on FreeBSD. I wonder if it's related...

[ayang64]

which version of freebsd? i can't reproduce on FreeBSD kiwi 14.0-CURRENT FreeBSD 14.0-CURRENT #5 main-n245559-d22883d71544: Sun Mar 21 16:51:44 EDT 2021 root@kiwi:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64

[ayang64]

also, are you running freebsd in a virtual machine for tests?

[bcmills]

Looks like it's FreeBSD 12.0 of some flavor, on a GCE n1-highcpu-16 VM:
https://github.com/golang/build/blob/d7600ec889c89d486672e9ccd1b30bab50c15047/dashboard/builders.go#L317-L324

@golang/release could probably get you more details.

[bcmills]

The failure rate is not particularly high, so you may also need a lot of runs to reproduce the failure.

[ayang64]

okay -- i just ran it on my local machine with -count 100 without any failures. i'll continue trying to reproduce and maybe try to isolate the commit that caused it. i'll also try under 12.0.

[bcmills]

There's one more in the logs today:
2021-03-25T00:15:42-80157b5/freebsd-amd64-race

That's three in two days, and none that I can find before that. So the regression is probably very recent, whatever it is.

[bcmills]

2021-04-15T10:21:10-b1c4cc5/freebsd-amd64-race
2021-04-13T22:55:30-8c2a8b1/freebsd-amd64-race
2021-04-13T13:56:21-13a4e8c/freebsd-amd64-race
2021-04-09T23:54:31-554d2c4/freebsd-amd64-race
2021-04-08T15:03:31-283b020/freebsd-amd64-race
2021-04-08T02:17:22-5159c83/freebsd-amd64-race
2021-04-07T05:19:22-1395432/freebsd-amd64-race
2021-04-06T22:55:58-b084073/freebsd-amd64-race
2021-04-03T01:45:13-93dcaba/freebsd-amd64-race
2021-03-31T20:09:03-44dd066/freebsd-amd64-race
2021-03-30T04:48:27-032ef4b/freebsd-amd64-race
2021-03-29T19:16:20-9fbd0f6/freebsd-amd64-race
2021-03-29T18:46:32-67d565d/freebsd-amd64-race
2021-03-25T19:55:42-7ce3617/freebsd-amd64-race
2021-03-25T19:21:34-2c8692d/freebsd-amd64-race
2021-03-25T00:15:42-80157b5/freebsd-amd64-race
2021-03-23T05:12:12-cd26192/freebsd-amd64-race
2021-03-23T04:23:52-0265b64/freebsd-amd64-race

And nothing before that.

Marking as release-blocker for Go 1.17, since this appears to be a regression starting in March.

[ayang64]

i still can't reproduce using 12.2-RELEASE-p6.

it looks like a likely culprit is commit 6694a6888 though.

[ayang64]

maybe grasping at straws but what if you update the test image to the latest release in the freebsd 12 line?

it seems like that'd be a good start. it looks like you're running 12.0 but the latest production release is 12.2.

we have the following errata that might cause this issue for which the fix is to update the kernel version:

https://www.freebsd.org/security/advisories/FreeBSD-EN-19:02.tcp.asc

i iust ran 100 iterations of the benchmark on a 16 core GCP instance running FreeBSD 12.2 without errors.

it seems reasonable to try the latest stable release in the 12 line instead of 12.0.

[toothrot]

We're working on updating the builders soon in #44431.
/cc @cagedmantis

[ayang64]

looks like the default builder has been updated to 12.2. is there a way to see the results?

[dmitshur]

The new builder will be used in future testing (existing commit with test results are not automatically re-tested at this time), so its results will show up when new commits are pushed, or by explicitly sending a (possibly empty) change and requesting it via TRY=freebsd-amd64-race SlowBot syntax.

[ayang64]

has this error returned? it looks like the freebsd builders have been relatively happy:

https://build.golang.org/

[toothrot]

There doesn't seem to be any recent occurrences. Closing this issue.