proposal: runtime/metrics: add mutex slow path counters

[bradfitz]

We modified the runtime & sync package a bit to add counters when a sync.Mutex.Lock and sync.RWMutex.RLock call go into their slow paths:

tailscale@effe2d1 (corp CLA on file; same license as Go)

That's a bit of hack (adding some global variables to the sync package) but it gets us information easily.

Would the runtime team be interested in adding this sort of information more officially via the runtime/metrics package?

We know about the "block" and "mutex" pprof types, but they're a bit more tedious to use to just get a counter.

Our use case is: we have a server that with some very hot mutexes and we want to high level metric showing how happy it is that'll become unhappy looking if people add too much code in the critical section.

[prattmic]

cc @golang/runtime @mknyszek

[mknyszek]

I'm certainly not opposed!

If you have a concrete implementation in mind and want to send a patch, I'm happy to review the changes. Otherwise I'll try to get around to it this cycle, but no promises. I was planning on adding some GC CPU counters this dev cycle, along with maybe a few more scheduling metrics.

[mknyszek]

Also, I think there might actually be another issue about this? #49881 I also believe there was an internal feature request for this, too.

Seems like a good idea. :)

[prattmic]

Just to be concrete, if I understand correctly, a mutex profile would work here, but you are looking for a general "contention level/rate" counter that can be easily monitored via a monitoring system, which could alert you that you may want to look at an actual profile.

I also think that sounds reasonable and useful.

[rsc]

Closing as duplicate of #49881, which seems to be slated for Go 1.20.

[seankhliao]

Duplicate of #49881