runtime: gc sweepone bad span

[chwetion]

What version of Go are you using (go version)?

Sorry, I can't find go version, just download 1.2.12 containerd in github, maybe 1.12.17

Does this issue reproduce with the latest release?

It has not been found for the time being, and the replacement of containerd is not allowed on env.

What operating system and processor architecture are you using (go env)?

linux arm64 and 4.14 kernel

What did you do?

Do nothing, just run containerd

What did you expect to see?

Keep containerd running

What did you see instead?

...
Jan 16 02:30:51 hua-dlzx1-k1011-gyt crond: postdrop: warning: unable to look up public/pickup: No such file or directory
Jan 16 02:40:01 hua-dlzx1-k1011-gyt systemd-logind: Failed to start user slice user-0.slice, ignoring: The maximum number of pending replies per connection has been reached (org.freedesktop.DBus.Error.LimitsExceeded)
Jan 16 02:40:01 hua-dlzx1-k1011-gyt systemd-logind: Failed to start session scope session-87079.scope: The maximum number of pending replies per connection has been reached
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: runtime: bad span s.state=3 s.sweepgen=21780159 sweepgen=21780160
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: fatal error: non in-use span in unswept list
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 3 [running]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: runtime.throw(0xaaaad0fd5dcb, 0x1f)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/runtime/panic.go:617 +0x54 fp=0x4000181730 sp=0x4000181700 pc=0xaaaad03f2c7c
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: runtime.sweepone(0xaaaad17013c0)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/runtime/mgcsweep.go:123 +0x4a4 fp=0x40001817a0 sp=0x4000181730 pc=0xaaaad03e6dbc
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: runtime.bgsweep(0x40001a8000)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/runtime/mgcsweep.go:73 +0xdc fp=0x40001817d0 sp=0x40001817a0 pc=0xaaaad03e6834
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: runtime.goexit()
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/runtime/asm_arm64.s:1128 +0x4 fp=0x40001817d0 sp=0x40001817d0 pc=0xaaaad04218ec
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: created by runtime.gcenable
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/runtime/mgc.go:208 +0x54
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 1 [chan receive, 427878 minutes]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/cmd/containerd/command.App.func1(0x40002522c0, 0xaaaad0fa56b8, 0x40002522c0)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/cmd/containerd/command/main.go:174 +0x890
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0xaaaad150e000, 0xaaaad16fe4b8, 0x40002522c0, 0x40001dc8a0, 0x0)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xd0
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0x4000484540, 0x40001c4190, 0x1, 0x1, 0x0, 0x0)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x40c
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: main.main()
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/cmd/containerd/main.go:33 +0x44
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 19 [syscall, 10 minutes]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: os/signal.signal_recv(0xaaaad1739960)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/runtime/sigqueue.go:139 +0xc4
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: os/signal.loop()
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/os/signal/signal_unix.go:23 +0x20
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: created by os/signal.init.0
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/os/signal/signal_unix.go:29 +0x38
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 20 [chan receive]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0xaaaad22d0260)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:879 +0x70
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: created by github.com/containerd/containerd/vendor/github.com/golang/glog.init.0
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:410 +0x2a4
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 159 [select]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Broadcaster).run(0x40001f2640)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:117 +0x108
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewBroadcaster
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:39 +0x164
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 165 [select, 10 minutes]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/cmd/containerd/command.handleSignals.func1(0x40001dc960, 0x40001dc900, 0xaaaad1754f60, 0x40001d6010, 0x40001ba300)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:46 +0x98
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: created by github.com/containerd/containerd/cmd/containerd/command.handleSignals
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:43 +0x78
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 161 [select]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: github.com/containerd/containerd/gc/scheduler.(*gcScheduler).run(0x40005c5200, 0xaaaad1754f60, 0x40001d6010)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:268 +0x148
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: created by github.com/containerd/containerd/gc/scheduler.init.0.func1
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /go/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:132 +0x394
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: goroutine 178 [syscall, 5024 minutes]:
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: syscall.Syscall6(0x16, 0x5, 0x40009dd7b8, 0x80, 0xffffffffffffffff, 0x0, 0x0, 0x1, 0x40009dd7b8, 0x0)
Jan 16 02:44:21 hua-dlzx1-k1011-gyt containerd: /usr/local/go/src/syscall/asm_linux_arm64.s:35 +0x10
...

[ianlancetaylor]

Is there a way that we can recreate the problem?

The current version of Go is 1.17.6. If you are really running 1.2.12 or 1.12.17, please upgrade.

[chwetion]

The frequency of this problem occurs about 2-3 times a year, I have no way of reproducing it right away, even after I upgrade I have no way of knowing if it's fixed.

I want to know what it means when s.state=3, h.sweepgen-s.sweepgen=1

[ianlancetaylor]

Please tell us exactly what version of Go you are using. What does go version print? Thanks.

[chwetion]

use strings /usr/bin/containerd | grep go1
seems go version 1.12.17

_countaacuteacceptaccessacirc;actionactiveacute;aelig;agravealeph;allocsalpha;amacr;amalg;angle;angrt;angst;aogon;appendaring;asymp;atildeattachawint;bcong;bdquo;bepsi;bitmapblank;blk12;blk14;blk34;block;bootIDborderboxDL;boxDR;boxDl;boxDr;boxHD;boxHU;boxHd;boxHu;boxUL;boxUR;boxUl;boxUr;boxVH;boxVL;boxVR;boxVh;boxVl;boxVr;boxdL;boxdR;boxdl;boxdr;boxhD;boxhU;boxhd;boxhu;boxuL;boxuR;boxul;boxur;boxvH;boxvL;boxvR;boxvh;boxvl;boxvr;branchbreve;brvbarbsemi;bsime;bsolb;bumpE;bumpe;bundlecapgetcapsetcaret;caron;causesccaps;ccedilccirc;ccups;cedil;cephfscgroupchan<-check;chrootcinderclientclosedclubs;colon;comma;configcookiecoordscpu.%scpusetcrarr;createcross;csube;csupe;ctdot;cuepr;cuesc;cupor;currencuvee;cuwed;cwint;daggerdarwindashv;dblac;ddarr;definedeletedelim(delta;devicedevptsdharl;dharr;diams;digestdisin;dividedomaindoteq;doubledriverdryRundtdot;dtrif;duarr;duhar;eDDot;eacuteecirc;efDot;efenceeffectegraveemacr;empty empty;eogon;eplus;epsiv;eqsim;equiv;erDot;erarr;errno errorsesdot;eventsexec: execveexist;existsexpectexportfchdirfchmodfchownfflig;field filig;fjlig;fllig;fltns;followforallforkv;formatfownerfrac12frac14frac34frasl;frown;frozenfsTypefsetidfstypegamma;gcirc;gescc;getcpugetcwdgetgidgetpidgetsidgettidgetuidgimel;gneqq;gnsim;go1.5.go1.6.golanggophergrave;groupsgrpclbgsime;gsiml;gtcir;gtdot;hangupharrw;hcirc;headerhealthheartsheighthelliphiddenhoarr;hostIPiacuteicirc;iexcl;igraveiiint;iiota;ijlig;imacr;image;imagesimath;imped;importindex-infin;ingestiogon;iopermip+netiprod;iquestisinE;isins;isinv;iukcy;jcirc;jmath;jsCtx(jukcy;k8s.iokappa;killedlAarr;lBarr;labelslambdalangd;laquo;larrb;lates;latestlayer-layerslbarr;lbbrk;lbrke;lceil;lchownldquo;leakedleaseslegacylescc;lfloorlhard;lharu;lhblk;limitslinkatlistenllarr;lltri;lneqq;lnsim;loang;loarr;lobrk;lookuplopar;lowastlrarr;lrhar;lrtri;lsaquolsime;lsimg;lsquo;ltcir;ltdot;ltrie;ltrif;mDDot;mailtomdash;mdstatmediummemorymethodmicro;middotminus;minutemips64mipselmlock2modifymqueuemremapmsgctlmsggetmsgrcvmsgsndmumap;munmapnabla;napid;napos;nativenatur;nbump;ncong;ndash;ndots:neArr;nearr;nedot;nesim;netbsdnetdnsngeqq;ngsim;nhArr;nharr;nhpar;nlArr;nlarr;nleqq;nless;nlsim;nltri;noexecnofilenomandnosuidnotifynotin;notni;npart;nprec;nrArr;nrarr;nrtri;nsime;nsmid;nspar;nsubE;nsube;nsucc;nsupE;nsupe;ntildenumbernumsp;nvsim;nwArr;nwarr;oacuteobjectocirc;odash;oelig;ofcir;ograveohbar;olarr;olcir;oline;omacr;omega;onlineopenatoperp;oplus;orarr;order;originoslashotildeotimesovbar;parentpariscparsl;pausedpdNamepermilpgpginphone;plusb;pluse;plusmnposterpound;prcue;preadvprefixprime;printfprnap;prsim;ptracequest;rAarr;rBarr;radic;rangd;range;raquo;rarrb;rarrc;rarrw;ratio;rbarr;rbbrk;rbrke;rceil;rdquo;readU4readatreals;reasonrebootref %vremoverenameresultresumeresyncreturnrfloorrhard;rharu;rlarr;rlhar;rnmid;roang;roarr;robrk;rootfsropar;rrarr;rsaquorslaversquo;rtpriortrie;rtrif;rttimerune1 sbquo;scalarscaronsccue;schemescirc;scnap;scopedscopesscriptscsim;sdotb;sdote;seArr;searr;secondsecretsecureselectsemctlsemgetsendtoseriesserversetenvsetgidsetmn;setsidsetuidsha256sha384sha512sharedsharessharp;shmctlshmgetsigma;sigmafsignalsimeq;simgE;simlE;simne;slarr;smile;smtes;socketsocks socks5sourcespadessplicesqcap;sqcup;sqsub;sqsup;srarr;srcdocsrcsetstarf;stat64state(statfsstatusstderrstdoutstringstrns;structsubnE;subne;supnE;supne;swArr;swarr;sweep switchsyncfssyslogsystemszlig;taintstargettelnettest.vtgkillthawedthere4theta;thinspthkap;thorn;tilde;times;tls13 token(toml: trade;trisb;tshcy;twixt;typeofuacuteubrcy;ucirc;udarr;udhar;ugraveuharl;uharr;uhblk;uint16uint32uint64ultri;umacr;umountunlinkunpigzunuseduogon;updateuplus;upsih;uring;urtri;usemapustar ustar
/* nil *//dev/full/dev/null/dev/zero/etc/mtab/proc/bus/proc/irq/proc/sys127.0.0.12001::/322002::/162441406253ffe::/16: expect : inuse: : status=; Domain=<a href="<invalid><unknown>APIGroup:Accuracy(Affinity:ArrayHashAssigned:Audience:AuthorityBassa_VahBearer %sBhaiksukiBigEndianBlkIOStatBoolValueCANCELLEDCAP_CHOWNCAP_MKNODCNI_ARGS=CNI_PATH=CODE_SIZECOMMITTEDCONTAINERCallSliceCapacity:ClaimRef:ClientIP:CommittedCompletedConfigMapContainerContinue:CpuCount:CpuQuota:CriuPath:CuneiformDATA_LOSSDDotrahd;DNSConfigDecimalSIDetails: DiacriticDiskName:DotEqual;Duration:EmptyDir:EndpointsEnvelope:ExitCode:ExitedAt:Expected:ExtraInfoFieldMaskFieldRef:FilePath:ForbiddenGC (idle)HISTOGRAMHTTP/1.0 HTTP/1.1 Help_LinkHex_DigitHostAliasHostPath:HostPort:Hostname:IP4:%+v, IP6:%+v, ImageRef:ImageSpecInheritedInstMatchInstRune1Integral;InterfaceJS_NORMALJS_NUMBERJS_STRINGKeyToPathKhudawadiLessLess;LifecycleListMeta:ListValueLoadImageMalayalamMessage: Metadata:MicroTimeMicrosoftMongolianMonitors:NEWCGROUPNOT_FOUNDNabataeanNamespaceNodeInfo:NodeName:NodePort:Not FoundNotEqual;NotTilde;NullValueOOMKilledOperator:Optional:Options: Package: Pagesize:PalmyreneParseBoolParseUintPartialD;Password:PodStatusPrecedes;Previous:Priority:Proc-TypeProtocol:PullImageQOSClass:RBDImage:REDACTED+RLIMIT_ASRPC: [%v]RPC: [OK]RdmaEntryReadArrayReadInt16ReadInt32ReadInt64ReadMapCBReadOnly:ReadUint8Readonly:Registry:RemoveAllReplicas:RepoTags:Requests:Required.Required:Requires:ResizePtyResource:Revision:RightTee;RxErrors:SHALL NOTSamaritanSearches:Seconds: See OtherSelector:SelfLink:SeparatorSeptemberService: Statuses:Subject: SubscribeSucceeds;SuchThat;SundaneseSuperset;TYPE_BOOLTYPE_ENUMTYPE_IPV4TYPE_IPV6TYPE_UNIXTemplate:Terminal:TimestampToo EarlyTotalRSS:Trailer: TxErrors:TypeMeta:TypeUrl: Uarrocir;UnderBar;UnmarshalUse ProxyUsername:Versions:VolumeID:Wednesday[%v = %d][:^word:][:alnum:][:alpha:][:ascii:][:blank:][:cntrl:][:digit:][:graph:][:lower:][:print:][:punct:][:space:][:upper:][a-z0-9]+[command][mapElem]"ABORTED""INVALID""UNKNOWN"^[0-9]+%$address,aaddressesallocatedandslope;angmsdaa;angmsdab;angmsdac;angmsdad;angmsdae;angmsdaf;angmsdag;angmsdah;angrtvbd;any.protoapi.protoapproxeq;atomicor8attempts:authorityautofocusavailableawconint;azureDiskazureFilebackcong;bad indirbarwedge;bbrktbrk;bigoplus;bigsqcup;biguplus;bigwedge;boxminus;boxtimes;broadcastbsolhsub;buddyinfobus errorbytes %d-bytes=%d-capbrcup;casttype=challengechan sendcircledR;circledS;cirfnint;claimNameclient_idclubsuit;clusterIPcomplex64componentconfigMapcontainercontinuedcopystackcpu_totalcpuset.%scriu-workctxt != 0cupbrcap;curlyvee;cwconint;d.nx != 0date-timedirectorydns,filesdnsConfigdnsPolicydocker.iodoteqdot;dotminus;draggabledrbkarow;dzigrarr;element <elinters;embedded=empty urlemptyset;endpointsepoll_ctleqvparsl;escaping eventTimeexpect %sfaccessatfadvise64fallocatefdatasyncfec0::/10fgetxattrfieldPathfiles,dnsfont/wofffork/execfpartint;fsetxattrfstatat64fstatfs64ftruncatefuncargs(futimesatgc failedgeqslant;gesdotol;getegid32geteuid32getgroupsgetitimergetrandomgetresgidgetresuidgetrlimitgetrusageglusterfsgnapprox;go1.12.17goroutinegrpc_codegrpc_typehksearow;hkswarow;hostnamesimage/bmpimage/gifimage/pngimagline;imagpart;infintie;integers;intercal;interfaceinterruptintlarhk;io_cancelio_submitip6.arpa.ipc_owneripv6-icmplabels.%qlabels.%slaemptyv;last hourlastStateldrushar;leqslant;lesdotor;lgetxattrlifecyclelistxattrllcorner;lnapprox;localhostlocaltimelog-levellowerdir=lrcorner;lsetxattrlurdshar;mSpanDeadmSpanFreemac_adminmachineIDmanifest-manifestsmapstoup;maxlengthmediatypememory.%smips64n32mode=0620mode=1777mountPathmq_notifymq_unlinkmulticastmultimap;namespacenanosleepnaturals;ncongdot;net/http.net/ip_vsnet_adminnewosprocnil errornot foundnotindot;nr_blocksnr_inodesomitemptyotimesas;overlayfsoverwritepanic: %vpanicwaitparallel;partitionplusacir;pointint;postStartprecneqq;precnsim;prlimit64procMountprofalar;profline;profsurf;projectedprotocol questionsraemptyv;raw-writereadaheadrealpart;reconcilerecover: referencereflect: renameat2requestIDresolvingresourcesrppolint;rtriltri;runAsUserrwxrwxrwxsandboxesschedulerscopeNamescpolint;secretRefsetgroupssetitimersetminus;setresgidsetresuidsetrlimitshareNameshim.sockshortmid;show helpsignal 32signal 33signal 34signal 35signal 36signal 37signal 38signal 39signal 40signal 41signal 42signal 43signal 44signal 45signal 46signal 47signal 48signal 49signal 50signal 51signal 52signal 53signal 54signal 55signal 56signal 57signal 58signal 59signal 60signal 61signal 62signal 63signal 64signalfd4signaturesigreturnsinceTimesizeBytessizeLimitsmeparsl;snapshotssqsubset;sqsupset;startFromstartTimestartedAtstartedatstatvers=stdinOncestorageosstream:%dsubdomainsubseteq;subvolumesucceededsuccneqq;succnsim;supseteq;symlinkatsys_adminsys_bytessys_pacctsys_rawiotailLinestargetReftcpSockettest.listthetasym;thicksim;time.TimetimeAddedtimesbar;tmpmountstotal_rsstriangle;trimSpacetriminus;trpezium;tx closedulcorner;underflowunhandledunix://%suntrustedupdatedaturcorner;urn:uuid:utimensatvalueFromvarkappa;varsigma;vartheta;video/avivideo/mp4websocketwriteback{Unknown}} stack=[ (deleted) MB goal,  actual
/go/src/github.com/containerd/containerd/vendor/golang.org/x/net/trace/trace_go17.go
/go/src/github.com/containerd/containerd/vendor/golang.org/x/net/context/go17.go

Please tell us exactly what version of Go you are using. What does go version print? Thanks.

[davecheney]

@chwetion thank you for confirming. You must upgrade before we can assist you further.

[seankhliao]

No new info