👋 We wrote this to figure out why our CI had slowed down a bit and found something was accidentally compiling in much more Kubernetes than an offline CLI needed. If you enjoy writing performant Go too, please do considering joining us in the fight against fraud: ravelin.com/careers.
actiongraph is a CLI for investigating where go build is spending its time
compiling. It consumes the file written using the compile.json output by go build -debug-actiongraph=compile.json, which includes information about compile
steps and dependencies between packages.
actiongraph can help you identify which packages take a lot of compilation
time with top, summarise where that time is spent with tree, and identify
where the dependencies come from with graph --why.
Alongside the -debug-actiongraph flag is the -debug-trace flag which this
program does not use, but is a similarly instructive tool which can help you
identify parallelism and build order. For completeness, it is also shown below.
ℹ Both the
-debug-actiongraphand-debug-traceflags are not supported and deliberately not documented by the Go team: https://github.com/golang/go/commit/8fce59eab5cb2facfafca89e047b4b43ba44785f.
go install github.com/icio/actiongraph@latest
# Optionally clear your go build cache, if you're wanting to understanding why
# a build in CI is taking a while (depending on its cache reuse):
go clean -cache
# Compile your program using the undocumented -debug-actiongraph flag:
go build -debug-actiongraph=compile.json ./my-prog
# Show the slowest individual packages:
actiongraph top -f compile.json
# Show aggregate time spent compiling nested packages:
actiongraph tree -f compile.json
# Show aggregate time spent compiling github packages:
actiongraph tree -f compile.json -L 2 github.com
# Render dependency diagrams of packages, focusing on why PKG was compiled in:
actiongraph graph --why PKG -f compile.json > compile-pkg.dot
dot -Tsvg -Grankdir=LR < compile-pkg.dot > compile-pkg.svg
In this example, we're going to look inside one of @icio's favourite CLIs, https://k9scli.io/.
First, let's check that we're all set up:
go version
actiongraph -h
then create our demo space:
mkdir demo
cd demo
We want to investigate a fresh build, so let's clear the build cache:
go clean -cache
Now we compile. We're going to specify body -debug-actiongraph and -debug-trace here so we can compare the results. (And we're going to send the compiled binary into a temporary directory.)
GOBIN=$(mktemp -d) go install -debug-actiongraph=k9s.json -debug-trace=k9s-trace.json github.com/derailed/k9s@v0.27.4
We should have a new file called k9s.json describing the actiongraph. Let's take a look at the first few entries jq '.[:5] | .[]' -c < k9s.json:
{"ID":0,"Mode":"go install","Package":"","Deps":[1],"Priority":1190,"TimeReady":"2023-05-12T09:24:39.647607494+01:00","TimeStart":"2023-05-12T09:24:39.647610965+01:00","TimeDone":"2023-05-12T09:24:39.647611018+01:00","Cmd":null}
{"ID":1,"Mode":"link-install","Package":"github.com/derailed/k9s","Deps":[2],"Objdir":"/tmp/go-build3966251150/b001/","Target":"/tmp/tmp.WIZyR7yX3Y/k9s","Priority":1189,"Built":"/tmp/tmp.WIZyR7yX3Y/k9s","BuildID":"LZKyYv0RT8N-_YjcRQ1R/PsuartwNDnqAz2PYjtcW/lvWIs0IgoEnXVRitK8yx/ClYoHVGOEFtTZ-Gnzq2h","TimeReady":"2023-05-12T09:24:39.647340894+01:00","TimeStart":"2023-05-12T09:24:39.647348289+01:00","TimeDone":"2023-05-12T09:24:39.647607252+01:00","Cmd":null}
{"ID":2,"Mode":"link","Package":"github.com/derailed/k9s","Deps":[3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023,1024,1025,1026,1027,1028,1029,1030,1031,1032,1033,1034,1035,1036,1037,1038,1039,1040,1041,1042,1043,1044,1045,1046,1047,1048,1049,1050,1051,1052,1053,1054,1055,1056,1057,1058,1059,1060,1061,1062,1063,1064,1065,1066,1067,1068,1069,1070,1071,1072,1073,1074,1075,1076,1077,1078,1079,1080,1081,1082,1083,1084,1085,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1096,1097,1098,1099,1100,1101,1102,1103,1104,1105,1106,1107,1108,1109,1110,1111,1112,1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1123,1124,1125,1126,1127,1128,1129,1130,1131,1132,1133,1134,1135,1136,1137,1138,1139,1140,1141,1142,1143,1144,1145,1146,1147,1148,1149,1150,1151,1152,1153,1154,1155,1156,1157,1158,1159,1160,1161,1162,1163,1164,1165,1166,1167,1168,1169,1170,1171,1172,1173,1174,1175,1176,1177,1178,1179,1180,1181,1182,1183,1184,1185,1186,1187,1188],"Objdir":"/tmp/go-build3966251150/b001/","Target":"/tmp/go-build3966251150/b001/exe/a.out","Priority":1188,"Built":"/tmp/go-build3966251150/b001/exe/a.out","ActionID":"LZKyYv0RT8N-_YjcRQ1R","BuildID":"LZKyYv0RT8N-_YjcRQ1R/PsuartwNDnqAz2PYjtcW/lvWIs0IgoEnXVRitK8yx/ClYoHVGOEFtTZ-Gnzq2h","TimeReady":"2023-05-12T09:24:33.62117428+01:00","TimeStart":"2023-05-12T09:24:33.621181576+01:00","TimeDone":"2023-05-12T09:24:39.647340491+01:00","Cmd":["/home/icio/sdk/go/pkg/tool/linux_amd64/link -o /tmp/go-build3966251150/b001/exe/a.out -importcfg /tmp/go-build3966251150/b001/importcfg.link -X=runtime.godebugDefault=panicnil=1 -buildmode=exe -buildid=LZKyYv0RT8N-_YjcRQ1R/PsuartwNDnqAz2PYjtcW/lvWIs0IgoEnXVRitK8yx/LZKyYv0RT8N-_YjcRQ1R -extld=gcc /tmp/go-build3966251150/b001/_pkg_.a"],"CmdReal":5757072143,"CmdUser":6457235000,"CmdSys":853740000}
{"ID":3,"Mode":"build","Package":"github.com/derailed/k9s","Deps":[4,5,6,7,8,1189],"Objdir":"/tmp/go-build3966251150/b001/","Priority":1187,"NeedBuild":true,"ActionID":"PsuartwNDnqAz2PYjtcW","BuildID":"PsuartwNDnqAz2PYjtcW/lvWIs0IgoEnXVRitK8yx","TimeReady":"2023-05-12T09:24:33.604116889+01:00","TimeStart":"2023-05-12T09:24:33.604117711+01:00","TimeDone":"2023-05-12T09:24:33.621174013+01:00","Cmd":["/home/icio/sdk/go/pkg/tool/linux_amd64/compile -o /tmp/go-build3966251150/b001/_pkg_.a -trimpath \"/tmp/go-build3966251150/b001=>\" -p main -lang=go1.20 -complete -buildid PsuartwNDnqAz2PYjtcW/PsuartwNDnqAz2PYjtcW -c=4 -nolocalimports -importcfg /tmp/go-build3966251150/b001/importcfg -pack /home/icio/go/pkg/mod/github.com/derailed/k9s@v0.27.4/main.go"],"CmdReal":15611237,"CmdUser":16273000}
{"ID":4,"Mode":"build","Package":"flag","Deps":[9,10,11,12,13,14,8,15,16,17,18],"Objdir":"/tmp/go-build3966251150/b002/","Priority":44,"NeedBuild":true,"ActionID":"fhRdV7jbdbpj3WwhqK7M","BuildID":"fhRdV7jbdbpj3WwhqK7M/fpr490Yv8t_PFttE-9YV","TimeReady":"2023-05-12T09:23:48.146874388+01:00","TimeStart":"2023-05-12T09:23:48.146904614+01:00","TimeDone":"2023-05-12T09:23:48.29620602+01:00","Cmd":["/home/icio/sdk/go/pkg/tool/linux_amd64/compile -o /tmp/go-build3966251150/b002/_pkg_.a -trimpath \"/tmp/go-build3966251150/b002=>\" -p flag -std -complete -buildid fhRdV7jbdbpj3WwhqK7M/fhRdV7jbdbpj3WwhqK7M -c=4 -nolocalimports -importcfg /tmp/go-build3966251150/b002/importcfg -pack /home/icio/sdk/go/src/flag/flag.go"],"CmdReal":140280033,"CmdUser":144616000,"CmdSys":22834000}
So let's find the compile steps that took the longest:
$ actiongraph -f k9s.json top
9.016s 2.75% build k8s.io/api/core/v1
6.026s 4.59% link github.com/derailed/k9s
5.071s 6.13% build github.com/aws/aws-sdk-go/service/s3
3.620s 7.23% build github.com/aws/aws-sdk-go/aws/endpoints
3.474s 8.29% build net/http
3.215s 9.27% build net
2.869s 10.15% build github.com/google/gnostic/openapiv2
2.846s 11.02% build github.com/google/gnostic/openapiv3
2.581s 11.80% build k8s.io/apimachinery/pkg/apis/meta/v1
2.476s 12.56% build google.golang.org/protobuf/internal/impl
2.186s 13.22% build github.com/gogo/protobuf/proto
1.974s 13.83% build k8s.io/api/extensions/v1beta1
1.919s 14.41% build github.com/derailed/tview
1.898s 14.99% build k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1
1.840s 15.55% build github.com/klauspost/compress/zstd
1.796s 16.10% build github.com/derailed/k9s/internal/view
1.671s 16.61% build k8s.io/kubectl/pkg/describe
1.658s 17.11% build github.com/prometheus/procfs
1.624s 17.61% build k8s.io/api/apps/v1
1.596s 18.09% build runtime
By default, actiongraph top will show the 20 slowest steps. This may be
overridden using the -n0 flag. For example, to get the fastest steps we could
check:
$ actiongraph -f k9s.json top -n0 | tail -5
0.006s 100.00% build google.golang.org/protobuf/internal/flags
0.000s 100.00% link-install github.com/derailed/k9s
0.000s 100.00% nop
0.000s 100.00% built-in package unsafe
0.000s 100.00% go install
Here we can see that the second column is showing the cumulative percentage of time spent up to that package.
We can summarise the time spent within package domain/directories using the
tree subcommand:
$ actiongraph -f k9s.json tree -L 1
322.013s (root)
154.673s k8s.io
83.357s github.com
40.294s std
12.338s sigs.k8s.io
10.886s golang.org
7.178s google.golang.org
5.413s helm.sh
2.428s gopkg.in
2.344s go.starlark.net
1.702s go.opentelemetry.io
1.400s oras.land
We can see that we spent 322 seconds compiling k9s, though some of that compilation would have happened in parallel, and so the
We've rolled up all standard libraries under std:
$ actiongraph -f k9s.json tree encoding
322.013s (root)
40.294s std
1.942s 0.017s std/encoding
0.572s 0.572s std/encoding/json
0.564s 0.564s std/encoding/xml
0.207s 0.207s std/encoding/asn1
0.192s 0.192s std/encoding/binary
0.096s 0.096s std/encoding/csv
0.080s 0.080s std/encoding/hex
0.074s 0.074s std/encoding/base64
0.073s 0.073s std/encoding/pem
0.067s 0.067s std/encoding/base32
Let's look at which github repos are taking the longest to compile:
$ actiongraph -f k9s.json tree github.com -L 2 | head -15
322.013s (root)
83.357s github.com
17.307s github.com/aws
17.307s github.com/aws/aws-sdk-go
16.360s github.com/derailed
7.674s 0.017s github.com/derailed/k9s
4.517s github.com/derailed/popeye
2.250s github.com/derailed/tcell
1.919s 1.919s github.com/derailed/tview
8.206s github.com/google
6.524s github.com/google/gnostic
1.014s github.com/google/go-cmp
0.277s 0.277s github.com/google/btree
0.163s 0.115s github.com/google/gofuzz
0.150s 0.150s github.com/google/uuid
We saw from top that package github.com/aws/aws-sdk-go/service/s3 was one of the slowest to compile. To understand why, we're going to use the graph subcommand which can filter down the dependency list to highlight all import paths leading from our build target to the package indicated by --why PKG:
actiongraph -f k9s.json graph --why github.com/aws/aws-sdk-go/service/s3 > k9s-s3.dot
and then render it using Graphviz's dot:
dot -Tsvg < k9s-s3.dot > k9s-s3.svg
which looks something like this:
Lastly we can open our k9s-trace.json in https://ui.perfetto.dev/ or Google Chrome/Chromium's chrome://tracing. This shows us how successfully the scheduler utilises the multiple cores on our machine when compiling each package:
At the time of writing, -debug-trace is under development in
golang/go#38714.