invalid memory address or nil pointer dereference in GCP bucket provider (v.0.29.0)

[kunzese]

Hiho,

we try to fetch some helm charts from a GCP bucket using Workload Identity, but the source-controller v.0.29.0 runs into an invalid memory address or nil pointer dereference error.

Bucket definition:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: Bucket
metadata:
  name: my-bucket
  namespace: flux-system
spec:
  interval: 1m
  bucketName: my-bucket
  endpoint: storage.googleapis.com
  provider: gcp

Error:

│ E0922 07:16:25.547477       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)                                                     │
│ goroutine 373 [running]:                                                                                                                                                                                                                  │
│ k8s.io/apimachinery/pkg/util/runtime.logPanic({0x4f08a0?, 0x3fd8ea0})                                                                                                                                                                     │
│     k8s.io/apimachinery@v0.25.0/pkg/util/runtime/runtime.go:75 +0x99                                                                                                                                                                      │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()                                                                                                                                                    │
│     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:110 +0xb9                                                                                                                                                │
│ panic({0x4f08a0, 0x3fd8ea0})                                                                                                                                                                                                              │
│     runtime/panic.go:838 +0x207                                                                                                                                                                                                           │
│ cloud.google.com/go/storage.getAgeCondition({0x32ca00?, 0x0?})                                                                                                                                                                            │
│     cloud.google.com/go/storage@v1.25.0/bucket.go:1539 +0xc5                                                                                                                                                                              │
│ cloud.google.com/go/storage.toLifecycle(0xc002d758c0?)                                                                                                                                                                                    │
│     cloud.google.com/go/storage@v1.25.0/bucket.go:1565 +0x1d1                                                                                                                                                                             │
│ cloud.google.com/go/storage.newBucket(0xc00113fa40)                                                                                                                                                                                       │
│     cloud.google.com/go/storage@v1.25.0/bucket.go:726 +0x77d                                                                                                                                                                              │
│ cloud.google.com/go/storage.(*httpStorageClient).GetBucket(0xc002d7d600, {0xe23b28, 0xc002e3ad50}, {0xc002b76570, 0x13}, 0x50?, {0xc0053256e0?, 0xe00000000032c940?, 0xc001727568?})                                                      │
│     cloud.google.com/go/storage@v1.25.0/http_client.go:279 +0x518                                                                                                                                                                         │
│ cloud.google.com/go/storage.(*BucketHandle).Attrs(0xc0017275f0, {0xe23af0?, 0xc002e13380?})                                                                                                                                               │
│     cloud.google.com/go/storage@v1.25.0/bucket.go:148 +0x344                                                                                                                                                                              │
│ github.com/fluxcd/source-controller/pkg/gcp.(*GCSClient).BucketExists(0xe23b28?, {0xe23af0?, 0xc002e13380?}, {0xc002b76570?, 0xc000945118?})                                                                                              │
│     github.com/fluxcd/source-controller/pkg/gcp/gcp.go:86 +0x1bc                                                                                                                                                                          │
│ github.com/fluxcd/source-controller/controllers.fetchEtagIndex({0xe23b28?, 0xc002e3a000?}, {0xe26200, 0xc00012cdf0}, 0xc00071cb40, 0xc002d15500, {0xc002d65d80, 0x36})                                                                    │
│     github.com/fluxcd/source-controller/controllers/bucket_controller.go:750 +0xf1                                                                                                                                                        │
│ github.com/fluxcd/source-controller/controllers.(*BucketReconciler).reconcileSource(0xc0004f24b0?, {0xe23b28, 0xc002e3a000}, 0xc00071cb40, 0xc002d15500, {0xc002d65d80, 0x36})                                                            │
│     github.com/fluxcd/source-controller/controllers/bucket_controller.go:503 +0x6bb                                                                                                                                                       │
│ github.com/fluxcd/source-controller/controllers.(*BucketReconciler).reconcile(0xe36030?, {0xe23b28, 0xc002e3a000}, 0xc00071cb40, {0xc001727c48, 0x3, 0x7f2386c97e00?})                                                                    │
│     github.com/fluxcd/source-controller/controllers/bucket_controller.go:358 +0x519                                                                                                                                                       │
│ github.com/fluxcd/source-controller/controllers.(*BucketReconciler).Reconcile(0xc0004f24b0, {0xe23b28, 0xc002e3a000}, {{{0xc000a4ccb0?, 0x10?}, {0xc000a5e4c8?, 0x2160327?}}})                                                            │
│     github.com/fluxcd/source-controller/controllers/bucket_controller.go:318 +0x55c                                                                                                                                                       │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xe23a80?, {0xe23b28?, 0xc002e3a000?}, {{{0xc000a4ccb0?, 0x7775a0?}, {0xc000a5e4c8?, 0x2155854?}}})                                                        │
│     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 +0xc8                                                                                                                                                │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000b06e0, {0xe23a80, 0xc000902140}, {0x5bd3a0?, 0xc002dc2880?})                                                                                 │
│     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 +0x33c                                                                                                                                               │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000b06e0, {0xe23a80, 0xc000902140})                                                                                                          │
│     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 +0x1d9                                                                                                                                               │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()                                                                                                                                                      │
│     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 +0x85                                                                                                                                                │
│ created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2                                                                                                                                               │
│     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:230 +0x325                                                                                                                                               │
│ {"level":"error","ts":"2022-09-22T07:16:25.547Z","msg":"Reconciler error","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket","bucket":{"name":"my-bucket","namespace":"flux-system"}, │
│ "namespace":"flux-system","name":"my-bucket","reconcileID":"4c2ffc10-417f-4794-88f9-5f5cc5fdb6c5","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}

The method getAgeCondition has been removed in the latest Storage API client v0.26.0:

(googleapis/google-cloud-go@storage/v1.25.0...storage/v1.26.0#diff-0c42e3f240ee8d43e271bb6004c8857419ae170063a1d69227cb9811325bd2de)

[kunzese]

@mschaller FYI

[mschaller]

same panic for source-controller 0.28.0 but 0.27.0 works fine without panics

[somtochiama]

The crash seems to be from the gcp library

@kunzese I am trying to recreate this. I have tried w a non-existent bucket, and an identity that has no access (and one that does)

[kunzese]

@somtochiama our bucket is existing and we confirmed that everything around the workload identity thingy from GKE is working correctly with

kubectl run debug -n flux-system -i --tty --restart=Never --rm --overrides='{ "spec": { "serviceAccount": "source-controller" } }' --image gcr.io/cloud-builders/gsutil ls gs://my-bucket

The content of the bucket is listed without any issues.