CONTROL_PLANE_STORAGE_BUFFER is larger than virtual-hardware vdevs and isn't configurable

[gjcolombo]

Development clusters set up using the instructions in how-to-run.adoc create a set of vdevs that serve as the fake sled's M.2 and U.2 storage devices. By default these vdevs are 20 GiB.

Nexus (as of the recent disk exhaustion prevention work) defines a CONTROL_PLANE_STORAGE_BUFFER constant that sets aside 250 GiB on each U.2 for control plane storage. This doesn't prevent the control plane from coming up, but it does render the resulting environment more or less unusable: all attempts to create disks or upload images to it will return 507 Insufficient Capacity, since all of the U.2s appear to be oversubscribed from the outset.

It's possible to work around this by patching CONTROL_PLANE_STORAGE_BUFFER to 0 and rebuilding Nexus, but it would be nice to make this configurable somehow (similar to what we do for reservoir sizes, though those are sled-agent tunables and not Nexus tunables).

[davepacheco]

Related: #7875.

[rcgoodfellow]

A workaround for this on a live dev environment is:

for ZPOOL in $(pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list -i);
do
    pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb -w db zpool set-storage-buffer "${ZPOOL}" 0
done