zed process exits after "Processing events since eid=0"

[beertje44]

System information

Type	Version/Name
Distribution Name	AlmaLinux
Distribution Version	9.5
Kernel Version	6.12.16
Architecture	x86_64
OpenZFS Version	2.3.1

Describe the problem you're observing

After updating to 2.3.1 the zed no longer starts. It exists after printing: "Processing events since eid=0".
Downgraded to 2.3.0 and it now runs again, didn't need a reboot so I guess it's not zfs kernel module related.

Describe how to reproduce the problem

• Update to 2.3.1
• reboot
• the systemd service zfs-zed fails to start

Include any warning/errors/backtraces from the system logs

[ningw42]

Same here. The log comes from https://github.com/openzfs/zfs/blob/f3e4043a36942e67ccbc05318479a07d242fc611/cmd/zed/zed_event.c#L217C1-L224C67.

I didn't reboot, but when NixOS activates the new generation, it fails to restart zfs-zed.service. Not sure if the mismatch between kernel mod and user land package played a role here. IIRC, it has always been the case (mismatch) after a zfs update until a reboot.

Mar 16 12:19:44 yuki systemd[1]: Started ZFS Event Daemon (zed).
Mar 16 12:19:44 yuki zed[519250]: ZFS Event Daemon 2.3.1-1 (PID 519250)
Mar 16 12:19:44 yuki zed[519250]: Failed to seek to eid=0
Mar 16 12:19:44 yuki zed[519250]: Processing events since eid=0
Mar 16 12:19:44 yuki zed[519250]: Exiting

❯ zfs --version
zfs-2.3.1-1
zfs-kmod-2.3.0-1

Update:
After a reboot, when the kernel mod matches userland package, zfs-zed.service successfully restarts.

❯ sudo systemctl status zfs-zed.service
● zfs-zed.service - ZFS Event Daemon (zed)
     Loaded: loaded (/etc/systemd/system/zfs-zed.service; enabled; preset: ignored)
    Drop-In: /nix/store/qih3pv2ibivh8jj8bbqa2fzj3hqbgs9p-system-units/zfs-zed.service.d
             └─overrides.conf
     Active: active (running) since Mon 2025-03-17 19:16:10 CST; 2min 35s ago
 Invocation: 4a038df44ed0429a87f5c11a9cb53a17
       Docs: man:zed(8)
   Main PID: 1982 (zed)
         IP: 0B in, 0B out
         IO: 236K read, 0B written
      Tasks: 3 (limit: 154348)
     Memory: 1.9M (peak: 6.9M)
        CPU: 36ms
     CGroup: /system.slice/zfs-zed.service
             └─1982 /nix/store/b492b02skizdlbk8rn47hv8kn709mask-zfs-user-2.3.1/sbin/zed -F

-- Boot 6c1c5ea7fd774ccebc339500b2643221 --
Mar 17 19:16:10 yuki systemd[1]: Started ZFS Event Daemon (zed).
Mar 17 19:16:10 yuki zed[1982]: ZFS Event Daemon 2.3.1-1 (PID 1982)
Mar 17 19:16:10 yuki zed[1982]: Processing events since eid=0
Mar 17 19:16:10 yuki zed[2018]: eid=2 class=config_sync pool='data'
Mar 17 19:16:10 yuki zed[2025]: eid=5 class=config_sync pool='data'

[amotin]

It is likely a known issue: #17137 .

[beertje44]

No, my mistake: zfs-dracut was installed. Since I don't use zfs on my root device it wasn't needed. However the zfs modules were updated by the dnf upgrade process but the initrd image wasn't. Still can be a problem I guess, but it shouldn't be for me.