Nix throws away crucial end of log when printed too fast

[roberth]

Describe the bug

If a derivation logs too fast before exiting, it is cut off. This removes the last lines, which tend to contain important information about the failure.

Steps To Reproduce

{ pkgs ? import <nixpkgs> {} }:

pkgs.runCommand "log-fast-die-young" { } ''
  for ((i=0; i<10000; i=i+1)) do
    echo "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa $i"
  done
  echo 'crucial info about the failure'
  exit 1
''

Expected behavior

When the builder exits, do something lenient but robust, like read another 128 KiB, and stop after 10 seconds.
If we hit one of the limits, add a line to the log to explain that the log is incomplete.

nix-env --version output

Additional context

I've talked about this problem before, but couldn't find an issue.
Keywords: Log truncation, derivation logging, rapid log output, truncated build logs, log cutoff issue, insufficient log capture, lost log data, build failure information loss, logging speed limitation, incomplete log output

Priorities

Add 👍 to issues you find important.

[roberth]

Possible workaround: avoid remote builder, --builders ""
This might increase the throughput, reducing the backpressure so that the logs can be read before the exit code is processed semi-independently. (hypothesis)