Filter programs e.g. iconv and petcat does not work in real time with -echo

[mobluse]

I can not get filter programs to work with x16emu -echo. I.e. this does not work in real time, but it does work if you close the emulator:
x16emu -echo | iconv --from-code=ISO8859-15 --to-code=UTF-8
x16emu -echo | petcat -c -nh
If we could get filter programs to work in real time we could show correct output in the terminal when using -echo, or better with the new suggested option -echoraw, see #135. PetCat is from VICE 3.3. IConv is included in Linux.

I tried to prevent all control characters and all 8-bit characters from printing when using -echo, but filters do still not work in real time, as they usually do in Linux.

Originally posted by @mobluse in #112 (comment)

[FreeFull]

By default, stdout is line buffered when outputted directly to a terminal, and block buffered when redirected with a pipe. Either flushing after every line or somehow changing things to always be line buffered should work.

[mist64]

fflush(stdout); is a simple fix, and at our data rates not too expensive.

[madebr]

Does expect unbuffer help?
unbuffer x16emu -echo | iconv --from-code=ISO8859-15 --to-code=UTF-8

or better, use setvbuf

setvbuff(stdout, NULL, _IONBF, 0)

The setvbuf() function may only be used after opening a stream and before any other operations have been performed on it.

[mobluse]

I had to install the package expect in Raspbian Buster to get unbuffer, but it didn't help. It acts as before.

[madebr]

@mobluse
Adding

    if (echo_mode) {
        int res = setvbuf(stdout, NULL, _IONBF, 0);
	if (res != 0) {
	    printf("Cannot disable buffering\n");
	    exit(1);
	}
    }

in main.c before initAudio() will let x16emu -echo | cat work as expected.
iconv does some internal buffering.

[mobluse]

One can switch on echo using POKE even though it was off when one started. #27

I believe this would not work if it's only done when initializing.

Another solution would be to create a new printf() function that would always flush.

One might want to filter the logs also.

[madebr]

@mobluse Looking back at it, I think using prinf + fflush is better as the setvbuf is global solution for a local problem.
Adding a fflush after this printf would also work, I think. So no need for a new printf function.
It will also work fine together with #26

x16-emulator/main.c

Line 844 in 0d50e33

printf("%c", c);

What filtering would you like to see applied?

[mobluse]

I would like to use iconv, petcat, awk, sed, and my own filters. I am working on something now which is currently built in to the emulator, but that could be an external filter in stead. #112

[mobluse]

The current echo changes CR = \r = 0x0D to LF = \n = 0x0A. Then it's not really raw. There are also \n sent from the ROM or some other system to the echo part, i.e. before the translation of CR to LF. There are more newlines in the output on the terminal than on screen. This could be because the X16 ignores them on the screen. E.g. when starting two extra linefeeds/newlines around READY. are sent before translation:

◥  ◥   ◤  ◤
 ◥  ◥ ◤  ◤  **** COMMANDER X16 BASIC V2 ****
  ◥     ◤
            2048K RAM SYSTEM
  ◤     ◥
 ◤  ◤ ◥  ◥  38655 BASIC BYTES FREE
◤  ◤   ◥  ◥

�
READY.
�

The extra newlines are shown here as �. Compare this to the output screen. This should perhaps be a new issue, but it's related to echo.

I discovered one place in the ROM that sends LF, see commanderx16/x16-rom#37

I think there should be several suboptions to -echo, e.g. raw, default, auto.

[mist64]

The original issue is fixed, closing.