Random tests fail and crash on Mac OS

[Gerold103]

Tarantool version:
master
OS version:
Mac
Bug description:
Every time I run the full suite on the master branch, I get spontaneous crashes and hangs. Example of a crash:

[032] box/func_reload.test.lua                                        
[032] 
[032] [Instance "box" killed by signal: 6 (SIGABRT)]
[032] 
[032] Last 15 lines of Tarantool Log file [Instance "box"][/Users/gerold/Work/Repositories/tarantool/test/var/032_box/box.log]:
[032] 2019-08-22 22:14:42.026 [38110] snapshot/101/main I> done
[032] 2019-08-22 22:14:42.029 [38110] main I> removed /Users/gerold/Work/Repositories/tarantool/test/var/032_box/box/00000000000000000120.snap
[032] 2019-08-22 22:14:42.029 [38110] wal I> removed /Users/gerold/Work/Repositories/tarantool/test/var/032_box/box/00000000000000000120.xlog
[032] 2019-08-22 22:14:42.453 [38110] main/109/main I> joining replica eb8f7d4a-0e37-4a9f-940d-eb57756314c8 at fd 43, aka unix/:/Users/gerold/Work/Repositories/tarantool/test
[032] 2019-08-22 22:14:42.578 [38110] main/109/main I> initial data sent.
[032] 2019-08-22 22:14:42.578 [38110] main I> assigned id 2 to replica eb8f7d4a-0e37-4a9f-940d-eb57756314c8
[032] 2019-08-22 22:14:42.579 [38110] relay/unix/:(socket)/101/main I> recover from `/Users/gerold/Work/Repositories/tarantool/test/var/032_box/box/00000000000000002296.xlog'
[032] 2019-08-22 22:14:42.579 [38110] main/109/main I> final data sent.
[032] 2019-08-22 22:14:42.622 [38110] main/109/main I> subscribed replica eb8f7d4a-0e37-4a9f-940d-eb57756314c8 at fd 43, aka unix/:/Users/gerold/Work/Repositories/tarantool/test
[032] 2019-08-22 22:14:42.622 [38110] main/109/main I> remote vclock {1: 2298} local vclock {1: 2298}
[032] 2019-08-22 22:14:42.623 [38110] relay/unix/:(socket)/101/main I> recover from `/Users/gerold/Work/Repositories/tarantool/test/var/032_box/box/00000000000000002296.xlog'
[032] 2019-08-22 22:14:42.711 [38110] relay/unix/:(socket)/101/main coio.cc:379 !> SystemError unexpected EOF when reading from socket, called on fd 43, aka unix/:/Users/gerold/Work/Repositories/tarantool/test: Broken pipe
[032] 2019-08-22 22:14:42.712 [38110] relay/unix/:(socket)/101/main C> exiting the relay loop
[032] 2019-08-22 22:14:42.774 [38110] main I> removed replica eb8f7d4a-0e37-4a9f-940d-eb57756314c8
[032] Assertion failed: (region_used(region) >= used), function region_truncate, file /Users/gerold/Work/Repositories/tarantool/src/lib/small/small/region.c, line 75.
[032] [ fail ]

Failing tests: engine/ddl, box/func_reload, app/fiber_channel, and others.

[Totktonada]

I guess a build with -DLUAJIT_ENABLE_GC64=ON on Linux will show the similar behaviour. See tarantool/tarantool#2643.

[mraleph]

I would guess this is the same issue as tarantool/tarantool#4427

[Totktonada]

@Gerold103 Is it reproduced after the tarantool/tarantool#4427 fix?

[Gerold103]

This one I didn't see for a long time. But sometimes I see mp_tuple_assert() failing. I didn't find time to properly file an issue for that yet.

[Totktonada]

mp_tuple_assert() looks similar to #234.

[igormunkin]

@Totktonada, @Gerold103, we have recenly applied the patch with the fix for #234 and #235. Could you please check, whether this issue is also gone if you consider it related to any of the mentioned above?

[Totktonada]

I performed a couple of experiments on Linux with GC64.

Experiment

Build:

$ cmake .                        \
    -DCMAKE_BUILD_TYPE=Debug     \
    -DENABLE_BACKTRACE=ON        \
    -DENABLE_DIST=ON             \
    -DENABLE_FEEDBACK_DAEMON=OFF \
    -DENABLE_BUNDLED_LIBCURL=OFF \
    -DENABLE_FEEDBACK_DAEMON=OFF \
    -DLUAJIT_ENABLE_GC64=ON      \
    && make -j

Verify build:

# tarantool> require('ffi').abi('gc64')
---
- true
...

Run the test many times in parallel:

$ ./test/test-run.py $(yes box/func_reload.test.lua | head -n 1000)

Results

I checked current master (2.11.0-entrypoint-205-g1cd1a2df6) and the commit right before tarantool/tarantool#7328 (2.11.0-entrypoint-177-ga85629a69, the pull request lands as 2.11.0-entrypoint-178-g34330b159).

Got no fails.

Can't add more here.

[Gerold103]

I didn't see any crashes for a long time.

[igormunkin]

Nice. Closing then. Feel free to reopen if this starts bothering anybody again.