Skip to content

Add default server restart to each test run #309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 21, 2021

Conversation

avtikhon
Copy link
Contributor

@avtikhon avtikhon commented Jun 10, 2021

Patchset of 3 patches:

  1. Found that the root cause of the issues happened with vinyl tests were
    backside effects of the not correct test 'vinyl/gh.test.lua' which
    leaved Tarantool worker process in inconsistent state. After it any
    other next test on the same Tarantool worker process could fail on
    running testings with snapshots calls, like test: flaky vinyl/bloom.test.lua fail because of previous vinyl/gh.test.lua on Snapshot is already in progress tarantool-qa#126:

      error: Snapshot is already in progress
    

    Either restarting Tarantool worker process could fail on stopping it,
    like ./test-run --gdb is broken in 1.6 tarantool#261 and test: flaky vinyl/gh.test.lua test tarantool-qa#205:

     E> failed to process vylog record: delete_slice{slice_id=115, }
     E> ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Slice 115 deleted but not registered
    

    Decided to avoid of such situations for now and in the future all tests
    must be run after Tarantool worker default server process was restarted.
    This patch moves this server restart call from check if the test failed
    to the common part of the tests run loop.

    Part of resources: disk bound tests need to run in special way tarantool-qa#97
    Fixes Problems with installing tarantool-modules on master tarantool#260
    Fixes ./test-run --gdb is broken in 1.6 tarantool#261
    Needed for test: flaky vinyl/deferred_delete.test.lua tarantool#5089
    Closes test: flaky vinyl/bloom.test.lua fail because of previous vinyl/gh.test.lua on Snapshot is already in progress tarantool-qa#126

  2. After the previous commit in the current patchset we don't use the same
    tarantool server process from test to test and we don't need to use any
    scripts like 'pretest_clean.lua' to cleanup it between tests. So this
    patch removes this script.

  3. Found that all currently used suite.ini files use this option enabled.
    Also there is no need to have any suite of tests where snap either
    log files whould be used by the next test from the previous, due to
    such test schema can be easily implemented within single test file.

@avtikhon avtikhon requested a review from Totktonada June 10, 2021 16:16
@avtikhon avtikhon self-assigned this Jun 10, 2021
@avtikhon avtikhon added the teamQ label Jun 10, 2021
@avtikhon avtikhon force-pushed the avtikhon/restart-tests branch 2 times, most recently from 501d14f to fde42a6 Compare June 12, 2021 19:05
@Totktonada
Copy link
Member

Needed for tarantool/tarantol#5089

Typo: tarantol -> tarantool.

Found that the root cause of the issues happened with vinyl tests were
backside effects of the not correct test 'vinyl/gh.test.lua' which
leaved Tarantool worker process in inconsistent state. After it any
other next test on the same Tarantool worker process could fail on
running testings with snapshots calls, like tarantool/tarantool-qa#126:

  error: Snapshot is already in progress

Either restarting Tarantool worker process could fail on stopping it,
like #261 and tarantool/tarantool#5141:

  E> failed to process vylog record: delete_slice{slice_id=115, }
  E> ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Slice 115 deleted but not registered

Decided to avoid of such situations for now and in the future all tests
must be run after Tarantool worker default server process was restarted.
This patch moves this server restart call from check if the test failed
to the common part of the tests run loop.

Fixes #260
Fixes #261
Needed for tarantool/tarantool#5089
Closes tarantool/tarantool-qa#126
@avtikhon avtikhon force-pushed the avtikhon/restart-tests branch from 4a082f9 to 650dd6f Compare June 18, 2021 05:18
@avtikhon avtikhon force-pushed the avtikhon/restart-tests branch from 650dd6f to fd3e5f2 Compare June 18, 2021 13:10
@avtikhon avtikhon requested a review from Totktonada June 18, 2021 13:13
@Totktonada
Copy link
Member

Totktonada commented Jun 18, 2021

Please, look at the comment in TarantoolServer.pretest_clean(). It becomes non-actual.

In fact, xlog/snap are removed between tests, but in very non-obvious way (and it does not depend on the pretest_clean suite.ini option anyhow). Please, at least update the comment. Maybe it also worth to drop the option at all and always perform clean up between tests.

Actually the following code removes xlog/snap:

color_log(' | Found old workdir, deleting...\n')
self.kill_old_server()
self.cleanup()

I propose to spin around the code for a hour and decide about some 'good enough' amount of changes we'll do now (postpone everything else a bit).

@avtikhon avtikhon force-pushed the avtikhon/restart-tests branch 2 times, most recently from b5d9279 to a9ecfa5 Compare June 18, 2021 16:46
avtikhon added 2 commits June 18, 2021 19:07
After the previous commit in the current patchset we don't use the same
tarantool server process from test to test and we don't need to use any
scripts like 'pretest_clean.lua' to cleanup it between tests. So this
patch removes this script.
Found that all currently used suite.ini files use this option enabled.
Also there is no need to have any suite of tests where snap either
log files whould be used by the next test from the previous, due to
such test schema can be easily implemented within single test file.
@avtikhon avtikhon force-pushed the avtikhon/restart-tests branch from a9ecfa5 to 5c6417f Compare June 18, 2021 19:07
@avtikhon
Copy link
Contributor Author

Please, look at the comment in TarantoolServer.pretest_clean(). It becomes non-actual.

In fact, xlog/snap are removed between tests, but in very non-obvious way (and it does not depend on the pretest_clean suite.ini option anyhow). Please, at least update the comment. Maybe it also worth to drop the option at all and always perform clean up between tests.

Actually the following code removes xlog/snap:

color_log(' | Found old workdir, deleting...\n')
self.kill_old_server()
self.cleanup()

I propose to spin around the code for a hour and decide about some 'good enough' amount of changes we'll do now (postpone everything else a bit).

I've tried with and w/o clean and saw that this function is still needed. Also checked that it calls on tests starts with 'core = tarantool' suite. Found that there is no any test suites suite.ini file that use this option. Also there is no need to have any suite of tests where snap either log files whould be used by the next test from the previous, due to such test schema can be easily implemented within single test file. It means that this option 'pretest_clean' is a dead code and not needed at all. So I've added tiny patch to remove it.

@Totktonada
Copy link
Member

To be honest, I'm not very happy about the code: it is more complex then it would be. However, it seems, everything works as expected, so it looks as a way to go.

@Totktonada Totktonada merged commit 3aa4666 into master Jun 21, 2021
@Totktonada Totktonada deleted the avtikhon/restart-tests branch June 21, 2021 19:01
Totktonada added a commit to tarantool/tarantool that referenced this pull request Jun 21, 2021
This update offers two changes:

* More intelligent error reporting for non UTF-8 TAP13 output ([1],
  [2]).
* Restart tarantool server before each test (to avoid unexpected and
  non-obvious dependencies between tests, see [3]).

Removed the pretest_clean suite.ini option: test-run does not read it
anymore.

[1]: tarantool/test-run#293
[2]: tarantool/test-run#297
[3]: tarantool/test-run#309
Totktonada added a commit to tarantool/tarantool that referenced this pull request Jun 21, 2021
This update offers two changes:

* More intelligent error reporting for non UTF-8 TAP13 output ([1],
  [2]).
* Restart tarantool server before each test (to avoid unexpected and
  non-obvious dependencies between tests, see [3]).

Removed the pretest_clean suite.ini option: test-run does not read it
anymore.

[1]: tarantool/test-run#293
[2]: tarantool/test-run#297
[3]: tarantool/test-run#309

(cherry picked from commit 7fa70b6)
Totktonada added a commit to tarantool/tarantool that referenced this pull request Jun 21, 2021
This update offers two changes:

* More intelligent error reporting for non UTF-8 TAP13 output ([1],
  [2]).
* Restart tarantool server before each test (to avoid unexpected and
  non-obvious dependencies between tests, see [3]).

Removed the pretest_clean suite.ini option: test-run does not read it
anymore.

[1]: tarantool/test-run#293
[2]: tarantool/test-run#297
[3]: tarantool/test-run#309

(cherry picked from commit 7fa70b6)
Totktonada added a commit to tarantool/tarantool that referenced this pull request Jun 21, 2021
This update offers two changes:

* More intelligent error reporting for non UTF-8 TAP13 output ([1],
  [2]).
* Restart tarantool server before each test (to avoid unexpected and
  non-obvious dependencies between tests, see [3]).

Removed the pretest_clean suite.ini option: test-run does not read it
anymore.

[1]: tarantool/test-run#293
[2]: tarantool/test-run#297
[3]: tarantool/test-run#309

(cherry picked from commit 7fa70b6)
@Totktonada
Copy link
Member

Updated the test-run submodule in tarantool in 2.9.0-104-g7fa70b661, 2.8.1-75-g029d66355, 2.7.2-62-g6f9ccad83, 1.10.10-25-g9b4acec8f.

ylobankov added a commit that referenced this pull request Aug 1, 2023
Now it is an irrelevant feature for us because we restart Tarantool
server before each test [1]. So the order of tests shouldn't affect
anything.

This patch just disables reproduce content printing. It is planned
to drop the related code later.

[1] #309
ylobankov added a commit that referenced this pull request Aug 1, 2023
Now it is an irrelevant feature for us because we restart Tarantool
server before each test [1]. So the order of tests shouldn't affect
anything.

This patch just disables reproduce content printing. It is planned
to drop the related code later.

[1] #309
ylobankov added a commit that referenced this pull request Aug 1, 2023
Now it is an irrelevant feature for us because we restart Tarantool
server before each test [1]. So the order of tests shouldn't affect
anything.

This patch just disables reproduce content printing. It is planned
to drop the related code later.

[1] #309
ylobankov added a commit that referenced this pull request Aug 1, 2023
Now it is an irrelevant feature for us because we restart Tarantool
server before each test [1]. So the order of tests shouldn't affect
anything.

This patch just disables reproduce content printing. It is planned
to drop the related code later.

[1] #309
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants