Collect statistics for 'vardisk' disk usage #308
Assignees
Comments
2 tasks
avtikhon
added a commit
that referenced
this issue
May 31, 2021
Added for 'vardir' path disk disk utilization collecting routine which
parses file:
/proc/diskstats
for the value milliseconds spent doing I/Os from the given device
and counts 'busy' value in percents, like iostat tool does for its
'%util' field. Check for more information linux kernel documentation
[1]. We use from the 'diskstats' file Field 10 which is in real at
the 12th position:
Field 9 -- # of I/Os currently in progress
The only field that should go to zero. Incremented as requests are
given to appropriate struct request_queue and decremented as they
finish.
Field 10 -- # of milliseconds spent doing I/Os
This field increases so long as field 9 is nonzero.
Decided to collect 'vardir' disk utilization per each run test in
standalone file and print to stdout after testing completed, up to
10 most biggest of it.
To count 'vardir' disk utilization for each test we need to know what
was the value of the disk bound on the test start and time when test
started to count later the utilization. On test finish we check the
test finish time and current disk bound and count disk utilization.
All of these checks and counts implemented as the new routine
'get_disk_bound_stat_busy()' in 'lib/utils.py' file.
To collect disk bounds and durations on test start/finish events used
'StatisticsWatcher' listener which has the following usable routines
for it:
process_result()
Using 'WorkerCurrentTask' queue message apperance to save the disk
bounds values at the start of the tests.
Using 'WorkerTaskResult' queue message appearance to save final
disk bounds values for finished tests.
print_statistics() - statistics printing to stdout after testing.
Prints disk bounds for failed tests and up to 10 most used it tests.
We use standalone 'statistics' directory in 'vardir' path to save
'durations.log' file in it with disk bounds for each tested task
in format:
<test task name> <disk bound>
Part of tarantool/tarantool-qa#97
Closes #308
[1]: https://www.kernel.org/doc/Documentation/iostats.txt
avtikhon
added a commit
that referenced
this issue
Jun 2, 2021
Added for 'vardir' path disk disk utilization collecting routine which
parses file:
/proc/diskstats
for the value milliseconds spent doing I/Os from the given device
and counts 'busy' value in percents, like iostat tool does for its
'%util' field. Check for more information linux kernel documentation
[1]. We use from the 'diskstats' file Field 10 which is in real at
the 12th position:
Field 9 -- # of I/Os currently in progress
The only field that should go to zero. Incremented as requests are
given to appropriate struct request_queue and decremented as they
finish.
Field 10 -- # of milliseconds spent doing I/Os
This field increases so long as field 9 is nonzero.
Decided to collect 'vardir' disk utilization per each run test in
standalone file and print to stdout after testing completed, up to
10 most biggest of it.
To count 'vardir' disk utilization for each test we need to know what
was the value of the disk bound on the test start and time when test
started to count later the utilization. On test finish we check the
test finish time and current disk bound and count disk utilization.
All of these checks and counts implemented as the new routine
'get_disk_bound_stat_busy()' in 'lib/utils.py' file.
To collect disk bounds and durations on test start/finish events used
'StatisticsWatcher' listener which has the following usable routines
for it:
process_result()
Using 'WorkerCurrentTask' queue message apperance to save the disk
bounds values at the start of the tests.
Using 'WorkerTaskResult' queue message appearance to save final
disk bounds values for finished tests.
print_statistics() - statistics printing to stdout after testing.
Prints disk bounds for failed tests and up to 10 most used it tests.
We use standalone 'statistics' directory in 'vardir' path to save
'durations.log' file in it with disk bounds for each tested task
in format:
<test task name> <disk bound>
Part of tarantool/tarantool-qa#97
Closes #308
[1]: https://www.kernel.org/doc/Documentation/iostats.txt
avtikhon
added a commit
that referenced
this issue
Jun 2, 2021
Added for 'vardir' path disk disk utilization collecting routine which
parses file:
/proc/diskstats
for the value milliseconds spent doing I/Os from the given device
and counts 'busy' value in percents, like iostat tool does for its
'%util' field. Check for more information linux kernel documentation
[1]. We use from the 'diskstats' file Field 10 which is in real at
the 12th position:
Field 9 -- # of I/Os currently in progress
The only field that should go to zero. Incremented as requests are
given to appropriate struct request_queue and decremented as they
finish.
Field 10 -- # of milliseconds spent doing I/Os
This field increases so long as field 9 is nonzero.
Decided to collect 'vardir' disk utilization per each run test in
standalone file and print to stdout after testing completed, up to
10 most biggest of it.
To count 'vardir' disk utilization for each test we need to know what
was the value of the disk bound on the test start and time when test
started to count later the utilization. On test finish we check the
test finish time and current disk bound and count disk utilization.
All of these checks and counts implemented as the new routine
'get_disk_bound_stat_busy()' in 'lib/utils.py' file.
To collect disk bounds and durations on test start/finish events used
'StatisticsWatcher' listener which has the following usable routines
for it:
process_result()
Using 'WorkerCurrentTask' queue message apperance to save the disk
bounds values at the start of the tests.
Using 'WorkerTaskResult' queue message appearance to save final
disk bounds values for finished tests.
print_statistics() - statistics printing to stdout after testing.
Prints disk bounds for failed tests and up to 10 most used it tests.
We use standalone 'statistics' directory in 'vardir' path to save
'durations.log' file in it with disk bounds for each tested task
in format:
<test task name> <disk bound>
Part of tarantool/tarantool-qa#97
Closes #308
[1]: https://www.kernel.org/doc/Documentation/iostats.txt
avtikhon
added a commit
that referenced
this issue
Jun 2, 2021
Added for 'vardir' path disk disk utilization collecting routine which
parses file:
/proc/diskstats
for the value milliseconds spent doing I/Os from the given device
and counts 'busy' value in percents, like iostat tool does for its
'%util' field. Check for more information linux kernel documentation
[1]. We use from the 'diskstats' file Field 10 which is in real at
the 12th position:
Field 9 -- # of I/Os currently in progress
The only field that should go to zero. Incremented as requests are
given to appropriate struct request_queue and decremented as they
finish.
Field 10 -- # of milliseconds spent doing I/Os
This field increases so long as field 9 is nonzero.
Decided to collect 'vardir' disk utilization per each run test in
standalone file and print to stdout after testing completed, up to
10 most biggest of it.
To count 'vardir' disk utilization for each test we need to know what
was the value of the disk bound on the test start and time when test
started to count later the utilization. On test finish we check the
test finish time and current disk bound and count disk utilization.
All of these checks and counts implemented as the new routine
'get_disk_bound_stat_busy()' in 'lib/utils.py' file.
To collect disk bounds and durations on test start/finish events used
'StatisticsWatcher' listener which has the following usable routines
for it:
process_result()
Using 'WorkerCurrentTask' queue message apperance to save the disk
bounds values at the start of the tests.
Using 'WorkerTaskResult' queue message appearance to save final
disk bounds values for finished tests.
print_statistics() - statistics printing to stdout after testing.
Prints disk bounds for failed tests and up to 10 most used it tests.
We use standalone 'statistics' directory in 'vardir' path to save
'durations.log' file in it with disk bounds for each tested task
in format:
<test task name> <disk bound>
Part of tarantool/tarantool-qa#97
Closes #308
[1]: https://www.kernel.org/doc/Documentation/iostats.txt
avtikhon
added a commit
that referenced
this issue
Jun 2, 2021
Added for 'vardir' path disk disk utilization collecting routine which
parses file:
/proc/diskstats
for the value milliseconds spent doing I/Os from the given device
and counts 'busy' value in percents, like iostat tool does for its
'%util' field. Check for more information linux kernel documentation
[1]. We use from the 'diskstats' file Field 10 which is in real at
the 12th position:
Field 9 -- # of I/Os currently in progress
The only field that should go to zero. Incremented as requests are
given to appropriate struct request_queue and decremented as they
finish.
Field 10 -- # of milliseconds spent doing I/Os
This field increases so long as field 9 is nonzero.
Decided to collect 'vardir' disk utilization per each run test in
standalone file and print to stdout after testing completed, up to
10 most biggest of it.
To count 'vardir' disk utilization for each test we need to know what
was the value of the disk bound on the test start and time when test
started to count later the utilization. On test finish we check the
test finish time and current disk bound and count disk utilization.
All of these checks and counts implemented as the new routine
'get_disk_bound_stat_busy()' in 'lib/utils.py' file.
To collect disk bounds and durations on test start/finish events used
'StatisticsWatcher' listener which has the following usable routines
for it:
process_result()
Using 'WorkerCurrentTask' queue message apperance to save the disk
bounds values at the start of the tests.
Using 'WorkerTaskResult' queue message appearance to save final
disk bounds values for finished tests.
print_statistics() - statistics printing to stdout after testing.
Prints disk bounds for failed tests and up to 10 most used it tests.
We use standalone 'statistics' directory in 'vardir' path to save
'durations.log' file in it with disk bounds for each tested task
in format:
<test task name> <disk bound>
Part of tarantool/tarantool-qa#97
Closes #308
[1]: https://www.kernel.org/doc/Documentation/iostats.txt
ochaplashkin
pushed a commit
to ochaplashkin/test-run
that referenced
this issue
Sep 18, 2023
By default, test-run captures the stderr stream and redirects its to the
log file:
$ cat /tmp/t/001_foo-luatest/foo_test.log:
[001] Some error log
[001] My warning log
...
Use the new option `--show-capture` (abbr. `-c`) to redirect stderr to
stdout. Use it instead of the deprecated `--verbose` option.
The `--verbose` option will be ignored and output:
Argument ['--verbose'] is deprecated and is ignored.
Close tarantool#308
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To resolve the issues in tests with disks overload decided initially to implement the disk usage statistics.
Part of tarantool/tarantool-qa#97
Working on this issue and after creating pull request with solution found that the current statistic is really good and fine for HDD based hosts. But for SSD this statistics doesn't show anything countable except that any of SSD parallel RW channels were in use, but not all, while SSD may have it and 16 and 64 and use only some of it doesn't feed and overload its use. We use in Github Actions and in MCS only hosts with SSD which are really fast and such patch won't help us any kind. Decided to stop this work till it will be needed.
The text was updated successfully, but these errors were encountered: