Skip to content

Starting to add resource metrics on runs of the tool. #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 9, 2023

Conversation

fosterbrereton
Copy link
Contributor

@fosterbrereton fosterbrereton commented Jan 9, 2023

We are looking to improve our understanding of how ORC utilizes the machine resources during processing, possibly in order to identify tradeoffs we can make in space v. time. This PR adds some initial memory footprint metrics to the end of the ORC run's output, along with a flag to enable it.

Here's an example output:

Resource metrics:
  String pool size / waste:
    0: 32 MiB (33554432) / 2.19 MiB (2292587) / 6.83%
    1: 32 MiB (33554432) / 2.21 MiB (2316317) / 6.90%
    2: 32 MiB (33554432) / 2.20 MiB (2310493) / 6.89%
    3: 32 MiB (33554432) / 2.31 MiB (2420043) / 7.21%
    4: 32 MiB (33554432) / 2.41 MiB (2527218) / 7.53%
    5: 32 MiB (33554432) / 2.13 MiB (2231512) / 6.65%
    6: 32 MiB (33554432) / 2.19 MiB (2292399) / 6.83%
    7: 32 MiB (33554432) / 2.16 MiB (2269894) / 6.76%
    8: 32 MiB (33554432) / 2.13 MiB (2237163) / 6.67%
    9: 32 MiB (33554432) / 2.26 MiB (2370840) / 7.07%
    10: 32 MiB (33554432) / 2.30 MiB (2411155) / 7.19%
    11: 32 MiB (33554432) / 2.32 MiB (2429004) / 7.24%
    12: 32 MiB (33554432) / 2.25 MiB (2355728) / 7.02%
    13: 32 MiB (33554432) / 2.42 MiB (2537523) / 7.56%
    14: 32 MiB (33554432) / 2.30 MiB (2408363) / 7.18%
    15: 32 MiB (33554432) / 2.30 MiB (2409418) / 7.18%
    16: 32 MiB (33554432) / 2.23 MiB (2337468) / 6.97%
    17: 32 MiB (33554432) / 2.43 MiB (2544395) / 7.58%
    18: 32 MiB (33554432) / 2.54 MiB (2664409) / 7.94%
    19: 32 MiB (33554432) / 2.15 MiB (2258360) / 6.73%
    20: 32 MiB (33554432) / 2.40 MiB (2514859) / 7.49%
    21: 32 MiB (33554432) / 2.35 MiB (2464329) / 7.34%
    22: 32 MiB (33554432) / 2.30 MiB (2411696) / 7.19%
    totals: 736 MiB (771751936) / 52.47 MiB (55015173) / 7.13%
  die footprint: 5.31 GiB (5702569584) 

Copy link
Contributor

@leethomason leethomason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Nice use of std::array as well.

@fosterbrereton fosterbrereton merged commit 34f16b0 into main Jan 9, 2023
@fosterbrereton fosterbrereton deleted the fosterbrereton/string-pool-reportings branch January 9, 2023 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants