b/390177544 cart: Support dynamic mercury/libfabric loglevels #15738
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Errors are Ticket number prefix incorrect,PR title is malformatted. See https://daosio.atlassian.net/wiki/spaces/DC/pages/11133911069/Commit+Comments,Unable to load ticket data |
|
Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15738/1/testReport/ |
jolivier23
previously approved these changes
Jan 16, 2025
Mercury and libfabric are much too spammy to enable at lower than warning level by default. Mercury does support adjustment of its log level and logged subsystems at runtime, so this patch provides access to that functionality from the existing `dmg server set-logmasks` tool. Change-Id: Ib04594dde5a7b3f50cce4176759edf1f109c0947 Signed-off-by: Michael MacDonald <mjmac@google.com>
mjmac
force-pushed
the
dev/mjmac/b/390177544
branch
from
49b253b to
1c33868
Compare
jolivier23
approved these changes
Jan 16, 2025
jolivier23
pushed a commit
that referenced
this pull request
Jan 23, 2025
mjmac
added a commit
that referenced
this pull request
Mar 7, 2025
Mercury and libfabric are much too spammy to enable at lower than warning level by default. Mercury does support adjustment of its log level and logged subsystems at runtime, so this patch provides access to that functionality from the existing `dmg server set-logmasks` tool. This version of the patch adds support for writing mercury/libfabric debug logs to a separate file, in order to avoid undue burden on the cloud logging pipeline. To use it, set HG_DBG_LOG_FILE in the engine config to a logfile path. NB: This is a "raw" log file, i.e. no rotation is done and it's flushed once per second. To enable debug logging for just the mercury/libfabric libraries: dmg server set-logmasks -m EXTERNAL=DEBUG To disable debug logging again: dmg server set-logmasks -m EXTERNAL=ERR Note that the CaRT/Mercury wrapper (crt_hg.c) still uses the normal dlog logger with the "hg" facility. If debug logging in this file is desired, then the mask MERCURY=DEBUG may be used for set-logmasks. Be aware that the debug logging produced by this file is rather copious and may cause problems for cloud logging, however! Change-Id: Ib6fb74c42507dbfba2c5984a92433ad43b6e2df3 Signed-off-by: Michael MacDonald <mjmac@google.com>
mjmac
added a commit
that referenced
this pull request
Mar 9, 2025
Mercury and libfabric are much too spammy to enable at lower than warning level by default. Mercury does support adjustment of its log level and logged subsystems at runtime, so this patch provides access to that functionality from the existing `dmg server set-logmasks` tool. This version of the patch adds support for writing mercury/libfabric debug logs to a separate file, in order to avoid undue burden on the cloud logging pipeline. To use it, set HG_DBG_LOG_FILE in the engine config to a logfile path. NB: This is a "raw" log file, i.e. no rotation is done and it's flushed once per second. To enable debug logging for just the mercury/libfabric libraries: dmg server set-logmasks -m EXTERNAL=DEBUG To disable debug logging again: dmg server set-logmasks -m EXTERNAL=ERR Note that the CaRT/Mercury wrapper (crt_hg.c) still uses the normal dlog logger with the "hg" facility. If debug logging in this file is desired, then the mask HG=DEBUG may be used for set-logmasks. Be aware that the debug logging produced by this file is rather copious and may cause problems for cloud logging, however! Change-Id: Ib6fb74c42507dbfba2c5984a92433ad43b6e2df3 Signed-off-by: Michael MacDonald <mjmac@google.com>
mjmac
added a commit
that referenced
this pull request
Mar 9, 2025
Mercury and libfabric are much too spammy to enable at lower than warning level by default. Mercury does support adjustment of its log level and logged subsystems at runtime, so this patch provides access to that functionality from the existing `dmg server set-logmasks` tool. This version of the patch adds support for writing mercury/libfabric debug logs to a separate file, in order to avoid undue burden on the cloud logging pipeline. To use it, set HG_DBG_LOG_FILE in the engine config to a logfile path. NB: This is a "raw" log file, i.e. no rotation is done and it's flushed once per second. To enable debug logging for just the mercury/libfabric libraries: dmg server set-logmasks -m EXTERNAL=DEBUG To disable debug logging again: dmg server set-logmasks -m EXTERNAL=ERR Note that the CaRT/Mercury wrapper (crt_hg.c) still uses the normal dlog logger with the "hg" facility. If debug logging in this file is desired, then the mask HG=DEBUG may be used for set-logmasks. Be aware that the debug logging produced by this file is rather copious and may cause problems for cloud logging, however! Change-Id: Ib6fb74c42507dbfba2c5984a92433ad43b6e2df3 Signed-off-by: Michael MacDonald <mjmac@google.com>
mjmac
added a commit
that referenced
this pull request
Mar 11, 2025
Mercury and libfabric are much too spammy to enable at lower than warning level by default. Mercury does support adjustment of its log level and logged subsystems at runtime, so this patch provides access to that functionality from the existing `dmg server set-logmasks` tool. This version of the patch adds support for writing DEBUG logs to a separate file in order to avoid problems with log ingestion pipelines that may be overwhelmed by the volume of DEBUG-level logging. To enable a standalone debug log, set the environment variable D_LOG_DEBUG_FILE to a path on disk where debug logs will be written. The log size and rotation behavior set for the main log file will be also applied to the debug log. To enable debug logging for just the mercury/libfabric libraries: dmg server set-logmasks -m EXTERNAL=DEBUG To disable debug logging again: dmg server set-logmasks -m EXTERNAL=ERR Change-Id: I667ed105545f3da717cb79ca07ae904673121d4b Signed-off-by: Michael MacDonald <mjmac@google.com>
mjmac
added a commit
that referenced
this pull request
Mar 12, 2025
Mercury and libfabric are much too spammy to enable at lower than warning level by default. Mercury does support adjustment of its log level and logged subsystems at runtime, so this patch provides access to that functionality from the existing `dmg server set-logmasks` tool. This version of the patch adds support for writing DEBUG logs to a separate file in order to avoid problems with log ingestion pipelines that may be overwhelmed by the volume of DEBUG-level logging. To enable a standalone debug log, set the environment variable D_LOG_DEBUG_FILE to a path on disk where debug logs will be written. The log size and rotation behavior set for the main log file will be also applied to the debug log. To enable debug logging for just the mercury/libfabric libraries: dmg server set-logmasks -m EXTERNAL=DEBUG To disable debug logging again: dmg server set-logmasks -m EXTERNAL=ERR Signed-off-by: Michael MacDonald <mjmac@google.com>
jolivier23
pushed a commit
that referenced
this pull request
Mar 12, 2025
#16077) * b/390177544 cart: Support dynamic mercury/libfabric loglevels (#15738) Mercury and libfabric are much too spammy to enable at lower than warning level by default. Mercury does support adjustment of its log level and logged subsystems at runtime, so this patch provides access to that functionality from the existing `dmg server set-logmasks` tool. This version of the patch adds support for writing DEBUG logs to a separate file in order to avoid problems with log ingestion pipelines that may be overwhelmed by the volume of DEBUG-level logging. To enable a standalone debug log, set the environment variable D_LOG_DEBUG_FILE to a path on disk where debug logs will be written. The log size and rotation behavior set for the main log file will be also applied to the debug log. To enable debug logging for just the mercury/libfabric libraries: dmg server set-logmasks -m EXTERNAL=DEBUG To disable debug logging again: dmg server set-logmasks -m EXTERNAL=ERR Change-Id: I667ed105545f3da717cb79ca07ae904673121d4b Signed-off-by: Michael MacDonald <mjmac@google.com>
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Mercury and libfabric are much too spammy to enable at lower
than warning level by default. Mercury does support adjustment
of its log level and logged subsystems at runtime, so this
patch provides access to that functionality from the
existing
dmg server set-logmaskstool.Change-Id: Ib04594dde5a7b3f50cce4176759edf1f109c0947
Signed-off-by: Michael MacDonald mjmac@google.com