Allow logs to be redirected to /dev/null

[msievers]

What does this PR do?

The documentation of the APM agent states that the logging output can be redirected to a file. Currently, this does not work when specifying /dev/null to silence the APM logs completely. When trying to do so, the agent complains with

Log file /dev/null is not writable. Falling back to System.out.

Checklist

• My code follows the style guidelines of this project
• I have rebased my changes on top of the latest master branch

• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

• I have updated CHANGELOG.asciidoc
• I have updated supported-technologies.asciidoc
• Added an API method or config option? Document in which version this will be introduced
• Added an instrumentation plugin? How did you make sure that old, non-supported versions are not instrumented by accident?

Author's Checklist

dir exists	dir writeable	file exists	file writeable	action
false	-	-	-	Create directory and file
true	false	false	-	Print error, fallback to STDOUT
true	false	true	false	Print error, fallback to STDOUT
true	false	true	true	Use given file (e.g. /dev/null)
true	true	false	-	Create new file
true	true	true	false	Print error, fallback to STDOUT
true	true	true	true	Use given file

Related issues

• Logging implementation is fixed #550
• Implement log rotation  #442 (comment)

Use cases

There are companies that have very strict requirements regarding application logs/log formats which may prevent the usage of Elastic APM agent, because there is currently no way, to configure/alter the log format, for example to output JSON.

As a workaround, it should be possible to silence the APM agent by redirecting its logs to /dev/null.

Screenshots

[cla-checker-service]

💚 CLA has been signed

[msievers]

Although this isn't very elegant, it seems to be the only way to retain the actual behavior while making the agent completely silent in case of /dev/null.

[felixbarny]

There are companies that have very strict requirements regarding application logs/log formats which may prevent the usage of Elastic APM agent, because there is currently no way, to configure/alter the log format, for example to output JSON.

You might be happy to hear that we're experimenting with some things regarding logging here.
Firstly, we make it possible to set the log format to JSON which uses https://github.com/elastic/ecs-logging-java to format the logs.
Additionally, the JSON logs can be automatically shipped to APM Server via the agent. It's of course also possible to do this manually by shipping the log with Filebeat, for example, or to log JSON to std out which can be useful in a container-based setup.

Out of curiosity, why can't just log to a file and ignore it? That way you at least have the option to manually look at the file if you need to.

[msievers]

We tried with logging to a file first, but for instance, in case of errors, the agent logs a huge amount of messages. So the fear is, that this may silently flood our storages.

Actually, the problem why it refuses to log to /dev/null is simply the writeable check on the parent directory, which for instance would also prevent you to log into a file you are allowed to write to, but if the directory is for instance not writeable.

We are currently facing the problem, that our OPs department want's us to shut down all agents if we don't get the logging right or silence to agent at all. That's why I would really appreciate it if we could allow logging to /dev/null as a workaround until some more sophisticated solution is available.

[msievers]

Btw, I just found a combination of directory/file permission where the MR fails, so I will push a fix together with a list of tested combinations (dir exists/writeable/file exists/writeable within the next hours).

[felixbarny]

We tried with logging to a file first, but for instance, in case of errors, the agent logs a huge amount of messages. So the fear is, that this may silently flood our storages.

This is also something we're tackling. You'll be able to configure a max log file size.

In the meantime, I think your fix can't do harm.

[msievers]

Nice, thank you 🎉 I'll ping you when I finished testing all possible combinations and updated the pull requests description.

[msievers]

@felixbarny I just double-checked all possible combinations of directory/file existence/writeability and wrote them down in the pull requests description under Author's Checklist.

[eyalkoren]

@msievers Thanks! Your permissions-result matrix in the Author's checklist is 👍
I did a tests PR, but wait for the conclusion of the following discussion before merging.

@felixbarny I think this wouldn't work with the new logging we are about to add - since we are going to use rolling logs and the state file, the directory probably needs to be writeable.
It may make it a breaking change to @msievers when we migrate to it.
Nevertheless, we need to offer a way to make the agent silent, maybe through configuration?

[eyalkoren]

@msievers we discussed that - while your solution works great now, we are soon to introduce a very useful enhancement to the agent logging that will rely on the directory being writeable.

However, what you really need is a silent agent, so let's do that 🙂. We thought of adding an OFF option to the log_level config to facilitate that. Would you like to do that?

[msievers]

@eyalkoren Sure, I can give it a try. I think I can report back by the end of the week 👍

[msievers]

@eyalkoren I couldn't keep my fingers from this and created a new PR to add OFF as log level. I will close this PR in favour of #1160