Logging implementation is fixed

[OrangeDog]

This project uses slf4j for logging, but appears to fix the implementation to SimpleLogger instead of what the host application has on its classpath.

This largely defeats the point of using slf4j, and severely reduces the ability to configure logging.

[felixbarny]

The problem is that the Java agent is not like any other library. It contains all the code, including dependencies and a logging implementation. The reason for that is that it runs inside the system/bootstrap class loader and we can't assume the instrumented application actually uses slf4j. Even if slf4j is used, it could be a different version, causing version conflicts. We also shade (i.e. relocate) slf4j from org.slf4j to co.elastic.apm.shaded.slf4j to avoid that these classes conflict with the classes in your application and also to avoid them being present on the classpath twice.

I can understand that it's a bit annoying to not be able to configure logging in the usual way but I'm afraid there's no real way around that.

However, we do have plans to offer more options to configure the logging. See also #442. What options are you currently missing and how would you like to configure the logging?

[OrangeDog]

I want to send the logs via https://github.com/logstash/logstash-logback-encoder, customise the log format, and manage loggers over JMX.

Since it's a classloader agent anyway, can you not detect the presence of SLF4J bindings and only add SimpleLogger if there isn't one?
Alternatively, can you provide an alternate artefact that relies on the classpath entirely for logging?

I'm using the ElasticApmAttacher in a Spring Boot application, tied to the ApplicationEnvironmentPreparedEvent, so I can load the agent properties via the Spring environment.

[felixbarny]

That's currently not easily possible as the ElasticApmAttacher just attaches the regular java agent which contains all of the shaded dependencies, including the shaded slf4j. But in theory, it's something that is possible. I'll give it some thought.

[OrangeDog]

Note you do already have a dependency on the application's own slf4j, for enable_log_correlation.

[felixbarny]

Yes, but that does not have a direct dependency but rather uses MethodHandles for decoupling. But maybe we could do logging integration with an application's own slf4j in a similar way 🤔.

[MrThreepwood]

Just wanted to add that I'm also running into this issue. I don't see any options for logging in JSON (as exists in other agents), which is a bit awkward as without that we can't actually view it's logs correctly in Kibana. We are loading our logs into our APM server and the only logs that don't show up with indexed levels etc. is the logs from our APM agent.

[felixbarny]

@MrThreepwood would you expect the agent logs to be in a separate file or in the same log file as your main application?

[OrangeDog]

Not who you asked, but I would expect to be able to choose, via my application’s logging configuration.

Out of the box it should all go to the same place.

[felixbarny]

What if there are multiple applications deployed to a servlet container? Should the agent logs be in every application's logs?

[OrangeDog]

I’m not familiar with how the agent or slf4j should work in a multi tenant application server.

I’m guessing the logging implementation is provided by the server, so it can be configured to send agent logs to the server log.

If instead each WAR provides its own, and the agent logs are duplicated in each one, then I could configure all but one to ignore them. I’m not sure how that could happen if the agent is attached to the server though.

You should see the theme emerging: it doesn’t matter as long as the agent doesn’t use its own fixed implementation (excepting if the host application doesn’t have a log implementation at all).

[MrThreepwood]

I'm definitely not familiar with the way that slf4j works with multiple applications on one server. I generally agree with OrangeDog that it's most important that there is SOME way to configure it. Ideally it would just use the one I've bound as I am only running a single application in my JVM, but if that's an issue I think it's important that there is some way to configure the log format for the agent in the very least.

[felixbarny]

Some noteworthy news here: #1154 (comment)

Would that solve some of your use cases?

[adrian-skybaker]

Would that solve some of your use cases?

Not really no. We configure our logging format using logback.xml with a whole bunch of environment configuration, control over the field names used in JSON output, and potentially custom kinesis appenders etc.

We don't want a whole separate logging system running with its own format output and config. We just want the APM logging to simply feed into slf4j.

[GeorgeGkinis]

We also have teams that implemented their custo JSON outputs with LogbackLogstashEncoder.

It would be awesome to use ECS reformatting from the agent and only add the ECS fields along the teams logback configured fields.