Netty instrumentation

[felixbarny]

Netty is used as the foundation of a lot of client and server implementations. If we have generic support for monitoring HTTP requests, we'd already provide basic insight for lots of web frameworks.

One complication could be when Netty is used as the foundation of a Servlet container where we already instrument at the Servlet/Filter level.

That also doesn't solve context propagation for frameworks like Play which use Netty as the foundation but also have their own threading model.

relates to #722

[brento1]

Just curious, what is this currently being blocked by? @eyalkoren
We have started using micronaut as our microservice framework, and would love to have things autoconfigured with APM.

[eyalkoren]

@brentonr94 this is not blocked, it just didn't get to the top of our priorities yet, but it's definitely an item on our long-term roadmap.

[gmoskovicz]

@eyalkoren have we had any news regarding this implementacion?

[eyalkoren]

@gmoskovicz unfortunately not a lot to update on that yet. We may eventually decide to write dedicated support for each higher-level framework that uses Netty instead.
What are you using in your application?

[michaelhyatt]

+1 - I had a question asking for Netty support with Spring reactive framework.

[BrianEstrada]

Would be nice to see this supported in Micronaut

[dniel]

Yes, we have been using Micronaut for over two years now and would love to see support Micronaut/Netty support in ElasticCloud APM

[pitkley]

While the Elastic Java APM agent does not have native support for Netty and thus Micronaut, Micronaut does come with its own native OpenTracing integration:

• Documentation up to version 3.2.7: https://docs.micronaut.io/3.2.7/guide/index.html#distributedTracing
• Documentation since version 3.3.0: https://docs.micronaut.io/latest/guide/index.html#distributedTracing

The integration starting with Micronaut 3.3.0 has full support for OTLP, so if you have a modern enough Elastic APM setup, you can probably get by without using any agent and configure Micronaut to send the data to the OTLP-compatible Elastic APM endpoint.

If your Elastic APM setup is older and doesn't have OTLP support, or you are still on a Micronaut version <3.3.0, or you want to use the Elastic APM agent for the other niceties it has (like JVM metrics), you can use Elastic's first-party OpenTracing bridge, instruct Micronaut to trace into that, and have the Elastic APM agent take care of everything.

How to set it up:

• Add Micronaut's micronaut-tracing dependency to your project, e.g. with Gradle:

  implementation("io.micronaut:micronaut-tracing:3.2.7")

• Add Elastic's apm-opentracing dependency to your project, e.g. with Gradle:

  implementation("co.elastic.apm:apm-opentracing:1.33.0")

• Set up a singleton io.opentracing.Tracer bean which Micronaut will use to trace with:

  /*
   * The MIT License
   *
   * Copyright (c) 2022, KAISER+KRAFT EUROPA GmbH <webshop-devops@kaiserkraft-europa.de>
   *
   * Permission is hereby granted, free of charge, to any person obtaining a copy
   * of this software and associated documentation files (the "Software"), to deal
   * in the Software without restriction, including without limitation the rights
   * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
   * copies of the Software, and to permit persons to whom the Software is
   * furnished to do so, subject to the following conditions:
   *
   * The above copyright notice and this permission notice shall be included in
   * all copies or substantial portions of the Software.
   *
   * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
   * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
   * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
   * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
   * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
   * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
   * THE SOFTWARE.
   */
  package apm;
  
  import co.elastic.apm.opentracing.ElasticApmTracer;
  import io.micronaut.context.annotation.Factory;
  import io.micronaut.context.annotation.Primary;
  import io.micronaut.http.context.ServerRequestContext;
  import io.opentracing.Tracer;
  import jakarta.inject.Singleton;
  
  @Factory
  public class ElasticApmTracerFactory {
      @Singleton
      @Primary
      Tracer elasticApmTracer() {
          return new ElasticApmTracer() {
              @Override
              public SpanBuilder buildSpan(String operationName) {
                  final SpanBuilder spanBuilder = super.buildSpan(operationName);
  
                  // Elastic APM requires that the `http.url` tag is present on spans (otherwise it will reject the
                  // transactions), so we ensure here to either set it to the current request's URI or to a default value.
                  final String httpUrl = ServerRequestContext.currentRequest()
                          .map(request -> request.getUri().toString())
                          .orElse("http://unknown.host/couldNotFindRequestContext");
                  return spanBuilder.withTag("http.url", httpUrl);
              }
          };
      }
  }

  (Make sure to modify the package according to where you actually put the class.)

4. Make sure to load the Java APM agent into your JVM and configure it as documented by Elastic, just as you would for any other Java application.

You should get the full integration you can expect from JVM-based services:

We have only tested the setup provided above with the "old" tracing module, although we have used it with Micronaut versions up until 3.5.3 without issues (we have not tested 3.6.0 yet, although I expect it to work, too). I assume that a similar solution can be achieved with the new tracing module as well (most likely by just depending on io.micronaut.tracing:micronaut-tracing-core instead), we simply haven't tried yet. If anyone here does, please let us know how it works out.

I hope this helps 👍

[eyalkoren]

@pitkley thanks for this wonderful writeup!! ❤️

Just wanted to add that if "you want to use the Elastic APM agent for the other niceties it has", you can do that with OpenTelemetry as well through our OpenTelemetry bridge.

Furthermore, I assume that if you use the micronaut-tracing-opentelemetry-http dependency, it will set the http.url attribute, as required by the OTel HTTP Client spec. So if you use Java agent 1.32.0 with APM Server 7.16 or higher, this attribute should be automatically collected, sent and mapped, so you may not even need to add this special tag handling.

If you get a chance to try it out, we'll be happy to get your feedback on that.

[dniel]

As Micronaut does not have any guides on how to provide the needed configuration to configure the integrated OTLP feature and Elastic APM it would be very helpful if anyone with knowledge about how to add the correct Micronaut config for OTLP could provide an example for Elastic APM so that its possible to use directly to Elastic APM instead of the agent.

[dniel]

@eyalkoren I tried to use Micronaut 3.6.0 and adding the micronut-tracing-opentelemetry-http without adding rest of the opentelemetry dependencies make micronaut just throw exceptions.

[eyalkoren]

@dniel there are two options to trace with OTel and monitor with Elastic:

• using an OTel agent and sending data to Elastic APM Server, typically using OTLP
• using OTel as API and Elastic APM Agent as the OTel implementation

I was referring to the second option that relies on the Elastic APM Java agent's OTel bridge.
My assumption is that the micronaut-tracing-opentelemetry-http dependency uses the OTel API to enable the related tracing. You may need to add the OTel API as well, as described in our OTel bridge documentation. This, in combination with our agent, may provide the complete tracing capabilities through the Elastic stack. I am definitely not too knowledgeable about the topic as I didn't actually try it out, I am just proposing to explore this option. Going forward, I think this would bring the best of both worlds: vendor-neutral instrumentation through OTel API and Elastic extended APM capabilities. Switching to another agent is as easy as attaching another OTel-compliant agent.

I hope this helps. If you do try it out and it doesn't work, please provide more details and we'll try to make it work.

[dniel]

@eyalkoren I'm going to attempt to find a way to integrate micronaut 3.6.0 with ElasticAPM and created an discussion in micronaut-projects/micronaut-tracing#154 to follow up with the Micronaut developers.

My first attempt with using otel api and annotations dependencies and elastic java agent in micronaut did not pick up any transactions at all in first attempt, will try again in the weekend. This is my repo that im experimenting in https://github.com/nsbno/trafficinfo-baseline-micronaut

[eyalkoren]

@dniel I hope you find a way to make it work. Unfortunately I cannot allocate the time I'd need to learn everything I need for this and actively assist, but if you have concrete issues, please raise a topic in our forum, preferably with some context like your full setup, code sample and logs related to the issue. First thing to make sure is that the agent gets attached. You should get a lot of info if you set log_level=debug and it will be separated from your standard output if you configure a log_file.

Are you trying with OTel annotations, or are you referring to some Micronaut annotations?

[dniel]

@eyalkoren my elasticapm agent is reporting jvm metrics to elastic apm, so it seems the agent is configured correctly and sending data. When looking the the logs of the agent I found

2022-08-22 16:07:37,462 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Not applying excluded instrumentation co.elastic.apm.agent.opentelemetry.GlobalOpenTelemetryInstrumentation
2022-08-22 16:07:37,462 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Not applying excluded instrumentation co.elastic.apm.agent.opentelemetry.ContextStorageInstrumentation
2022-08-22 16:07:37,462 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Not applying excluded instrumentation co.elastic.apm.agent.opentelemetry.ArrayBasedContextInstrumentation

Even though I have added alle the opentelemetry depenendencies think is needed to micronaut

annotationProcessor("io.micronaut.tracing:micronaut-tracing-opentelemetry-annotation")
implementation("io.opentelemetry:opentelemetry-api")
implementation("io.opentelemetry:opentelemetry-exporter-otlp")

Even if I enable opentelemetry with https://www.elastic.co/guide/en/apm/agent/java/master/config-core.html#config-enable-instrumentations configuration it always say "not applying excluded instrumentation"

Found this config https://www.elastic.co/guide/en/apm/agent/java/master/config-core.html#config-enable-experimental-instrumentations and tried it just to see if anything happens, and found some new loglines in the agent log.

2022-08-22 20:21:34,990 [default-nioEventLoopGroup-1-2] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating OTelBridgeContext[{opentelemetry-trace-span-key=SdkSpan{traceId=7806ad8f8f72f6d60d1080a1849019c1, spanId=ea8cf1f450f24b79, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=GET - /changes, kind=SERVER, attributes=AttributesMap{data={http.method=GET, net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.flavor=1.1, net.transport=ip_tcp, http.server_name=localhost}, capacity=128, totalAddedValues=10}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1661192494975657156, endEpochNanos=0}, opentelemetry-http-server-route-key=io.opentelemetry.instrumentation.api.internal.HttpRouteState@7c2047fa, http-server-request-metrics-state=State{startAttributes={http.method=GET, net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.flavor=1.1, net.transport=ip_tcp, http.server_name=localhost}, startTimeNanos=368122076594065}, opentelemetry-traces-local-root-span=SdkSpan{traceId=7806ad8f8f72f6d60d1080a1849019c1, spanId=ea8cf1f450f24b79, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=GET - /changes, kind=SERVER, attributes=AttributesMap{data={http.method=GET, net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.flavor=1.1, net.transport=ip_tcp, http.server_name=localhost}, capacity=128, totalAddedValues=10}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1661192494975657156, endEpochNanos=0}, opentelemetry-traces-span-key-http-server=SdkSpan{traceId=7806ad8f8f72f6d60d1080a1849019c1, spanId=ea8cf1f450f24b79, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=GET - /changes, kind=SERVER, attributes=AttributesMap{data={http.method=GET, net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.flavor=1.1, net.transport=ip_tcp, http.server_name=localhost}, capacity=128, totalAddedValues=10}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1661192494975657156, endEpochNanos=0}}] on thread 65
2022-08-22 20:21:35,091 [default-nioEventLoopGroup-1-2] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Deactivating OTelBridgeContext[{opentelemetry-trace-span-key=SdkSpan{traceId=7806ad8f8f72f6d60d1080a1849019c1, spanId=ea8cf1f450f24b79, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=GET - /changes, kind=SERVER, attributes=AttributesMap{data={net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.flavor=1.1, net.transport=ip_tcp, http.request_content_length=-1, http.method=GET, http.status_code=200, http.response_content_length=-1, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.server_name=localhost}, capacity=128, totalAddedValues=13}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1661192494975657156, endEpochNanos=1661192495090706133}, opentelemetry-http-server-route-key=io.opentelemetry.instrumentation.api.internal.HttpRouteState@7c2047fa, http-server-request-metrics-state=State{startAttributes={http.method=GET, net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.flavor=1.1, net.transport=ip_tcp, http.server_name=localhost}, startTimeNanos=368122076594065}, opentelemetry-traces-local-root-span=SdkSpan{traceId=7806ad8f8f72f6d60d1080a1849019c1, spanId=ea8cf1f450f24b79, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=GET - /changes, kind=SERVER, attributes=AttributesMap{data={net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.flavor=1.1, net.transport=ip_tcp, http.request_content_length=-1, http.method=GET, http.status_code=200, http.response_content_length=-1, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.server_name=localhost}, capacity=128, totalAddedValues=13}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1661192494975657156, endEpochNanos=1661192495090706133}, opentelemetry-traces-span-key-http-server=SdkSpan{traceId=7806ad8f8f72f6d60d1080a1849019c1, spanId=ea8cf1f450f24b79, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=GET - /changes, kind=SERVER, attributes=AttributesMap{data={net.peer.ip=127.0.0.1, http.host=localhost:8080, net.peer.port=51628, http.target=/changes, http.flavor=1.1, net.transport=ip_tcp, http.request_content_length=-1, http.method=GET, http.status_code=200, http.response_content_length=-1, http.route=GET - /changes, http.user_agent=curl/7.81.0, http.server_name=localhost}, capacity=128, totalAddedValues=13}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1661192494975657156, endEpochNanos=1661192495090706133}}] on thread 65

Now got lots of this stuff in the agent log, but cant find anything in the Elastic APM where I hoped they should show up as transactions.

This is a call to a micronaut controller endpoint without any special annotations, but also when adding WithSpan opentelemetry annotations to scheduled jobs they where discovered as well but not reported. Using the Micronaut @NewSpan didn't change anything, still no transactions shown.

Changed Java agent from Elastic APM java agent to the standard OpentTelemetry Java Agent and configured it like described on https://www.elastic.co/guide/en/apm/guide/current/open-telemetry.html worked on first try, but the drawback is that no more JVM metrics is reported from the agent.

Unfortunately the agent started throwing the following warning, most probably you could try to get the instance from the agent and return instead in Micronaut.

[otel.javaagent 2022-08-22 22:18:09:229 +0200] [scheduled-executor-thread-2] WARN io.opentelemetry.api.GlobalOpenTelemetry - You are currently using the OpenTelemetry Instrumentation Java Agent; all GlobalOpenTelemetry.set calls are ignored - the agent provides the global OpenTelemetry object used by your application.
java.lang.Throwable
	at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:97)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:370)
	at io.micronaut.tracing.opentelemetry.DefaultOpenTelemetryFactory.defaultOpenTelemetry(DefaultOpenTelemetryFactory.java:97)
	at io.micronaut.tracing.opentelemetry.$DefaultOpenTelemetryFactory$DefaultOpenTelemetry0$Definition.build(Unknown Source)

[eyalkoren]

You are probably not using one of our recent agent versions, where OTel bridge is on by default and there's no need to set the enable_experimental_instrumentations config. So first thing to do is upgrade to the latest version.

This is a call to a micronaut controller endpoint without any special annotations, but also when adding WithSpan opentelemetry annotations to scheduled jobs they where discovered as well but not reported.

Our OTel bridge does not support the OTel annotation API yet, including WithSpan. We plan to add it in the future. In the mean time you can try with the non-annotation API. This should work.
As for not seeing anything without any custom instrumentation, I am not sure what should be the case and what should be turned-on/added-on in Micronaut. Did you enable the micronaut-tracing-opentelemetry-http dependency?

[dniel]

Just for info, I configured Micronaut 3.6.0 integrated open telemetry feature which run the opentelemetry-exporter in Micronaut.
I configured the opentelemetry-exporter like the official otel docs describes by setting the environment variables to my application like below.

OTEL_EXPORTER_OTLP_ENDPOINT=https://xxxxx.apm.eu-west-1.aws.cloud.es.io:443;
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer xxxxxx;
OTEL_METRICS_EXPORTER=otlp;
OTEL_RESOURCE_ATTRIBUTES=service.name=checkoutService,service.version=1.1,deployment.environment=local

This sends OTEL transactions directly to Elastic APM without any agent and using both Otel Annotations and manual transactions to send traces to Elasti APM. I didnt get any stacktraces reported under errors and no JVM metrics is reported but otherwise requests to the controllers and methods annotated with WithSpan is reported without any workaround or extra code in Micronaut and without any running agents.

[ghilainm]

Also hoping to have this working :). I am using https://github.com/mock-server/mockserver which uses netty and nothing comes out of it when plugin the APM agent on it.