Trace your Java Lambda functions with New Relic and OpenTelemetry

This guide will cover how you can use OpenTelemetry Lambda to trace your Java Lambda functions using AWS's managed OpenTelemetry Lambda Layers. OpenTelemetry makes it easy to instrument your functions, including automatic instrumentation for many popular libraries.

Prerequisites

This guide assumes you have the following:

• A New Relic account. If you don't have one, create one for free.
• An AWS account. If you don't have one, create one for free.
• A Java Lambda Function running under either the java8.al2 or java11 runtimes. If you don't have one yet, create one now.

Step 1: Enable X-Ray

Even though we're not using AWS X-Ray in this setup, we still need to enable X-Ray in the Lambda function to enable the tracing features built into AWS OpenDistro.

To enable X-Ray:

1. Open your function in the Lambda Console.
2. Choose Configuration, then Monitoring Tools.
3. Choose Edit.
4. Under X-Ray, enable Active tracing.

Alternatively, if you're using SAM (Serverless Application Model) or CloudFormation templates, you can enable X-Ray by adding the following to your Lambda function's properties:

yourFunctionHere:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Policies:
        - AWSLambdaBasicExecutionRole
        ...
        - AWSXrayWriteOnlyAccess
      Tracing: Active

Step 2: Install the layer

AWS publishes a managed layer that includes the OpenTelemetry Lambda collector, the OpenTelemetry Java SDK, and the ADOT auto instrumentation agent.

To install it:

1. Open your function in the Lambda Console.

2. Under Layers in the Designer section, choose Add a layer.

3. Under Specify an ARN, paste one of the layer ARNs for your function's architecture form the list below. Replace {region}with your AWS region, such as us-east-1.

4. Choose Add.

   • AMD64 / X86_64: arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-agent-amd64-ver-1-11-1:1
   • ARM64: arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-agent-arm64-ver-1-11-1:1

For SAM or CloudFormation templates, add this to your function's properties:

yourFunctionHere:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Layers:
        # Use this if using x86_64 architecture
        - !Sub arn:${AWS::Partition}:lambda:${AWS::Region}:901920570463:layer:aws-otel-java-agent-amd64-ver-1-11-1:1
        # Use this if using arm64 architecture
        - !Sub arn:${AWS::Partition}:lambda:${AWS::Region}:901920570463:layer:aws-otel-java-agent-arm64-ver-1-11-1:1

Step 3: Configure the layer

To configure the layer we need to configure an exec wrapper. The exec wrapper is a script that gets run during function initialization. In this case, the script configures OpenTelemetry.

1. Open your function in the Lambda Console.

2. Choose Configuration and then Environment variables.

3. Under Environment variables, choose Edit.

4. Choose Add environment variable.

5. For the Key set it to AWS_LAMBDA_EXEC_WRAPPER and set the Value to one of the following options (depends on your handler type). Then choose Save.

   • /opt/otel-handler: for wrapping regular handlers (implementing RequestHandler)
   • /opt/otel-proxy-handler: for wrapping regular handlers (implementing RequestHandler) proxied through API Gateway, enabling HTTP context propagation
   • /opt/otel-stream-handler: for wrapping streaming handlers (implementing RequestStreamHandler), enabling HTTP context propagation for HTTP requests

For SAM or CloudFormation templates, add this to your function's properties:

yourFunctionHere:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Environment:
        Variables:
          AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler

Step 4: Add New Relic environment variables

To send the OpenTelemetry data that this layer collects to New Relic, we need to configure the OpenTelemetry Lambda Collector that is packaged with the layer to export the telemetry it receives to the New Relic OpenTelemetry Endpoint. Before we do that, we first need to set some environment variables that it will depend upon.

1. Generate and copy a New Relic license key from your New Relic account.

2. Open up your function in the Lambda Console.

3. Choose Configuration and then Environment variables.

4. Under Environment variables, choose Edit.

5. Choose Add environment variable.

6. For the Key set it to NEW_RELIC_LICENSE_KEY and set the Value to the value of the license key you generated in step 1. Then choose Save.

7. Choose Add environment variable again.

8. For the Key set it to NEW_RELIC_OPENTELEMETRY_ENDPOINT and set the Value to one of the options below (depends on your New Relic region). Then choose Save.

   • otlp.nr-data.net:4317: If your New Relic account is in the US region
   • otlp.eu01.nr-data.net:4317: If your New Relic account is in the EU region

For SAM and CloudFormation templates, add the following to your function's properties. Be sure to replace your-license-key-here with your license key and otlp.nr-data.net:4317 with the New Relic OpenTelemetry Endpoint for your region.

yourFunctionHere:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Environment:
        Variables:
          ...
          NEW_RELIC_LICENSE_KEY: your-license-key-here
          NEW_RELIC_OPENTELEMETRY_ENDPOINT: otlp.nr-data.net:4317

Step 5: Configure the collector

Now we will override the OpenTelemetry Lambda Collector's default configuration with one that exports telemetry to the New Relic OpenTelemetry Endpoint. To do this we must include a collector.yaml config file with our function.

Create a collector.yaml file in your function's root directory with the following contents:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlp:
    endpoint: ${NEW_RELIC_OPENTELEMETRY_ENDPOINT}
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [otlp]

Bundle this collector.yaml file in the root directory of your function's zip package and re-deploy.

1. Open your function in the Lambda Console.
2. Choose Configuration and then Environment variables.
3. Under Environment variables, choose Edit.
4. Choose Add environment variable.
5. For the Key set OPENTELEMETRY_COLLECTOR_CONFIG_FILE and set the Value to /var/task/collector.yaml.
6. Then choose Save.

For SAM and CloudFormation templates, add this to your function's properties:

yourFunctionHere:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Environment:
        Variables:
          ...
          OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml

Step 6: View your data in the New Relic UI [view-data]

First you will want to invoke your Lambda function a few times to start generating telemetry. From there, head over to New Relic to find your traces, metrics, and logs.

Your telemetry will not appear under New Relic Serverless. Instead, you will find your telemetry data under the New Relic OpenTelemetry Nerdlets.

Automatic instrumentation overhead

ADOT Lambda Layer for Java auto-instrumentation agent has a notable impact on startup time on AWS Lambda, and you will generally need to use this along with provisioned concurrency to serve production requests without causing timeouts on initial requests while it initializes. We recommend testing this configuration in a non-production environment first to determine the appropriate settings for your use case.

Alternatively, you can use the following manual instrumentation method. By default, manual instrumentation requires fewer resources at function initialization. However, this method requires a code change in most cases.

Manual instrumentation

The manual instrumentation method is similar to the automatic instrumentation procedure in this document. The only difference is the Lambda Layer ARN you specify in step 1.

For manual instrumentation, instead of using the layer ARNs in Step 2, use one of the layer ARNs in the list below for your function's architecture:

• AMD64 / X86_64: arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-11-1:1
• ARM64: arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-wrapper-arm64-ver-1-11-1:1

Replace the {region} with your AWS region, such as us-east-1.

All other steps remain the same.

This alternative Lambda Layer will still wrap your Lambda function like the automatic method. It will also instrument the AWS SDK automatically. But all other libraries you use will require you to include the library's OpenTelemetry instrumentation library from the OpenTelemetry instrumentation repository in your function's dependencies and modify your code to initialize it.

You can see an example with OKHttp on GitHub.

Distributed Tracing

In some cases you may see fragmented distributed traces within New Relic with this configuration. This occurs when a trace starts or involves a service that is outside the ADOT context (for example, a managed AWS service). Spans from that service are created by X-Ray, not OpenTelemetry, and ADOT does not forward them to New Relic. Although the traces appear fragmented, they still provide complete insight into performance within the Lambda function as well as other services whose spans were forwarded to New Relic.

More information

For more information, check out the New Relic OpenTelemetry quick start.