Trace your .NET Lambda functions with New Relic and OpenTelemetry

Install the layer

AWS publishes a managed layer that includes the OpenTelemetry Lambda Collector.

To install it:

1. Open your function in the Lambda Console.

2. Under Layers in the Designer section, choose Add a layer.

3. Under Specify an ARN, paste one of the layer ARNs for your function's architecture from the list below. Replace {region}with your AWS region, such as us-east-1.

4. Choose Add.

   • AMD64 / X86_64: arn:aws:lambda:{region}:901920570463:layer:aws-otel-collector-amd64-ver-0-90-1:1
   • ARM64: arn:aws:lambda:\<region>:901920570463:layer:aws-otel-collector-arm64-ver-0-90-1:1

   For SAM and CloudFormation templates, add the following to your function's properties:

   yourFunctionHere:
     Type: AWS::Serverless::Function
     Properties:
       # ...
       Layers:
         # Use this if using x86_64 architecture
         - !Sub arn:${AWS::Partition}:lambda:${AWS::Region}:901920570463:layer:aws-otel-collector-amd64-ver-0-90-1:1
         # Use this if using arm64 architecture
         - !Sub arn:${AWS::Partition}:lambda:${AWS::Region}:901920570463:layer:aws-otel-collector-arm64-ver-0-90-1:1

   Copy

   Important

   Refer to the latest ARNs published by AWS to verify the layer ARNs above are up to date.

Add New Relic environment variables

To send the OpenTelemetry data that this layer collects to New Relic, we need to configure the OpenTelemetry Lambda Collector that is packaged with the layer to export the telemetry it receives to the New Relic OpenTelemetry Endpoint. Before we do that, we first need to set some environment variables that it will depend upon.

1. Generate and copy a New Relic license key from your New Relic account.
2. Open up your function in the Lambda Console.
3. Choose Configuration and then Environment variables.
4. Under Environment variables, choose Edit.
5. Choose Add environment variable.
6. For the Key set it to NEW_RELIC_LICENSE_KEY and set the Value to the value of the license key you generated in step 1. Then choose Save.
7. Choose Add environment variable again.
8. For the Key set it to NEW_RELIC_OPENTELEMETRY_ENDPOINT and set the Value to one of the options below (depends on your New Relic region). Then choose Save.
9. Choose Add environment variable again.
10. For the Key set it to OTEL_SERVICE_NAME and set the Value to your function's name. Then choose Save.

• otlp.nr-data.net:4317: If your New Relic account is in the US region

• otlp.eu01.nr-data.net:4317: If your New Relic account is in the EU region

  For SAM and CloudFormation templates, add the following to your function's properties. Be sure to replace your-license-key-here with your license key and otlp.nr-data.net:4317 with the New Relic OpenTelemetry Endpoint for your region.

  yourFunctionHere:
    Type: AWS::Serverless::Function
    Properties:
      # ...
      Environment:
        Variables:
          # ...
          NEW_RELIC_LICENSE_KEY: your-license-key-here
          NEW_RELIC_OPENTELEMETRY_ENDPOINT: otlp.nr-data.net:4317
          OTEL_SERVICE_NAME: your-function-name-here

  Copy

  Important

  Replace your-license-key-here with your New Relic license key and otlp.nr-data.net:4317 with the endpoint appropriate for your New Relic region (see list above).

Configure the Collector

Now we will override the OpenTelemetry Lambda Collector's default configuration with one that exports telemetry to the New Relic OpenTelemetry Endpoint. To do this we must include a collector.yaml config file with our function.

Create a collector.yaml file in your function's root directory with the following contents:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlp:
    endpoint: ${NEW_RELIC_OPENTELEMETRY_ENDPOINT}
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [otlp]

Bundle this collector.yaml file in the root directory of your function's zip package.

An example *.csproj config might look like this:

<ItemGroup>
  <Content Include="collector.yaml">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </Content>  
</ItemGroup>

Then re-deploy your function.

1. Open your function in the Lambda Console.
2. Choose Configuration and then Environment variables.
3. Under Environment variables, choose Edit.
4. Choose Add environment variable.
5. For the Key set OPENTELEMETRY_COLLECTOR_CONFIG_FILE and set the Value to /var/task/collector.yaml.
6. Then choose Save.

For SAM and CloudFormation templates, add this to your function's properties:

yourFunctionHere:
  Type: AWS::Serverless::Function
  Properties:
    # ...
    Environment:
      Variables:
        # ...
        OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml

Instrument Your Function

First add the OpenTelemetry SDK for AWS Lambda, as well as the OTLP exporter package. You can add more OpenTelemetry instrumentation packages, such as OpenTelemetry.Instrumentation.AWS and OpenTelemetry.Instrumentation.Http, to get additional visibility into your function's behavior.

$
dotnet add package OpenTelemetry.Instrumentation.AWSLambda
$
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
$
dotnet add package OpenTelemetry.Instrumentation.AWS
$
dotnet add package OpenTelemetry.Instrumentation.Http

Add calls to AddAWSLambdaConfigurations() and AddOtlpExporter() from TracerProvider in your function's static constructor.

TracerProvider tracerProvider = Sdk.CreateTracerProviderBuilder()
    // add other instrumentations here
    .AddAWSLambdaConfigurations(options => options.DisableAwsXRayContextExtraction = true)
    .AddOtlpExporter()
    .Build();

Create a wrapper function with the same signature as the original Lambda handler function. Call AWSLambdaWrapper.Trace() API and pass TracerProvider, the original Lambda function, and its inputs as parameters.

// new Lambda function handler passed in
public string TracingFunctionHandler(JObject input, ILambdaContext context) =>
    AWSLambdaWrapper.Trace(tracerProvider, OriginalFunctionHandler, input, context);

public string OriginalFunctionHandler(JObject input, ILambdaContext context) {
    return input?.ToString();
}

If your original handler is an async function, use the TraceAsync() API instead of Trace().

public Task<APIGatewayProxyResponse> TracingFunctionHandler(APIGatewayProxyRequest request,
    ILambdaContext context)
    => AWSLambdaWrapper.TraceAsync(tracerProvider, OriginalFunctionHandler, request, context);

public async Task<APIGatewayProxyResponse> OriginalFunctionHandler(APIGatewayProxyRequest apigProxyEvent, ILambdaContext context)
{
    //your function here.
}

For example, a basic API Gateway Lambda function would look like this:

using System;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using OpenTelemetry;
using OpenTelemetry.Instrumentation.AWSLambda;
using OpenTelemetry.Trace;

namespace Example
{
    public class Function
    {
        public static TracerProvider tracerProvider;

        static Function()
        {
            tracerProvider = Sdk.CreateTracerProviderBuilder()
                .AddAWSLambdaConfigurations(options => options.DisableAwsXRayContextExtraction = true)
                .AddOtlpExporter()
                .Build();

            // use AwsSdkSample::AwsSdkSample.Function::TracingFunctionHandler as input Lambda handler instead
            public APIGatewayProxyResponse TracingFunctionHandler(APIGatewayProxyRequest request, ILambdaContext context)
            {
                return AWSLambdaWrapper.Trace(tracerProvider, FunctionHandler, request, context);
            }

            /// <summary>
            /// A simple function that takes a APIGatewayProxyRequest and returns a APIGatewayProxyResponse
            /// </summary>
            /// <param name="input"></param>
            /// <param name="context"></param>
            /// <returns></returns>
            public APIGatewayProxyResponse FunctionHandler(APIGatewayProxyRequest request, ILambdaContext context)
            {
                return new APIGatewayProxyResponse() {
                    StatusCode = 200,
                    Body = Environment.GetEnvironmentVariable("_X_AMZN_TRACE_ID")
                };
            }
        }
    }
}

Then set the wrapper function as the Lambda function's handler. For the class above, the handler would be function::Example.Function::TracingFunctionHandler.

For a working example, including tracing the AWS SDK, see this sample app.

The above is just a basic example, for more advanced instrumentation refer to the OpenTelemetry .NET SDK documentation.

View your data in the New Relic UI

First you will want to invoke your Lambda function a few times to start generating telemetry. From there, head over to New Relic to find your traces, metrics, and logs.

Your telemetry will not appear under New Relic Serverless. Instead, you will find your telemetry data under the New Relic OpenTelemetry Nerdlets.