AWS FireLens plugin for log forwarding

Use our AWS FireLens integration is built on our Fluentbit output plugin to connect your FireLens monitored log data to New Relic. Read on to learn how to enable this feature.

Requirements

To forward logs to New Relic using FireLens, ensure your configuration meets the following requirements:

Enable FireLens for log management

To enable log management with FireLens:

  1. Configure the FireLens log router container to run as a sidecar.
  2. Configure the Application container.
  3. Generate some traffic and wait a few minutes, then check your account for data.

Configure the FireLens log router container

New Relic uses a Fluent Bit image to configure the FireLens Log Router container. This container handles all log routing from application plugins.

To enable FireLens with Logs, you need to add a sidecar container to your pre-existing ECS task definition that will act as the Firelens log router. For help configuring ECS log routing, see Custom Log Routing, substituting the recommended images with the New Relic Fluentbit Output plugin image for your AWS region.

AWS Region Full Image Name
us-east-1 533243300146.dkr.ecr.us-east-1.amazonaws.com/newrelic/logging-firelens-fluentbit
us-east-2 533243300146.dkr.ecr.us-east-2.amazonaws.com/newrelic/logging-firelens-fluentbit
us-west-1 533243300146.dkr.ecr.us-west-1.amazonaws.com/newrelic/logging-firelens-fluentbit
us-west-2 533243300146.dkr.ecr.us-west-2.amazonaws.com/newrelic/logging-firelens-fluentbit
ca-central-1 533243300146.dkr.ecr.ca-central-1.amazonaws.com/newrelic/logging-firelens-fluentbit
eu-central-1 533243300146.dkr.ecr.eu-central-1.amazonaws.com/newrelic/logging-firelens-fluentbit
eu-west-1 533243300146.dkr.ecr.eu-west-1.amazonaws.com/newrelic/logging-firelens-fluentbit
eu-west-2 533243300146.dkr.ecr.eu-west-2.amazonaws.com/newrelic/logging-firelens-fluentbit
eu-west-3 533243300146.dkr.ecr.eu-west-3.amazonaws.com/newrelic/logging-firelens-fluentbit
eu-north-1 533243300146.dkr.ecr.eu-north-1.amazonaws.com/newrelic/logging-firelens-fluentbit

For example:

{
    "essential": true,
    // Image below is New Relic's fluentbit output plugin available on ECR
    "image": "533243300146.dkr.ecr.us-east-2.amazonaws.com/newrelic/logging-firelens-fluentbit",
    "name": "log_router",
    "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
            "enable-ecs-log-metadata": "true"
        }
    }
}

Note: EC2-type clusters will require setting the "memoryReservation" attribute for this container as well

Configure the application container

AWS Secrets Manager (recommended)

To prevent exposing your Insights Insert key in your task definition, we strongly recommend using the AWS Secrets Manager service.

When adding the secret, use the Plaintext tab. Once you've added the secret to the Secrets Manager, you can then reference it using the logConfiguration block suggested below, replacing SECRET_NAME with the name of your AWS secret.

"logConfiguration": {
     "logDriver":"awsfirelens",
     "options": {
        "Name": "newrelic"
     },
     "secretOptions": [{
        "name": "apiKey",
        "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:SECRET_NAME"
     }]
}

Plaintext Key Configuration

During configuration, outlined in FireLens Task Definitions, use the logConfiguration block suggested below, replacing INSERT_API_KEY with your New Relic Insert API key.

"logConfiguration": {
     "logDriver":"awsfirelens",
     "options": {
        "Name": "newrelic",
        "apiKey": "INSERT_API_KEY"
    }

Example configuration

Example Task Definition configuration (Fargate)

Example Task Definition for a basic nginx server:

{
    "family": "newrelic-firelens",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "containerDefinitions": [
        // FireLens log router container
        {
            "essential": true,
            // Image below is New Relic's fluentbit output plugin available on ECR
            "image": "533243300146.dkr.ecr.us-east-2.amazonaws.com/newrelic/logging-firelens-fluentbit",
            "name": "log_router",
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "enable-ecs-log-metadata": "true"
                }
            }
         },
         // Application container
         {
            "essential": true,
            "name": "webserver",
            // Application image goes here
            "image": "nginx",
            "cpu": 512,
            "memoryReservation": 1024,
            "portMappings": [{
                "containerPort": 5000
            }],
            "environment": [{
                "name": "VERSION",
                "value": "V1"
            }],
            // New Relic Fluentbit Output configuration
             "logConfiguration": {
                 "logDriver":"awsfirelens",
                 "options": {
                    "Name": "newrelic"
                 },
                 "secretOptions": [{
                    "name": "apiKey",
                    "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
                 }]
            }
        }
    ],
    // Use your own role here
    "executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecsTaskExecutionRole",
    "taskRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecsTaskExecutionRole",
    "cpu": "1 vcpu",
    "memory": "2 gb"
}
Example Task Definition configuration (EC2)

Example Task Definition for a basic nginx server:

{
    "family": "newrelic-firelens",
    "networkMode": "bridge",
    "requiresCompatibilities": [
        "EC2"
    ],
    "containerDefinitions": [
        // FireLens log router container
        {
            "essential": true,
            // Image below is New Relic's fluentbit output plugin available on ECR
            "image": "533243300146.dkr.ecr.us-east-2.amazonaws.com/newrelic/logging-firelens-fluentbit",
            "name": "log_router",
            "memoryReservation": 50,
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "enable-ecs-log-metadata": "true"
                }
            }
         },
         // Application container
         {
            "essential": true,
            "name": "webserver",
            // Application image goes here
            "image": "nginx",
            "cpu": 512,
            "memoryReservation": 1024,
            "portMappings": [{
                "containerPort": 5000
            }],
            "environment": [{
                "name": "VERSION",
                "value": "V1"
            }],
            // New Relic Fluentbit Output configuration
             "logConfiguration": {
                 "logDriver":"awsfirelens",
                 "options": {
                    "Name": "newrelic"
                 },
                 "secretOptions": [{
                    "name": "apiKey",
                    "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
                 }]
            }
        }
    ],
    // Use your own role here
    "executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecsTaskExecutionRole",
    "taskRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecsTaskExecutionRole",
    "cpu": "1 vcpu",
    "memory": "2 gb"
}

Sending logs to an EU New Relic account

If you want to send logs from Firelens to an EU account then you need to add an additional property to the options field of the logConfiguration object in your application containers.

"endpoint": "https://log-api.eu.newrelic.com/log/v1"

View log data

If everything is configured correctly and your data is being collected, you should see data logs in both of these places:

What's next?

Now that you've enabled Logs, here are some potential next steps:

If no data appears after you enable log management, follow the troubleshooting procedures.

For more help