ECS integration troubleshooting: Generate verbose logs

Problem

When troubleshooting the on-host ECS integration, you can generate verbose logs for a few minutes to find and investigate errors. This can be useful for conducting your own troubleshooting or when providing information to New Relic support.

Verbose logging generates a lot of data very quickly. When finished generating logs, be sure to set verbose: 0 to reduce disk space consumption.

You can automate this process by using the newrelic-infra-ctl command. For more information, see Troubleshooting a running agent.

Solution

Generating verbose log files requires editing your task definition file. For a sample config file that includes all applicable settings, see Infrastructure configuration settings.

You have several options for implementing verbose logs:

Change the task definition environment variable and do a task restart
For EC2 launch type: Retrieve logs via SSH
Forward to CloudWatch and download with awscli
Run a command from the running container

Using task definition environment variable

To enable verbose logs by changing the environment variable and doing a task restart:

Edit your task definition. Change the value of NRIA_VERBOSE from 0 to:
- 1 for always-on verbose logs
- 2 for smart logging
- 3 for sending to New Relic
Read more about these options.
Save your task definition.
Update your service to use the newly registered task definition.
If you chose NRIA_VERBOSE=3 and you're not sending the logs directly to New Relic, you have two options for viewing and downloading the logs:
- For EC2 launch type: you can retrieve the logs via SSH, or
- Forward logs to CloudWatch
Return your settings to default:
1. Disable verbose logging by editing your task definition and setting NRIA_VERBOSE to 0.
2. Save your task definition.
3. Update your service to the latest version of your task.
Examine the log file for errors.
If you need to send your log file to New Relic support:
1. Include the line in your log file that contains the ECS integration version:
  New Relic ECS integration version X.YY.ZZZ
2. Attach the log file to your support ticket, along with your task definition .yml file.

Retrieve logs via SSH (EC2 launch type only)

To get logs via SSH:

Edit your task definition. Change the value of NRIA_VERBOSE from 0 to:
- 1 for always-on verbose logs
- 2 for smart logging
- 3 for sending to New Relic
Read more about these options.
SSH into one of your container instances.
Find the container ID of the New Relic integration container, by running the command docker ps -a. The name of the container should be nri-ecs.
Save the logs from the container with the command docker logs NRI_ECS_CONTAINER_ID > logs.txt. Leave the command running for about three minutes to generate sufficient logging data.
Continue with the instructions in the enable verbose logs section.

Forward logs to CloudWatch and download them with awscli

To get logs via CloudWatch:

Edit your task definition. Change the value of NRIA_VERBOSE from 0 to:
- 1 for always-on verbose logs
- 2 for smart logging
- 3 for sending to New Relic
Read more about these options.

We use a CloudWatch log group called /newrelic-infra/ecs to forward the logs to. To see if it already exists, run:

aws logs describe-log-groups --log-group-name-prefix /newrelic-infra/ecs

If a log group exists with that prefix, you'll get this output:

{
  "logGroups": [
    {
      "logGroupName": "/newrelic-infra/ecs",
      "creationTime": 1585828615225,
      "metricFilterCount": 0,
      "arn": "arn:aws:logs:YOUR_REGION:YOUR_AWS_ACCOUNT:log-group:/newrelic-infra/ecs:*",
      "storedBytes": 122539356
    }
  ]
}

Because this command matches log groups with prefixes, ensure the log group name returned is exactly /newrelic-infra/ecs. If the log group doesn't exist, the output will be:

{
  "logGroups": []
}

If the log group doesn't exist, create it by running:

aws logs create-log-group --log-group-name /newrelic-infra/ecs

Edit your task definition. In the container definition for the newrelic-infra container, add the following logConfiguration:

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/newrelic-infra/ecs",
    "awslogs-region": "AWS_REGION_OF_YOUR_CLUSTER",
    "awslogs-stream-prefix": "verbose"
  }
}

Next you'll look for the relevant log stream. If you have multiple instances of the task running, they'll all send their logs to the same log group but each will have its own log stream. Log streams names follow the structure AWSLOGS_STREAM_PREFIX/TASK_FAMILY_NAME/TASK_ID. In this case, it will be verbose/newrelic-infra/TASK_ID.

To get all the log streams for a given log group, run this command:

aws logs describe-log-streams --log-group-name /newrelic-infra/ecs

The following is an example output of a log group with two streams:

{
    "logStreams": [
        {
            "logStreamName": "verbose/newrelic-infra/9dfb28114e40415ebc399ec1e53a21b7",
            "creationTime": 1586166741197,
            "firstEventTimestamp": 1586166742030,
            "lastEventTimestamp": 1586173933472,
            "lastIngestionTime": 1586175101220,
            "uploadSequenceToken": "49599989655680038369205623273330095416487086853777112338",
            "arn": "arn:aws:logs:AWS_REGION_OF_YOUR_CLUSTER:YOUR_AWS_ACCOUNT:log-group:/newrelic-infra/ecs:log-stream:verbose/newrelic-infra/9dfb28114e40415ebc399ec1e53a21b7",
            "storedBytes": 0
        },
        {
            "logStreamName": "verbose/newrelic-infra/f6ce0be416804bc4bfa658da5514eb00",
            "creationTime": 1586166745643,
            "firstEventTimestamp": 1586166746491,
            "lastEventTimestamp": 1586173037927,
            "lastIngestionTime": 1586175100660,
            "uploadSequenceToken": "49605664273821671319096446647846424799651902350804230514",
            "arn": "arn:aws:logs:AWS_REGION_OF_YOUR_CLUSTER:YOUR_AWS_ACCOUNT:log-group:/newrelic-infra/ecs:log-stream:verbose/newrelic-infra/f6ce0be416804bc4bfa658da5514eb00",
            "storedBytes": 0
        }
    ]
}

From the previous list of log streams, identify the one with the task ID for which you want to retrieve the logs and use the logStreamName in this command:
```
aws logs get-log-events --log-group-name /newrelic-infra/ecs --log-stream-name "LOG_STREAM_NAME" --output text > logs.txt
```
Continue with the enable verbose logs instructions.

From running container

To enable verbose logs by running a command from the running container:

SSH into one of your container instances.
Find the container ID of the New Relic integration container by running the command docker ps -a. The name of the container should be nri-ecs.
Enable verbose logs for a limited period of time by using newrelic-infra-ctl. Run the command:
```
docker exec INTEGRATION_CONTAINER_ID /usr/bin/newrelic-infra-ctl
```
For more details, see Troubleshoot the agent.
Save the logs from the container with the command
```
docker logs INTEGRATION_CONTAINER_ID > logs.txt
```
Leave the command running for about three minutes to generate sufficient logging data.
Examine the log file for errors.

If you need to send your log file to New Relic support:

Include the line in your log file that contains the ECS integration version:
```
New Relic ECS integration version X.YY.ZZZ
```
Attach the log file to your support ticket, along with your task definition .yml file.