Monitor ECS Fargate with OpenTelemetry

Monitor Amazon ECS tasks running on AWS Fargate by deploying OpenTelemetry Collector Contrib as a sidecar container. This comprehensive guide walks you through creating task definitions, configuring the collector, and setting up monitoring for your serverless ECS workloads.

Installation steps

Follow these steps in order to set up monitoring for your ECS Fargate tasks.

Before you begin

Make sure your environment meets these requirements:

Store your New Relic license key

Save your license key as a Systems Manager (SSM) parameter to securely store credentials for the OpenTelemetry Collector:

bash

$aws ssm put-parameter \
>  --name "/newrelic-infra/ecs/license-key" \
>  --type SecureString \
>  --description 'New Relic license key for ECS monitoring' \
>  --value "YOUR_NEW_RELIC_LICENSE_KEY"

Create IAM policy and execution role

Create an IAM policy so your ECS containers can securely retrieve the New Relic license key:

bash

$aws iam create-policy \
>  --policy-name "NewRelicSSMLicenseKeyReadAccess" \
>  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ssm:GetParameters"],"Resource":["arn:aws:ssm:*:*:parameter/newrelic-infra/ecs/license-key"]}]}' \
>  --description "Provides read access to the New Relic SSM license key parameter"

Create an IAM role to be used as the task execution role:

bash

$aws iam create-role \
>  --role-name "NewRelicECSTaskExecutionRole" \
>  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ecs-tasks.amazonaws.com"},"Action":"sts:AssumeRole"}]}' \
>  --description "ECS task execution role for New Relic infrastructure"

Attach the required managed policies to the role:

bash

$# Attach the standard ECS task execution policy
$aws iam attach-role-policy \
>  --role-name "NewRelicECSTaskExecutionRole" \
>  --policy-arn "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
$
$# Attach the New Relic SSM license key read access policy
$aws iam attach-role-policy \
>  --role-name "NewRelicECSTaskExecutionRole" \
>  --policy-arn "arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/NewRelicSSMLicenseKeyReadAccess"

Store collector configuration

Store the OpenTelemetry Collector configuration in AWS Systems Manager Parameter Store so you can manage and update settings without rebuilding container images:

bash

$aws ssm put-parameter \
>  --name "/ecs/otel-collector/fargate-config" \
>  --type "String" \
>  --value "$(cat <<EOF
$receivers:
$  awsecscontainermetrics:
$    collection_interval: <COLLECTION_INTERVAL>
$
$processors:
$  metricstransform/containers:
$    transforms:
$      - include: container.cpu.utilized
$        action: insert
$        new_name: container.cpu.utilization
$      - include: container.memory.usage
$        action: insert
$        new_name: container.memory.usage.total
$      - include: container.storage.read_bytes
$        action: insert
$        new_name: container.blockio.io_service_bytes_recursive
$        operations:
$          - action: add_label
$            new_label: operation
$            new_value: read
$      - include: container.storage.write_bytes
$        action: insert
$        new_name: container.blockio.io_service_bytes_recursive
$        operations:
$          - action: add_label
$            new_label: operation
$            new_value: write
$
$  batch:
$    send_batch_size: <SEND_BATCH_SIZE>
$    timeout: <BATCH_TIMEOUT>
$
$  resource:
$    attributes:
$      - key: ClusterName
$        from_attribute: aws.ecs.cluster.name
$        action: insert
$      - key: ServiceName
$        from_attribute: aws.ecs.service.name
$        action: insert
$      - key: TaskId
$        from_attribute: aws.ecs.task.id
$        action: insert
$      - key: TaskDefinitionFamily
$        from_attribute: aws.ecs.task.family
$        action: insert
$      - key: LaunchType
$        from_attribute: aws.ecs.launch_type
$        action: insert
$      - key: cloud.platform
$        value: \"aws_ecs\"
$        action: upsert
$      - key: docker.host
$        from_attribute: aws.ecs.task.id
$        action: insert
$      - key: docker.imageName
$        from_attribute: container.image.name
$        action: insert
$      - key: docker.containerId
$        from_attribute: container.id
$        action: insert
$      - key: docker.state
$        from_attribute: aws.ecs.container.know_status
$        action: insert
$
$
$exporters:
$  otlphttp:
$    endpoint: https://otlp.nr-data.net:443
$    headers:
$      api-key: \${NEW_RELIC_LICENSE_KEY}
$
$  debug:
$    verbosity: basic
$
$service:
$  pipelines:
$    metrics/containers:
$      receivers: [awsecscontainermetrics]
$      processors: [metricstransform/containers, resource, batch]
$      exporters: [otlphttp, debug]
$EOF
$)"

Configuration parameters

The following parameters can be customized in the OpenTelemetry Collector configuration:

Parameter	Description
`<COLLECTION_INTERVAL>`	The interval in seconds for collecting metrics from the ECS container metrics endpoint.
`<MEMORY_LIMIT_MIB>`	Memory limit for the OpenTelemetry Collector in MiB
`<SEND_BATCH_SIZE>`	Number of metrics to batch before sending to New Relic
`<BATCH_TIMEOUT>`	Maximum time to wait before sending a batch
`<RESOURCE_DETECTION_TIMEOUT>`	Timeout for resource detection processors

Create task definition

Create a new ECS task definition for Fargate that includes the OpenTelemetry Collector sidecar container. Choose the appropriate task definition for your container platform:

{
  "family": "otel-ecs-fargate-metrics",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "<TASK_CPU>",
  "memory": "<TASK_MEMORY>",
  "executionRoleArn": "arn:aws:iam::YOUR_ACCOUNT:role/NewRelicECSTaskExecutionRole",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  },
  "containerDefinitions": [
    {
      "name": "busybox",
      "image": "busybox:latest",
      "cpu": <APP_CPU>,
      "memory": <APP_MEMORY>,
      "essential": true,
      "command": [
        "sh",
        "-c",
        "while true; do echo 'Hello from BusyBox'; sleep 30; done"
      ],
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "<APP_LOG_GROUP>",
          "awslogs-create-group": "true",
          "awslogs-region": "<AWS_REGION>",
          "awslogs-stream-prefix": "<APP_LOG_STREAM_PREFIX>"
        }
      }
    },
    {
      "name": "otel-collector",
      "image": "otel/opentelemetry-collector-contrib:latest",
      "cpu": <COLLECTOR_CPU>,
      "memoryReservation": <COLLECTOR_MEMORY_RESERVATION>,
      "essential": true,
      "command": ["--config=env:OTEL_CONFIG"],
      "environment": [
        {
          "name": "NRIA_IS_FORWARD_ONLY",
          "value": "true"
        },
        {
          "name": "NRIA_PASSTHROUGH_ENVIRONMENT",
          "value": "ECS_CONTAINER_METADATA_URI,ECS_CONTAINER_METADATA_URI_V4,FARGATE"
        },
        {
          "name": "FARGATE",
          "value": "true"
        },
        {
          "name": "NRIA_OVERRIDE_HOST_ROOT",
          "value": ""
        }
      ],
      "secrets": [
        {
          "name": "OTEL_CONFIG",
          "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/ecs/otel-collector/fargate-config"
        },
        {
          "name": "NEW_RELIC_LICENSE_KEY",
          "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/newrelic-infra/ecs/license-key"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "<COLLECTOR_LOG_GROUP>",
          "awslogs-create-group": "true",
          "awslogs-region": "<AWS_REGION>",
          "awslogs-stream-prefix": "<COLLECTOR_LOG_STREAM_PREFIX>"
        }
      }
    }
  ]
}

{
  "family": "otel-ecs-fargate-metrics-windows",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "<TASK_CPU>",
  "memory": "<TASK_MEMORY>",
  "executionRoleArn": "arn:aws:iam::YOUR_ACCOUNT:role/NewRelicECSTaskExecutionRole",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "WINDOWS_SERVER_2022_FULL"
  },
  "containerDefinitions": [
    {
      "name": "busybox",
      "image": "winamd64/hello-world",
      "cpu": <APP_CPU>,
      "memoryReservation": <APP_MEMORY>,
      "essential": true,
      "entryPoint": ["cmd"],
      "command": [
        "/c",
        "for /L %i in () do @(echo Running... && timeout /t 30 /nobreak >nul)"
      ],
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "<APP_LOG_GROUP>",
          "awslogs-region": "<AWS_REGION>",
          "awslogs-stream-prefix": "<APP_LOG_STREAM_PREFIX>"
        }
      }
    },
    {
      "name": "otel-collector",
      "image": "otel/opentelemetry-collector-contrib:latest-windows-2022-amd64",
      "cpu": <COLLECTOR_CPU>,
      "memoryReservation": <COLLECTOR_MEMORY_RESERVATION>,
      "essential": true,
      "entryPoint": ["C:\\otelcol-contrib.exe"],
      "command": ["--config", "env:OTEL_CONFIG"],
      "environment": [
        {
          "name": "NRIA_IS_FORWARD_ONLY",
          "value": "true"
        },
        {
          "name": "NRIA_PASSTHROUGH_ENVIRONMENT",
          "value": "ECS_CONTAINER_METADATA_URI,ECS_CONTAINER_METADATA_URI_V4,FARGATE"
        },
        {
          "name": "FARGATE",
          "value": "true"
        },
        {
          "name": "NRIA_OVERRIDE_HOST_ROOT",
          "value": ""
        }
      ],
      "secrets": [
        {
          "name": "OTEL_CONFIG",
          "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/ecs/otel-collector/fargate-config-windows"
        },
        {
          "name": "NEW_RELIC_LICENSE_KEY",
          "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/newrelic-infra/ecs/license-key"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "<COLLECTOR_LOG_GROUP>",
          "awslogs-create-group": "true",
          "awslogs-region": "<AWS_REGION>",
          "awslogs-stream-prefix": "<COLLECTOR_LOG_STREAM_PREFIX>"
        }
      }
    }
  ]
}

Task definition parameters

The following parameters can be customized in your ECS Fargate task definition:

Parameter	Description
`<TASK_CPU>`	Total CPU units for the Fargate task
`<TASK_MEMORY>`	Total memory for the Fargate task in MiB
`<APP_CPU>`	CPU units allocated to your application container
`<APP_MEMORY>`	Memory allocated to your application container in MiB
`<COLLECTOR_CPU>`	CPU units allocated to the OpenTelemetry Collector
`<COLLECTOR_MEMORY_RESERVATION>`	Memory reservation for the OpenTelemetry Collector in MiB
`<APP_LOG_GROUP>`	CloudWatch log group name for your application container
`<COLLECTOR_LOG_GROUP>`	CloudWatch log group name for the OpenTelemetry Collector
`<AWS_REGION>`	AWS region for CloudWatch logs
`<APP_LOG_STREAM_PREFIX>`	Log stream prefix for your application container
`<COLLECTOR_LOG_STREAM_PREFIX>`	Log stream prefix for the OpenTelemetry Collector

ヒント

Key differences between Linux and Windows Fargate containers:

Runtime platform: Windows requires explicit operatingSystemFamily: "WINDOWS_SERVER_2022_FULL"
Images: Windows uses specific Windows container images
Entry point: Windows collector specifies C:\\otelcol-contrib.exe as entry point
Memory allocation: Uses memoryReservation for flexible memory management

重要

Replace YOUR_ACCOUNT and region values with your actual AWS account ID and AWS region.

Deploy and run the task

Deploy your task definition to your ECS cluster:

bash

$aws ecs register-task-definition --cli-input-json file://task-definition.json

Create a service:

bash

$aws ecs create-service \
>  --cluster your-cluster-name \
>  --service-name otel-monitoring-service \
>  --task-definition otel-ecs-fargate-metrics:1 \
>  --desired-count 1 \
>  --launch-type FARGATE \
>  --network-configuration "awsvpcConfiguration={subnets=[subnet-12345,subnet-67890],securityGroups=[sg-abcdef],assignPublicIp=ENABLED}"

Verify data collection

Check that your data is flowing to New Relic:

Check OpenTelemetry Collector status: Review container logs to confirm the collector is running without errors and successfully connecting to New Relic:
bash
```
$aws logs get-log-events \
>  --log-group-name "/ecs/otel-collector-fargate" \
>  --log-stream-name "otel/otel-collector/TASK_ID"
```
Verify data in New Relic UI: Navigate to one.newrelic.com > All Capabilities > Infrastructure to confirm your ECS containers appear with metrics. For detailed guidance on exploring your data, see Find and query your ECS monitoring data.

Configuration differences from EC2

Key differences when running on Fargate:

No host-level access: Cannot access underlying host metrics like in EC2
Network mode: Must use awsvpc network mode
Resource constraints: Limited by Fargate CPU/memory combinations
Storage: Ephemeral storage only, no persistent volumes
Container insights: Relies on Task Metadata Endpoint v4 for container stats

Next steps

After setting up monitoring, you can:

Create custom dashboards for your Fargate metrics
Set up alerts for container and task-level issues
Correlate ECS metrics with application traces and logs

Installation steps

Before you begin

ECS requirements

Fargate requirements

AWS permissions

New Relic requirements

Store your New Relic license key

Create IAM policy and execution role

Store collector configuration

Configuration parameters

Create task definition

Linux containers

Windows containers

Task definition parameters

ヒント

重要

Deploy and run the task

Verify data collection

Configuration differences from EC2

Next steps

Monitor ECS Fargate with OpenTelemetry

Installation steps .css-21sua1{background:none;border:none;width:0;padding:0;}

Before you begin

Fargate requirements

AWS permissions

New Relic requirements

Store your New Relic license key

Create IAM policy and execution role

Store collector configuration

Configuration parameters

Create task definition

Linux containers

Windows containers

Task definition parameters

ヒント

重要

Deploy and run the task

Verify data collection

Configuration differences from EC2

Next steps

Installation steps