Install EKS Fargate

New Relic supports monitoring Kubernetes workloads on EKS Fargate by automatically injecting a sidecar containing the infrastructure agent and the nri-kubernetes integration in each pod that needs to be monitored.

If the same Kubernetes cluster also contains EC2 nodes, our solution will also be deployed as a DaemonSet in all of them. No sidecar will be injected into pods scheduled in EC2 nodes, and no DaemonSet will be deployed to Fargate nodes. Here's an example of a hybrid instance with both Fargate and EC2 nodes:

New Relic collects all the supported metrics for all Kubernetes objects regardless of where they are scheduled, whether it's Fargate or EC2 nodes. Please note that, due to the limitations imposed by Fargate, the New Relic integration is limited to running in unprivileged mode on Fargate nodes. This means that metrics that are usually fetched from the host directly, like running processes, will not be available for Fargate nodes.

The agent in both scenarios will scrape data from Kube State Metrics (KSM), Kubelet, and cAdvisor and send data in the same format.

Other components of the New Relic solution for Kubernetes, such as nri-prometheus, nri-metadata-injection, and nri-kube-events, do not have any particularities and will be deployed by our Helm Chart normally as they would in non-Fargate environments.

Installation

You can choose between two alternatives for installing New Relic full observability in your EKS Fargate cluster:

• Automatic injection (recommended)
• Manual injection

Regardless of the approach you choose, the experience is exactly the same after it's installed. The only difference is how the container is injected. We do recommend setting up automatic injection with the New Relic Infrastructure Operator since it will eliminate the need to manually edit each deployment you want to monitor.

Automatic injection (recommended)

By default, when Fargate support is enabled, New Relic will deploy an operator to the cluster (newrelic-infra-operator). Once deployed, this operator will automatically inject the monitoring sidecar to pods that are scheduled into Fargate nodes, while also managing the creation and the update of Secrets, ClusterRoleBindings, and any other related resources.

This operator accepts a variety of advanced configuration options that can be used to narrow or widen the scope of the injection, through the use of label selectors for both pods and namespaces.

What the operator does

Behind the scenes, the operator sets up a MutatingWebhookConfiguration, which allows it to modify the pod objects that are about to be created in the cluster. On this event, and when the pod being created matches the user’s configuration, the operator will:

1. Add a sidecar container to the pod containing the New Relic Kubernetes integration.
2. If a secret doesn't exist, create one in the same namespace as the pod containing the New Relic license key, which is needed for the sidecar to report data.
3. Add the pod’s service account to a ClusterRoleBinding previously created by the operator chart, which will grant this sidecar the required permissions to hit the Kubernetes metrics endpoints.

The ClusterRoleBinding grants the following permissions to the pod being injected:

- apiGroups: [""]
  resources:
    - "nodes"
    - "nodes/metrics"
    - "nodes/stats"
    - "nodes/proxy"
    - "pods"
    - "services"
  verbs: ["get", "list"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

Here's the injection workflow:

Automatic injection installation

First, add the New Relic Helm repository if you have not done so before:

$
helm repo add newrelic https://helm-charts.newrelic.com

Then, in order to install the operator in charge of injecting the infrastructure sidecar, please create a file named values-newrelic.yaml, which will be used to define your configuration:

global:
  licenseKey: _YOUR_NEW_RELIC_LICENSE_KEY_
  cluster: _K8S_CLUSTER_NAME_
  fargate: true  # Enable fargate support

prometheus:
  # Automatically scrape prometheus metrics for annotated services in the cluster
  # Collecting prometheus metrics for large clusters might impact data usage significantly
  enabled: true
  lowDataMode: true  # Skip ingesting cluster-level metrics
webhook:
  # Deploy our webhook to link APM and Kubernetes entities
  enabled: true
kubeEvents:
  # Report Kubernetes events
  enabled: true
logging:
  # Report logs for containers running in the cluster
  enabled: true
ksm:
  # Deploy kube-state-metrics in the cluster.
  # Set this to true unless it is already deployed.
  enabled: true
newrelic-infra-operator:
  # Deploy the infrastructure operator, which will inject the monitoring sidecar on fargate pods
  enabled: true

Finally, after creating and tweaking the file, you can deploy the solution using the following Helm command:

$
helm upgrade --install newrelic newrelic/nri-bundle \
>
--namespace newrelic --create-namespace \
>
-f values-newrelic.yaml \
>
--devel

Configuration

You can configure different aspects of the automatic injection. By default, the operator will inject the monitoring sidecar to all pods deployed in Fargate nodes which are not part of a Job or a BatchJob.

This behavior can be changed through configuration options. For example, you can define selectors to narrow or widen the selection of pods that are injected, assign resources to the operator, and tune the sidecar. Also, you can add other attributes, labels, and environment variables. Please refer to the chart README.md and values.yaml.

Update to the latest version or to a new configuration

To update to the latest version of the EKS Fargate integration, upgrade the Helm repository using helm repo update newrelic and reinstall the bundle by simply running again the command above.

To update the configuration of the infrastructure agent injected or the operator itself, modify the values-newrelic.yaml and upgrade the Helm release with the new configuration. The operator is updated immediately, and your workloads will be instrumented with the new version on their next restart. If you wish to upgrade them immediately, you can force a restart of your workloads by running:

$
kubectl rollout restart deployment YOUR_APP

Uninstall the Fargate integration

In order to uninstall the sidecar performing the automatic injection but keep the rest of the New Relic solution, using Helm, disable the infra-operator by setting infra-operator.enabled to false, either in the values.yaml file or in the command line (--set), and re-run the installation command above.

We strongly recommend keeping the --set global.fargate=true flag, since it does not enable automatic injection but makes other components of the installation Fargate-aware, preventing unwanted behavior.

To uninstall the whole solution:

1. Completely uninstall the Helm release.
2. Rollout the pods in order to remove the sidecar:
   bash

   Copy

   $
   kubectl rollout restart deployment YOUR_APP

3. Garbage collect the secrets:
   bash

   Copy

   $
   kubectl delete secrets -n YOUR_NAMESPACE -l newrelic/infra-operator-created=true

Known limitations: automatic injection

Here are some issues to be aware of when using automatic injection:

1. Currently there is no controller that watches the whole cluster to make sure that secrets that are no longer needed are garbage collected. However, all objects share the same label that you can use to remove all resources, if needed. We inject the label newrelic/infra-operator-created: true, which you can use to delete resources with a single command.

2. At the moment, it's not possible to use the injected sidecar to monitor services running in the pod. The sidecar will only monitor Kubernetes itself. However, advanced users might want to exclude these pods from automatic injection and manually inject a customized version of the sidecar with on-host integrations enabled by configuring them and mounting their configurations in the proper place. For help, see this tutorial.

Manual injection

If you have any concerns about the automatic injection, you can inject the sidecar manually directly by modifying the manifests of the workloads scheduled that are going to be scheduled on Fargate nodes. Please note that adding the sidecar into deployments scheduled into EC2 nodes may lead into incorrect or duplicate data, especially if those nodes are already being monitored with the DaemonSet.

The following objects are required for the sidecar to successfully report data:

• The ClusterRole providing the permission needed by the nri-kubernetes integration
• A ClusterRoleBinding linking the ClusterRole and the service account of the pod
• The secret storing the New Relic licenseKey in each Fargate namespace
• The sidecar container in the spec template of the monitored workload

Manual injection installation

Complete the following for manual injection:

1. If ClusterRole doesn't exist, create it and grant the permissions required to hit the metrics endpoints. This only needs to be done once, even for monitoring multiple applications in the same cluster.

   ClusterRole

   You can use this snippet as it appears below, without any changes:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     labels:
       app: newrelic-infrastructure
     name: newrelic-newrelic-infrastructure-infra-agent
   rules:
     - apiGroups:
         - ""
       resources:
         - nodes
         - nodes/metrics
         - nodes/stats
         - nodes/proxy
         - pods
         - services
       verbs:
         - get
         - list
     - nonResourceURLs:
         - /metrics
       verbs:
         - get

   Copy

2. For each workload you want to monitor, add an additional sidecar container for the newrelic/infrastructure-k8s image. Here is an example of an injected sidecar.

   ContainerToInject

   Take the container of the following snippet and inject it in the workload you want to monitor, specifying the name of your FargateProfile in the customAttributes variable. Note that the volumes can be defined as emptyDir: {}.

   Tip

   In the special case of a KSM deployment, you also need to remove the DISABLE_KUBE_STATE_METRICS environment variable and increase the resources requests and limits.

   apiVersion: apps/v1
   kind: Deployment
   spec:
     template:
       spec:
         containers:
         - name: newrelic-infrastructure
           env:
           - name: NRIA_LICENSE_KEY
             valueFrom:
               secretKeyRef:
                 key: license
                 name: newrelic-newrelic-infrastructure-config
           - name: NRIA_VERBOSE
             value: "1"
           - name: DISABLE_KUBE_STATE_METRICS
             value: "true"
           - name: CLUSTER_NAME
             value: testing-injection
           - name: COMPUTE_TYPE
             value: serverless
           - name: NRK8S_NODE_NAME
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: spec.nodeName
           - name: NRIA_DISPLAY_NAME
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: spec.nodeName
           - name: NRIA_CUSTOM_ATTRIBUTES
             value: '{"clusterName":"$(CLUSTER_NAME)", "computeType":"$(COMPUTE_TYPE)", "fargateProfile":"[YOUR FARGATE PROFILE]"}'
           - name: NRIA_PASSTHROUGH_ENVIRONMENT
             value: KUBERNETES_SERVICE_HOST,KUBERNETES_SERVICE_PORT,CLUSTER_NAME,CADVISOR_PORT,NRK8S_NODE_NAME,KUBE_STATE_METRICS_URL,KUBE_STATE_METRICS_POD_LABEL,TIMEOUT,ETCD_TLS_SECRET_NAME,ETCD_TLS_SECRET_NAMESPACE,API_SERVER_SECURE_PORT,KUBE_STATE_METRICS_SCHEME,KUBE_STATE_METRICS_PORT,SCHEDULER_ENDPOINT_URL,ETCD_ENDPOINT_URL,CONTROLLER_MANAGER_ENDPOINT_URL,API_SERVER_ENDPOINT_URL,DISABLE_KUBE_STATE_METRICS,DISCOVERY_CACHE_TTL
           image: newrelic/infrastructure-k8s:2.4.0-unprivileged
           imagePullPolicy: IfNotPresent
           resources:
             limits:
               memory: 100M
               cpu: 200m
             requests:
               cpu: 100m
               memory: 50M
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             runAsUser: 1000
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
           volumeMounts:
           - mountPath: /var/db/newrelic-infra/data
             name: tmpfs-data
           - mountPath: /var/db/newrelic-infra/user_data
             name: tmpfs-user-data
           - mountPath: /tmp
             name: tmpfs-tmp
           - mountPath: /var/cache/nr-kubernetes
             name: tmpfs-cache
   [...]

   Copy

3. Create a ClusterRoleBinding, or add to a previously created one the ServiceAccount of the application that is going to be monitored. All the workloads may share the same ClusterRoleBinding, but the ServiceAccount of each one must be added to it.

   ClusterRoleBinding

   Create the following ClusterRoleBinding that has as subjects the service account of the pods you want to monitor.

   Tip

   You don't need to repeat the same service account twice. Each time you want to monitor a pod with a service account that isn't included yet, just add it to the list.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: newrelic-newrelic-infrastructure-infra-agent
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: newrelic-newrelic-infrastructure-infra-agent
   subjects:
     - kind: ServiceAccount
       name: [INSERT_SERVICE_ACCOUNT_NAME_OF_WORKLOAD]
       namespace: [INSERT_SERVICE_ACCOUNT_NAMESPACE_OF_WORKLOAD]

   Copy

4. Create a secret containing the New Relic license key. Each namespace needs its own secret.

   Secret

   Create the following Secret that has a license with the Base64 encoded value of your license key. One secret is needed in each namespace where a pod you want to monitor is running.

   apiVersion: v1
   data:
     license: INSERT_YOUR_NEW_RELIC_LICENSE_ENCODED_IN_BASE64
   kind: Secret
   metadata:
     name: newrelic-newrelic-infrastructure-config
     namespace: [INSERT_NAMESPACE_OF_WORKLOAD]
   type: Opaque

   Copy

Configuration

When adding the manifest of the sidecar agent manually, you can use any agent configuration option to configure the agent behavior. For help, see Infrastructure agent configuration settings.

Update to the latest version

To update any of the components, you just need to modify the deployed yaml.

Updating any of the fields of the injected container will cause the pod to be re-created.

Uninstall the Fargate integration

To remove the injected container and the related resources, you just have to remove the following:

• The sidecar from the workloads that should be no longer monitored.
• All the secrets containing the newrelic license.
• ClusterRole and ClusterRoleBinding objects.

Notice that removing the sidecar container will cause the pod to be re-created.

Logging

New Relic logging isn't available on Fargate nodes because of security constraints imposed by AWS, but here are some logging options:

• If you're using Fluentbit for logging, see Kubernetes plugin for log forwarding.
• If your log data is already being monitored by AWS FireLens, see AWS FireLens plugin for log forwarding.
• If your log data is already being monitored by Amazon CloudWatch Logs, see Stream logs using Kinesis Data Firehose.
• See AWS Lambda for sending CloudWatch logs.
• See Three ways to forward logs from Amazon ECS to New Relic.

Troubleshooting: DaemonSet replicas are being deployed into Fargate nodes

If you notice that any Infra DaemonSet replicas are being scheduled on Fargate nodes, it might be because the nodeAffinity rules are not configured properly.

Double-check that the solution was installed with the global.fargate option to true, either through the command line (--set global.fargate=true) or in the values.yaml file. If the installation method was not Helm, you’ll need to manually add nodeAffinity rules to exclude Fargate nodes.

View your EKS data

Here's an example of what a Fargate node looks like in the New Relic UI:

To view your AWS data:

1. Go to one.newrelic.com > Infrastructure > Kubernetes and do one of the following:
   • Select an integration name to view data.
   • Select the Explore data icon to view AWS data.
2. Filter your data using two Fargate tags:
   • computeType=serverless
   • fargateProfile=[name of the Fargate profile to which the workload belongs]