Install containerized private minions (CPMs)

Important

As of August 26, 2024, you can no longer create new monitors using legacy runtimes on public or private locations.

On October 22, 2024, we will end of life the containerized private minion (CPM) and the legacy synthetics runtime versions it supports. Please review our recommended migration steps to avoid degradation of your private location monitors.

You can use New Relic's containerized private minions (CPM). These are Docker container-based private minions that accept and execute synthetic monitors against your private locations.

The CPM can operate in a Docker container system environment or a Kubernetes container orchestration system environment. The CPM will auto-detect its environment to select the appropriate operating mode.

General private minion features

Because the CPM operates as a container instead of a virtual machine, it delivers many features:

Easy to install, start, and update
Runs on:
- Linux
- macOS
- Windows
Enhanced security and support for non-root user execution
Ability to leverage a Docker container as a sandbox environment
Customizable monitor check timeout
Custom provided modules for scripted monitor types

Kubernetes-specific features

Also, the CPM delivers the following features in a Kubernetes environment:

Integrates with the Kubernetes API to delegate runtime lifecycle management to Kubernetes
Does not require privileged access to the Docker socket
Supports hosted and on-premise Kubernetes clusters
Supports various container engines such as Docker and Containerd
Deployable via Helm charts as well as configuration YAMLs
Allows job (ping vs. non-ping checks) based resource allocation for optimum resource management
Observability offered via the New Relic Kubernetes cluster explorer

System requirements and compatibility

To host CPMs, your system must meet the minimum requirements for the chosen system environment.

Caution

Do not modify any CPM files. New Relic is not liable for any modifications you make. For more information, contact your account representative or a New Relic technical sales rep.

Compatibility for	Requirements
Platform	The OS and architecture must be a platform supported by Docker. You must also configure Docker to run Linux containers in order for CPMs to work on Windows-based Docker hosts.
Memory	2.5 GiB of RAM per CPU core (dedicated)
Disk size	A minimum of 50 GiB per host
Disk performance (IOPS)	Ping: Negligible Scripted API: 7 Simple browser: 27 Scripted browser: 28 Measured by monitoring write throughput for https://newrelic.com on an AWS EC2 m5.xlarge instance with: AL2, gp2 storage class, 50 GiB root volume, a default Docker CPM install, and 1 monitor at a time set to 1-minute frequency. Efficiency gains are expected with multiple monitors running. These values may be higher or lower than your own depending on many factors. Baseline for Docker CPM is 0.5 IOPS with no monitor jobs running.
Disk file system	NFSv4.1 or higher (if using NFS)
Docker version	Docker Engine versions starting with 17.12.1-ce up to and including version 25.
Private location key	You must have a private location key

Caution

The Docker CPM does not support Docker Engine 26.0 or higher due to breaking changes. Customers seeking Docker 26+ support should migrate to Synthetics Job Manager

Caution

The Docker CPM is not designed for use with container orchestrators like AWS ECS, Docker Swarm, Apache Mesos, Azure Container Instances, etc. Running the Docker CPM in a container orchestrator will result in unexpected issues because it is itself a container orchestrator. If you're using container orchestration, see our Kubernetes CPM requirements.

Compatibility for	Requirements
Operating system	Linux kernel: 3.10 or higher macOS: 10.11 or higher Linux containers, including the containerized private minion, only run on Linux K8s nodes.
Processor	A modern, multi-core CPU
Minion pod	CPU (vCPU/Core): 0.5 up to 0.75 Memory: 800Mi up to 1600Mi Resources allocated to a Minion pod are user configurable.
Runner pod	CPU (vCPU/Core): 0.5 up to 1 Memory: 1250Mi up to 3000Mi For a scripted API check, 1250Mi will be requested with a limit of 2500Mi. For a simple browser or scripted browser check, 2000Mi will be requested with a limit of 3000Mi. Additional considerations: Resources allocated to a runner pod are not user configurable. The maximum limit-request resource ratio for both CPU and memory is 2.
Disk size	Root volume: A minimum of 50 GiB (node + PV) Persistent volume (PV): A minimum of 20 GiB If a `ReadWriteOnce` (RWO) PV is provided to the minion, an implicit node affinity will be established to ensure the minion and the runner containers are scheduled on the same node. This is required to allow the minion and the associated runners access to the PV, as an RWO PV can be accessed only by a single node in the cluster.
Disk performance (IOPS)	Ping: Negligible Scripted API: 11 Simple browser: 33 Scripted browser: 31 Measured by monitoring write throughput for https://newrelic.com on an AWS EKS 1.21 cluster backed by one EC2 m5.xlarge node running with: gp2 storage class, 50 GiB root volume, 20 GiB PV/PVC with RWO access mode, a default Kubernetes CPM install via Helm, and 1 monitor at a time set to 1-minute frequency. Efficiency gains are expected with multiple monitors running. These values may be higher or lower than your own depending on many factors. Baseline for Kubernetes CPM is 3.0 IOPS with no monitor jobs running.
Disk file system	NFSv4.1 or higher (if using NFS)
Kubernetes version	We recommend that your Kubernetes cluster supports Kubernetes v1.15. Caution For Kubernetes v1.21 or newer, use minion release v3.0.61 or newer.
Private location key	You must have a private location key
Helm	EOL NOTICE After August 2022, we're discontinuing support for several capabilities, including our original private minion Helm chart URL. For more details, including how you can easily prepare for this transition, see our Support Forum post. Follow installation instructions for Helm v3 for your OS.
Kubectl	Follow installation instructions for Kubectl for your OS.

To view versions, dependencies, default values for how many runner pods start with each minion, the Persistent volume access mode, and more, please see Show help and examples below.

Private location key

Before launching CPMs, you must have a private location key. Your CPM uses the key to authenticate against New Relic and run monitors associated with that private location.

To find the key for existing private location:

Go to one.newrelic.com > Synthetic monitoring > Private locations.
In the Private locations index, locate the private location you want your CPM to be assigned to.
Note the key associated with the private location with the key icon.

Sandboxing and Docker dependencies

Sandboxing and Docker dependencies are applicable to the CPM in a Docker container system environment.

The CPM runs in Docker and is able to leverage Docker as a sandboxing technology. This ensures complete isolation of the monitor execution, which improves security, reliability, and repeatability. Every time a scripted or browser monitor is executed, the CPM creates a brand new Docker container to run it in called a runner.

The minion container needs to be configured to communicate with the Docker engine in order to spawn additional runner containers. Each spawned container is then dedicated to run a check associated with the synthetic monitor running on the private location the minion container is associated with.

There are two crucial dependencies at launch. To enable sandboxing, ensure that:

Your writable and executable directory is mounted at /tmp. The writable directory can be any directory you want the CPM to write into, but New Relic recommends the system's own /tmp to make things easy.
Your writable Docker UNIX socket is mounted at /var/run/docker.sock or DOCKER_HOST environment variable. For more information, see Docker's Daemon socket option.
Caution
Core count on the host determines how many runner containers the CPM can run concurrently on the host. Since memory requirements are scaled to the expected count of runner containers, we recommend not running multiple CPMs on the same host to avoid resource contention.
For additional information on sandboxing and running as a root or non-root user, see Security, sandboxing, and running as non-root.

Install and update CPM versions

Both installing and updating the CPM use the same command to pull the latest Docker image from the Quay.io repository where the CPM Docker image is hosted. Go to quay.io/repository/newrelic/synthetics-minion for a list of all the releases.

CPM images are also hosted on Docker Hub. Go to hub.docker.com/r/newrelic/synthetics-minion/tags for a list of all the releases.

Unless hosting the images in a local image repository, connections to either quay.io or docker.io will need to be allowed through your firewall in order for Docker to pull the synthetics-minion and synthetics-minion-runner images. The "runner" image is pulled automatically on startup of the synthetics-minion container. See Docker environment configuration and Kubernetes environment configuration for details on how to set a local repository and the runner registry endpoint.

Start the CPM

To start the CPM, follow the applicable Docker or Kubernetes instructions.

Locate your private location key.
Ensure you've enabled Docker dependencies for sandboxing and installed CPM on your system.
Run the appropriate script for your system. Tailor the common defaults for /tmp and /var/run/docker.sock in the following examples to match your system.
- Linux/macOS:
  bash
```
$docker run \
>    --name YOUR_CONTAINER_NAME \
>    -e "MINION_PRIVATE_LOCATION_KEY=YOUR_PRIVATE_LOCATION_KEY" \
>    -v /tmp:/tmp:rw \
>    -v /var/run/docker.sock:/var/run/docker.sock:rw \
>    -d \
>    --restart unless-stopped \
>    quay.io/newrelic/synthetics-minion:latest
```
- Windows:
  bash
```
$docker run ^
$    --name YOUR_CONTAINER_NAME ^
$    -e "MINION_PRIVATE_LOCATION_KEY=YOUR_PRIVATE_LOCATION_KEY" ^
$    -v /tmp:/tmp:rw ^
$    -v /var/run/docker.sock:/var/run/docker.sock:rw ^
$    -d ^
$    --restart unless-stopped ^
$    quay.io/newrelic/synthetics-minion:latest
```
View the logs for your minion container:
bash
```
$docker logs --follow YOUR_CONTAINER_NAME
```
When a message similar to Synthetics Minion is ready and servicing location YOUR_PRIVATE_LOCATION_LABEL appears, your CPM is up and ready to run monitors assigned to that location.

EOL NOTICE

After August 2022, we're discontinuing support for several capabilities, including our original private minion Helm chart URL. For more details, including how you can easily prepare for this transition, see our Support Forum post.

Locate your private location key.
Set up the a namespace for the CPM in your Kubernetes cluster:
bash
```
$kubectl create namespace YOUR_NAMESPACE
```
Copy the Helm charts from the New Relic Helm repo.
- If you are copying the charts for the first time:
  bash
```
$helm repo add YOUR_REPO_NAME https://helm-charts.newrelic.com
```
- If you previously copied the Helm charts from the New Relic Helm repo, then get the latest:
  bash
```
$helm repo update
```

Install the CPM with the following Helm command:

For a fresh installation of the CPM:

bash

$helm install YOUR_CPM_NAME YOUR_REPO_NAME/synthetics-minion -n YOUR_NAMESPACE --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY

To update an existing CPM installation:

bash

$helm upgrade YOUR_CPM_NAME YOUR_REPO_NAME/synthetics-minion -n YOUR_NAMESPACE --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY

Check if the minion pod is up and running:
bash
```
$kubectl get -n YOUR_NAMESPACE pods
```
Once the status attribute of each pod is shown as running, your CPM is up and ready to run monitors assigned to your private location.

Stop or delete the CPM

On a Docker container system environment, use the Docker stop procedure to stop the CPM from running. On a Kubernetes container orchestration system environment, use the Kubernetes delete procedure to stop the CPM from running.

You can stop a Docker container either by the container name, or the container ID.

Container name stop for Linux, macOS, and Windows:

bash

$docker stop YOUR_CONTAINER_NAME
$docker rm YOUR_CONTAINER_NAME

Container ID stop for Linux/macOS:
In the examples the container is stopped and removed. To only stop the container, omit docker rm $CONTAINER_ID.
bash
```
$CONTAINER_ID=$(docker ps -aqf name=YOUR_CONTAINER_NAME)
$docker stop $CONTAINER_ID
$docker rm $CONTAINER_ID
```

Container ID stop for Windows:

In the examples the container is stopped and removed. To only stop the container, omit docker rm $CONTAINER_ID.

bash

$FOR /F "tokens=*" %%CID IN ('docker ps -aqf name=YOUR_CONTAINER_NAME') do (SET CONTAINER_ID=%%CID)
$docker stop %CONTAINER_ID%
$docker rm %CONTAINER_ID%

Get the MINION_POD_INSTALLATION_NAME of the minion pod you want to delete:
bash
```
$helm list -n YOUR_NAMESPACE
```

Delete the minion pod:

bash

$helm uninstall -n YOUR_NAMESPACE MINION_POD_INSTALLATION_NAME

Delete the namespace set up for the CPM in your Kubernetes cluster:
bash
```
$kubectl delete namespace YOUR_NAMESPACE
```

Show help and examples

Use these options as applicable:

To get a list of the most commonly used run options directly in the command line interface, run the show help command.
To show CPM usage examples as well as the list of all the available run options, run this command:
bash
```
$docker run quay.io/newrelic/synthetics-minion:latest help
```
To keep track of Docker logs and verify the health of your monitors, see Containerized private minion (CPM) maintenance and monitoring.
For a CPM in the Kubernetes container orchestration system environment, the following Helm show commands can be used to view the chart.yaml and the values.yaml, respectively:
bash
```
$helm show chart YOUR_REPO_NAME/synthetics-minion
```
bash
```
$helm show values YOUR_REPO_NAME/synthetics-minion
```

Show license information

To show the licensing information for the open source software that we use in the CPM, run the LICENSE command.

Run this command to view license information for CPM versions 2.2.27 or higher:

bash

$docker run quay.io/newrelic/synthetics-minion:latest LICENSE

Some of our open-source software is listed under multiple software licenses, and in that case we have listed the license we've chosen to use. Our license information is also available in the our licenses documentation.

Configure CPM

You can configure the containerized private minion with custom node modules, preserve data between launches, use environment variables, and more. For more information, see CPM configuration.

Networks

For both Docker and Kubernetes, the CPM and its runner containers will inherit network settings from the host. For an example of this on a Docker container system environment, see the Docker site.

A new bridge network is created for each runner container. This means networking command options like --network and --dns passed to the CPM container at launch (such as through Docker run commands on a Docker container system environment) are not inherited or used by the runner containers.

When these networks are created, they pull from the default IP address pool configured for daemon. For an example of this on a Docker container system environment, see the Docker site.

Typically, the runner network is removed after the check is completed. However, if a CPM exits while a check is still running, or exits in another unexpected circumstance, these networks may get orphaned. This can potentially use up IP address space that is available to the Docker daemon.

If this happens, you may see INTERNAL ENGINE ERROR code: 31 entries in your CPM logging when trying to create a new runner container. To clean these up in Docker container system environments only, run docker network prune.

Security, sandboxing, and running as non-root

By default, the software running inside a CPM is executed with root user privileges. This is suitable for most scenarios, as the execution is sandboxed.

In a Docker container system environment: To change the default AppArmor profile used by containers that CPM spawns to run monitors, see the environment variable MINION_RUNNER_APPARMOR (CPM version 3.0.3 or higher) or MINION_DOCKER_RUNNER_APPARMOR (CPM version up to v3.0.2).

To run the CPM as a non-root user, additional steps are required:

For more information, see Docker's official documentation about security and AppArmor security profiles.

If your environment requires you to run the CPM as a non-root user, follow this procedure. In the following example, the non-root user is my_user.

Ensure that my_user can use the Docker engine on the host:
Verify that my_user belongs to the "docker" system group. Scoped root access to the docker.sock is still needed to create bridge networks.
OR
Enable the Docker TCP socket option, and pass the DOCKER_HOST environment variable to CPM.
Verify that my_user has read/write permissions to all the directories and volumes passed to CPM. To set these permission, use the chmod command.
Get the uid of my_user for use in the run command: id -u my_user.
Once these conditions are met, use the option "-u <uid>:<gid>" when launching CPM:
bash
```
$docker run ... -u 1002 ...
```
OR
bash
```
$docker run ... -u 1002 -e DOCKER_HOST=http://localhost:2375 ...
```

Docker image repository

A single CPM Docker image serves both the Docker container system environment and Kubernetes container orchestration system environment. The Docker image is hosted on quay.io. To make sure your Docker image is up-to-date, see the quay.io newrelic/synthetics-minion repository.

Additional considerations for CPM connection

Connection	Description
CPMs without Internet access	A CPM can operate without access to the internet, but with some exceptions. The public internet health check can be disabled using the environment variables named `MINION_NETWORK_HEALTHCHECK_DISABLED` for a Docker container system environment or `synthetics.minionNetworkHealthCheckDisabled` for a Kubernetes container orchestration system environment. The CPM needs to be able to contact the `"synthetics-horde.nr-data.net"` domain. This is necessary for it to report data to New Relic and to receive monitors to execute. Ask your network administration if this is a problem and how to set up exceptions.
Communicate with synthetics via a proxy	To set up communication with New Relic by proxy, use the environment variables named `MINION_API_PROXY*.`
Arguments passed at launch	This applies to a Docker container environment only. Arguments passed to the CPM container at launch do not get passed on to the containers spawned by the CPM. Docker has no concept of "inheritance" or a "hierarchy" of containers, and we don't copy the configuration that is passed from CPM to the monitor-running containers. The only shared configuration between them is the one set at the Docker daemon level.

Install containerized private minions (CPMs)

Important

General private minion features

Kubernetes-specific features

System requirements and compatibility

Caution

Docker container system environment requirements

Caution

Caution

Kubernetes container orchestration system environment requirements (CPM v3.0.0 or higher)

Caution

EOL NOTICE

Private location key

Sandboxing and Docker dependencies

Docker dependencies

Caution

Install and update CPM versions

Start the CPM

Docker start procedure

Kubernetes start procedure

EOL NOTICE

Stop or delete the CPM

Docker stop procedure

Kubernetes delete procedure

Show help and examples

Show license information

Configure CPM

Networks

Security, sandboxing, and running as non-root

Run as non-root user for Docker

Docker image repository

Additional considerations for CPM connection

Install containerized private minions (CPMs)

Important

General private minion features .css-21sua1{background:none;border:none;width:0;padding:0;}

Kubernetes-specific features

System requirements and compatibility

Caution

Kubernetes container orchestration system environment requirements (CPM v3.0.0 or higher)

Private location key

Sandboxing and Docker dependencies

Docker dependencies

Install and update CPM versions

Start the CPM

Docker start procedure

Kubernetes start procedure

Stop or delete the CPM

Docker stop procedure

Kubernetes delete procedure

Show help and examples

Show license information

Configure CPM

Networks

Security, sandboxing, and running as non-root

Run as non-root user for Docker

Docker image repository

Additional considerations for CPM connection

General private minion features