Install containerized private minions (CPMs)

Private minions and locations are an add-on feature for paid Synthetics accounts. You may not modify any CPM files and New Relic is not liable for any modifications you make. For more information, contact your account representative or a New Relic technical sales rep.

This document describes the New Relic containerized private minion (CPM), a Docker container based private minion that accepts and executes Synthetics monitors against your private locations.

The CPM can operate in a Docker container system environment or a Kubernetes container orchestration system environment. The CPM will auto-detect its environment to select the appropriate operating mode.

General private minion features

Because the CPM operates as a container instead of a virtual machine, it delivers many features:

  • Easy to install, start, and update
  • Runs on:
  • Enhanced security and support for non-root user execution
  • Ability to leverage a Docker container as a sandbox environment
  • Customizable monitor check timeout
  • Custom provided modules for scripted monitor types

Kubernetes-specific features

Also, the CPM delivers the following features in a Kubernetes environment:

  • Integrates with the Kubernetes API to delegate runtime lifecycle management to Kubernetes
  • Does not require privileged access to the Docker socket
  • Supports hosted and on-premise Kubernetes clusters
  • Supports various container engines such as Docker and Containerd
  • Deployable via Helm charts as well as configuration YAMLs
  • Allows job (ping vs. non-ping checks) based resource allocation for optimum resource management
  • Observability offered via the New Relic One Kubernetes cluster explorer

System requirements and compatibility

To host CPMs, your system must meet the minimum requirements for the chosen system environment.

Docker iconDocker container system environment requirements
Compatibility for Requirements
Operating system

Linux kernel: 3.10 or higher

macOS: 10.11 or higher

Windows: Windows 10 64-bit or higher

Processor A modern, multi-core CPU
Memory 2.5 GB of RAM per CPU core (dedicated)
Disk space A minimum of 10 GB per host
Docker version Docker 17.03 or higher
Private location key You must have a private location key
img-integration-k8s@2x.pngKubernetes container orchestration system environment requirements (CPM v3.0.0 or higher)
Compatibility for Requirements
Operating system

Linux kernel: 3.10 or higher

macOS: 10.11 or higher

Windows: Windows 10 64-bit or higher

Processor A modern, multi-core CPU
Minion pod

CPU (vCPU/Core): 0.5 up to 0.75

Memory: 800 Mi up to 1.6 Gi

Resources allocated to a Minion pod are user configurable.

Runner pod

CPU (vCPU/Core): 0.5 up to 1

Memory: 1.25 Gi up to 3 Gi

  • For a Scripted API check, 1.25 Gi will be requested with a limit of 2.5 Gi.

  • For a Simple Browser or Scripted Browser check, 2 Gi will be requested with a limit of 3 Gi.

Additional considerations:

  • Resources allocated to a Runner pod are not user configurable.

  • The maximum limit-request resource ratio for both CPU and Memory is 2.

Disk space Persistent volume of at least 3 Gi in size
Kubernetes version We recommend that your Kubernetes cluster supports Kubernetes v1.15.
Private location key You must have a private location key
Helm Follow installation instructions installation instructions for Helm v3 for your OS.
Kubectl Follow installation instructions for Kubectl for your OS.

Private location key

Before launching CPMs, you must have a private location key. Your CPM uses the key to authenticate against New Relic Synthetics and run monitors associated with that private location.

To find the key for existing private location:

  1. Go to synthetics.newrelic.com > Private locations.
  2. In the Private locations index, locate the private location you want your CPM to be assigned to.
  3. Note the key associated with the private location with the key key icon.

Sandboxing and Docker dependencies

Sandboxing and Docker dependencies are applicable to the CPM in a Docker container system environment.

Docker iconDocker dependencies

The CPM runs in Docker and is able to leverage Docker as a sandboxing technology. This ensures complete isolation of the monitor execution, which improves security, reliability, and repeatability. Every time a scripted or browser monitor is executed, the CPM creates a brand new Docker container to run it in called a runner.

The minion container needs to be configured to communicate with the Docker engine in order to spawn additional runner containers. Each spawned container is then dedicated to run a check associated with the Synthetics monitor running on the private location the minion container is associated with.

There are two crucial dependencies at launch. To enable sandboxing, ensure that:

  • Your writable and executable directory is mounted at /tmp. The writable directory can be any directory you want the CPM to write into, but New Relic recommends the system's own /tmp to make things easy.
  • Your writable Docker UNIX socket is mounted at /var/run/docker.sock or DOCKER_HOST environment variable. For more information, see Docker's Daemon socket option.

Core count on the host determines how many runner containers the CPM can run concurrently on the host. Since memory requirements are scaled to the expected count of runner containers, we recommend not running multiple CPMs on the same host to avoid resource contention.

For additional information on sandboxing and running as a root or non-root user, see Security, sandboxing, and running as non-root.

Install and update CPM versions

Both installing and updating the CPM use the same command to pull the latest Docker image from the Quay.io repository where the CPM Docker image is hosted. Go to quay.io/repository/newrelic/synthetics-minion for a list of all the releases.

CPM images are also hosted on Docker Hub. Go to hub.docker.com/r/newrelic/synthetics-minion/tags for a list of all the releases.

Start the CPM

To start the CPM, follow the applicable Docker or Kubernetes instructions.

Docker icon Docker start procedure
  1. Locate your private location key.
  2. Ensure you've enabled Docker dependencies for sandboxing and installed CPM on your system.
  3. Run the appropriate script for your system. Tailor the common defaults for /tmp and /var/run/docker.sock in the following examples to match your system.
    • Linux/macOS:

        docker run \
          --name YOUR_CONTAINER_NAME \
          -e "MINION_PRIVATE_LOCATION_KEY=YOUR_PRIVATE_LOCATION_KEY" \
          -v /tmp:/tmp:rw \
          -v /var/run/docker.sock:/var/run/docker.sock:rw \
          quay.io/newrelic/synthetics-minion:latest
      
    • Windows:

        docker run ^
          --name YOUR_CONTAINER_NAME ^
          -e "MINION_PRIVATE_LOCATION_KEY=YOUR_PRIVATE_LOCATION_KEY" ^
          -v /tmp:/tmp:rw ^
          -v /var/run/docker.sock:/var/run/docker.sock:rw ^
          quay.io/newrelic/synthetics-minion:latest
      

When a message similar to Synthetics Minion is ready and servicing location YOUR_PRIVATE_LOCATION_LABEL appears, your CPM is up and ready to run monitors assigned to that location.

img-integration-k8s@2x.png Kubernetes start procedure
  1. Locate your private location key.
  2. Set up the a namespace for the CPM in your Kubernetes cluster:

    kubectl create namespace YOUR_NAMESPACE
  3. Copy the Helm charts from the New Relic Helm repo.
    • If you are copying the charts for the first time:
      helm repo add YOUR_REPO_NAME https://helm-charts.newrelic.com/charts
    • If you previously copied the Helm charts from the New Relic Helm repo, then get the latest:

      helm repo update
  4. Install the CPM with the following Helm command:

    • For a fresh installation of the CPM:

      helm install YOUR_CPM_NAME YOUR_REPO_NAME/synthetics-minion -n YOUR_NAMESPACE --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY
    • To update an existing CPM installation:

      helm upgrade YOUR_CPM_NAME YOUR_REPO_NAME/synthetics-minion -n YOUR_NAMESPACE --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY
  5. Check if the minion pod is up and running:

    kubectl get -n YOUR_NAMESPACE pods

Once the status attribute of each pod is shown as running, your CPM is up and ready to run monitors assigned to your private location.

Stop or delete the CPM

On a Docker container system environment, use the Docker stop procedure to stop the CPM from running. On a Kubernetes container orchestration system environment, use the Kubernetes delete procedure to stop the CPM from running.

Docker iconDocker stop procedure

You can stop a Docker container either by the container name, or the container ID.

  • Container name stop for Linux, macOS, and Windows:

      docker stop YOUR_CONTAINER_NAME
      docker rm YOUR_CONTAINER_NAME
    
  • Container ID stop for Linux/macOS:

    In the examples the container is stopped and removed. To only stop the container, omit docker rm $CONTAINER_ID.

      CONTAINER_ID=$(docker ps -aqf name=YOUR_CONTAINER_NAME)
      docker stop $CONTAINER_ID
      docker rm $CONTAINER_ID
    
  • Container ID stop for Windows:

    In the examples the container is stopped and removed. To only stop the container, omit docker rm $CONTAINER_ID.

      FOR /F "tokens=*" %%CID IN ('docker ps -aqf name=YOUR_CONTAINER_NAME') do (SET CONTAINER_ID=%%CID)
      docker stop %CONTAINER_ID%
      docker rm %CONTAINER_ID%
    
img-integration-k8s@2x.pngKubernetes delete procedure
  1. Get the MINION_POD_INSTALLATION_NAME of the minion pod you want to delete:

    helm list -n YOUR_NAMESPACE
  2. Delete the minion pod:

    helm uninstall -n YOUR_NAMESPACE MINION_POD_INSTALLATION_NAME
    
  3. Delete the namespace set up for the CPM in your Kubernetes cluster:

     kubectl delete namespace YOUR_NAMESPACE

Show help and examples

Use these options as applicable:

  • To get a list of the most commonly used run options directly in the command line interface, run the show help command.
  • To show CPM usage examples as well as the list of all the available run options, run this command:

      docker run quay.io/newrelic/synthetics-minion:latest help
  • To keep track of Docker logs and verify the health of your monitors, see Containerized private minion (CPM) maintenance and monitoring.

Show license information

To show the licensing information for the open source software that we use in the CPM, run the LICENSE command.

Run this command to view license information for CPM versions 2.2.27 or higher:

docker run quay.io/newrelic/synthetics-minion:latest LICENSE

Some of our open-source software is listed under multiple software licenses, and in that case we have listed the license we've chosen to use. Our license information is also available in the New Relic Synthetics licenses documentation.

Configure CPM

You can configure the containerized private minion with custom npm modules, preserve data between launches, use environment variables, and more. For more information, see CPM configuration.

Networks

For both Docker and Kubernetes, the CPM and its runner containers will inherit network settings from the host. For an example of this on a Docker container system environment, see the Docker site.

A new bridge network is created for each runner container. This means networking command options like --network and --dns passed to the CPM container at launch (such as through Docker run commands on a Docker container system environment) are not inherited or used by the runner containers.

When these networks are created, they pull from the default IP address pool configured for daemon. For an example of this on a Docker container system environment, see the Docker site.

Typically, the runner network is removed after the check is completed. However, if a CPM exits while a check is still running, or exits in another unexpected circumstance, these networks may get orphaned. This can potentially use up IP address space that is available to the Docker daemon.

If this happens, you may see INTERNAL ENGINE ERROR code: 31 entries in your CPM logging when trying to create a new runner container. To clean these up in Docker container system environments only, run docker network prune.

Security, sandboxing, and running as non-root

By default, the software running inside a CPM is executed with root user privileges. This is suitable for most scenarios, as the execution is sandboxed.

In a Docker container system environment: To change the default AppArmor profile used by containers that CPM spawns to run monitors, see the environment variable MINION_RUNNER_APPARMOR (CPM version 3.0.3 or higher) or MINION_DOCKER_RUNNER_APPARMOR (CPM version up to v3.0.2).

To run the CPM as a non-root user, additional steps are required:

Docker iconRun as non-root user for Docker

For more information, see Docker's official documentation about security and AppArmor security profiles.

If your environment requires you to run the CPM as a non-root user, follow this procedure. In the following example, the non-root user is my_user.

  1. Ensure that my_user can use the Docker engine on the host:

    Verify that my_user belongs to the "docker" system group.

    OR

    Enable the Docker TCP socket option, and pass the DOCKER_HOST environment variable to CPM.

  2. Verify that my_user has read/write permissions to all the directories and volumes passed to CPM. To set these permission, use the chmod command.
  3. Get the uid of my_user for use in the run command: id -u my_user.

Once these conditions are met, use the option "-u <uid>:<gid>" when launching CPM:

  docker run ... -u 1002 ...

OR

  docker run ... -u 1002 -e DOCKER_HOST=http://localhost:2375 ...

Docker image repository

A single CPM Docker image serves both the Docker container system environment and Kubernetes container orchestration system environment. The Docker image is hosted on quay.io. To make sure your Docker image is up-to-date, see the quay.io newrelic/synthetics-minion repository.

Additional considerations for CPM connection

Connection Description
CPMs without Internet access A CPM can operate without access to the internet, but with some exceptions. The CPM needs to be able to contact the "synthetics-horde.nr-data.net" domain. This is necessary for it to report data to New Relic Synthetics and to receive monitors to execute. Ask your network administration if this is a problem and how to set up exceptions.
Communicate with Synthetics via a proxy

To set up communication with New Relic Synthetics by proxy, use the environment variables named MINION_API_PROXY*.

Arguments passed at launch This applies to a Docker container environment only. Arguments passed to the CPM container at launch do not get passed on to the containers spawned by the CPM. Docker has no concept of "inheritance" or a "hierarchy" of containers, and we don't copy the configuration that is passed from CPM to the monitor-running containers. The only shared configuration between them is the one set at the Docker daemon level.

For more help

Recommendations for learning more: