Containerized private minion (CPM) maintenance and monitoring

After installing your containerized private minion (CPM), you can keep track of its maintenance and monitoring in several ways:

You can also get notified of monitor failures with New Relic Alerts.

Check CPM status via Synthetics REST API

Querying a running CPM with the Synthetics REST API is the easiest way to check if it's healthy and working. The container exposes two ports: 8080 and 8180. You can check the CPM with the following endpoints:

  • :8080/status/check: provides details about internal health checks that the minion performs. HTTP 200 means the status is healthy.
  • :8080/status: provides details about a minion's status, which is the same data published in Insights as SyntheticsPrivateMinion event.
  • :8180/: provides JVM application admin endpoints. This is an advanced view of a minion's Java Development Kit (JDK) internal state.

Check if your private location requires more minions

If your private location has multiple monitor checks queued up and you experience delays, you may need more minions available to execute the monitor checks. To learn how to verify this, see Does my private location need more minions?

Review logs

You can monitor your minion's health by looking at CPM container logs.

Docker iconReview Docker logs

This is an example of a CPM log indicating that the minion is working properly in a Docker container system environment:

  $docker logs [YOUR_CONTAINER_NAME]
  2018-10-10 11:33:29,856 - Minion ID: a21f6d7f-4f65-4dec-92fb-88cb975d2a19
  2018-10-10 11:33:29,869 - Publishing resources for Private Minion API: /status/check, /build-info, /status
  2018-10-10 11:33:40,527 - Minion is configured, checking if it is healthy...
  2018-10-10 11:33:43,471 - Launching in PRIVATE Location: 123456-example_private_loc-480
  2018-10-10 11:33:43,723 - Configured 2 heavy worker threads, and 50 light worker threads
  2018-10-10 11:33:43,796 -
  2018-10-10 11:33:43,796 - **************************************************************************
  2018-10-10 11:33:43,796 - * Synthetics Minion is ready and servicing location 'example_private_location'
  2018-10-10 11:33:43,796 - **************************************************************************
  ... logging continues ...
img-integration-k8s@2x.pngReview Kubernetes logs

This is an example of a CPM log indicating that the minion is working properly in a Kubernetes container orchestration system environment:

First, get the name of the CPM pod you want to review logs for:

kubectl get pods -n YOUR_NAMESPACE

Then, interact with that CPM pod:

$ kubectl logs -n YOUR_NAMESPACE YOUR_CPM_NAME
2020-05-11 22:57:24,084 - Minion will use 2 heavy workers
2020-05-11 22:57:24,149 - Minion will use 50 lightweight workers
2020-05-11 22:57:27,973 - Minion Container System: KUBERNETES
2020-05-11 22:57:30,158 - Minion deployment mode: PRIVATE_MINION_POD_KUBERNETES
2020-05-11 22:57:30,178 - No volume mounted at '/var/lib/newrelic/synthetics' in ':rw' mode: Private Minion's ID will change with each boot
2020-05-11 22:57:30,284 - Minion ID: a21f6d7f-4f65-4dec-92fb-88cb975d2a19
2020-05-11 22:57:30,654 - Publishing resources for Private Minion API: /status/check, /build-info, /status
2020-05-11 22:57:31,595 - Minion is configured, checking if it is healthy...
2020-05-11 22:57:35,457 - Launching in PRIVATE Location: 123456-example_private_loc-480
2020-05-11 22:57:36,060 - Executor for async-worker-* threads configured with a max pool size of 16
2020-05-11 22:57:36,072 - Configured 2 heavy worker threads, and 50 lightweight worker threads
2020-05-11 22:57:36,087 -
2020-05-11 22:57:36,087 - **************************************************************************
2020-05-11 22:57:36,087 - * Synthetics Minion 3.0.1 is ready and servicing location 'example_private_location'
2020-05-11 22:57:36,087 - **************************************************************************
2020-05-11 22:57:36,087 -
... logging continues ...

Enable debug logs

If you experience issues with your CPM, you can enable debug logs to help troubleshoot issues.

The default level of logging is set to only inform the user of key information and actionable errors. If this is insufficient, you can enable a more verbose logging by using the MINION_LOG_LEVEL environment variable.

Docker iconEnable Docker debug logs

Adding -f to the Docker logs makes the command follow logs.

 
  docker run ... -e MINION_LOG_LEVEL=DEBUG ...
  docker logs -f YOUR_CONTAINER_NAME
  ... verbose logging continues ...
img-integration-k8s@2x.pngEnable Kubernetes debug logs

Adding -f to the Kubernetes logs makes the command follow logs.

To enable DEBUG logs add the --set synthetics.minionLogLevel=DEBUG option when running your helm install:

helm install YOUR_CPM_NAME YOUR_REPO_NAME/synthetics-minion -n YOUR_NAMESPACE --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY --set synthetics.minionLogLevel=DEBUG

Get the name of the CPM pod you want to review logs for:

kubectl get pods -n YOUR_NAMESPACE

Then, interact with that CPM pod:

kubectl logs -f -n YOUR_NAMESPACE YOUR_CPM_POD_NAME

... verbose logging continues ...

Retrieve Kubernetes debugging information

If you experience issues with your CPM in a Kubernetes container orchestration system environment, you can retrieve information about the CPM pod and the node it is running on to help troubleshoot.

To retrieve information for the CPM pod:

kubectl describe pod -n YOUR_NAMESPACE YOUR_CPM_POD_NAME

To retrieve information for the node the CPM pod is running on, identify the node, and then:

kubectl describe node NODE_ASSOCIATED_WITH_YOUR_CPM_POD_NAME

Monitor CPMs with New Relic Infrastructure

New Relic Infrastructure supports advanced Docker monitoring and advanced Kubernetes monitoring. To add support for this, Synthetics labels the containers spawned by CPM with a series of informative labels, all prefixed with synthetics-minion-. You can use these labels to identify CPM-spawned containers that are running your monitors. Example labels include:

  • synthetics-minion-runner-role
  • synthetics-minion-runner-version
  • synthetics-minion-container-id
  • synthetics-minion-id
  • synthetics-minion-build-number
  • synthetics-minion-job
  • synthetics-minion-account
  • synthetics-minion-monitor
  • synthetics-minion-monitor-version
  • synthetics-minion-monitor-type
  • synthetics-minion-monitor-type-label

For more help

Recommendations for learning more: