Synthetics job manager configuration

This doc will guide you through configuring your synthetics job manager by showing you how to:

Use environment variables to configure your synthetics job manager.
Set up custom modules for scripted API or scripted browser monitors.
Provide user-defined variables in your configuration.

Configuration using environment variables

Environmental variables allow you to fine-tune the synthetics job manager configuration to meet your specific environmental and functional needs.

The variables are provided at startup using the -e, --env argument.

The following table shows all the environment variables that synthetics job manager supports. PRIVATE_LOCATION_KEY is required, and all other variables are optional.

Name	Description
`PRIVATE_LOCATION_KEY`	REQUIRED. Private location key, as found on the Private Location entity list.
`DOCKER_API_VERSION`	Format: `"vX.Y"` API version to be used with the given Docker service. Default: `v1.35.`
`DOCKER_HOST`	Points the synthetics job manager to a given `DOCKER_HOST`. If absent, the default value is `/var/run/docker.sock.`
`HORDE_API_ENDPOINT`	For US-based accounts, the endpoint is: `https://synthetics-horde.nr-data.net.` For EU-based accounts, the endpoint is: `https://synthetics-horde.eu01.nr-data.net/` Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor.
`DOCKER_REGISTRY`	The Docker Registry domain where the runtime images are hosted. Use this to override `docker.io` as the default.
`DOCKER_REPOSITORY`	The Docker repository or organization where the runtime images are hosted. Use this to override `newrelic` as the default.
`HORDE_API_PROXY_HOST`	Proxy server host used for Horde communication. Format: `"localhost"`.
`HORDE_API_PROXY_PORT`	Proxy server port used for Horde communication. Format: `8888`.
`HORDE_API_PROXY_USERNAME`	Proxy server username used for Horde communication. Format: `"username"`.
`HORDE_API_PROXY_PW`	Proxy server password used for Horde communication. Format: `"password"`.
`HORDE_API_PROXY_ACCEPT_SELF_SIGNED_CERT`	Accept self signed proxy certificates for the proxy server connection used for Horde communication? Acceptable values: `true`
`CHECK_TIMEOUT`	The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes). Default: 180 seconds
`LOG_LEVEL`	Default: `INFO.` Additional options: `WARN`, `ERROR`, `DEBUG`
`HEAVYWEIGHT_WORKERS`	The number of concurrent heavyweight jobs (Browser/Scripted Browser and Scripted API) that can run at one time. Default: Available CPUs - 1.
`DESIRED_RUNTIMES`	An array that may be used to run specific runtime images. Format: ['newrelic/synthetics-ping-runtime:latest','newrelic/synthetics-node-api-runtime:latest','newrelic/synthetics-node-browser-runtime:latest'] Default: all latest runtimes.
`VSE_PASSPHRASE`	If set, enables verified script execution and uses this value as a passphrase.
`USER_DEFINED_VARIABLES`	A locally hosted set of user defined key value pairs.
`ENABLE_WASM`	If set, enables webassembly for node browser runtime. To use webassembly, your synthetics job manager minimum version should be release-367 or higher and node browser runtime version should be 2.3.21 or higher.

The variables are provided at startup using the --set argument.

The following list shows all the environment variables that synthetics job manager supports. synthetics.privateLocationKey is required, and all other variables are optional.

A number of additional advanced settings are available and fully documented in our Helm chart README

Name	Description
`synthetics.privateLocationKey`	REQUIRED if `synthetics.privateLocationKeySecretName` is not set. Private location key of the private location, as found on the private location web page.
`synthetics.privateLocationKeySecretName`	REQUIRED if `synthetics.privateLocationKey` is not set. Name of the Kubernetes secret that contains the key `privateLocationKey`, which contains the authentication key associated with your synthetics private location.
`imagePullSecrets`	The name of the secret object used to pull an image from a specified container registry.
`fullnameOverride`	Name override used for your Deployment, replacing the default.
`appVersionOverride`	Release version of synthetics-job-manager to use instead of the version specified in chart.yml.
`synthetics.logLevel`	Default: `INFO.` Additional options: `WARN`, `ERROR`
`synthetics.hordeApiEndpoint`	For US-based accounts, the endpoint is: `https://synthetics-horde.nr-data.net.` For EU-based accounts, the endpoint is: `https://synthetics-horde.eu01.nr-data.net/` Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor.
`synthetics.minionDockerRunnerRegistryEndpoint`	The Docker Registry and Organization where the Minion Runner image is hosted. Use this to override `quay.io/newrelic` as the default (for example, `docker.io/newrelic`)
`synthetics.vsePassphrase`	If set, it enables verified script execution, and uses this value as a passphrase.
`synthetics.vsePassphraseSecretName`	If set, enables verified script execution and uses this value to retrieve the passphrase from a Kubernetes secret with a key called `vsePassphrase`.
`synthetics.enableWasm`	If set, enables webassembly for node browser runtime. To use webassembly, your synthetics job manager minimum version should be release-367 or higher and node browser runtime version should be 2.3.21 or higher.
`synthetics.apiProxyHost`	Proxy server used for Horde communication. Format: `"host"`.
`synthetics.apiProxyPort`	Proxy server port used for Horde communication. Format: `port`.
`synthetics.hordeApiProxySelfSignedCert`	Accept self signed certificates when using a proxy server for Horde communication. Acceptable values: `true`.
`synthetics.hordeApiProxyUsername`	Proxy server username for Horde communication. Format: `"username"`
`synthetics.hordeApiProxyPw`	Proxy server password for Horde communication. Format: `"password"`.
`synthetics.userDefinedVariables.userDefinedJson`	A JSON string of user-defined variables. The user may access these variables in their script. Format: `'{"key":"value","key2":"value2"}'`.
`synthetics.userDefinedVariables.userDefinedFile`	A path local to the user to a JSON file containing user-defined variables. This is passed in via `--set-file` and cannot be set in the Values file.
`synthetics.userDefinedVariables.userDefinedPath`	A path on the user's provided PersistentVolume to the user_defined_variables.json file. User must provide a PersistentVolume or PersistentVolumeClaim if this variable is populated.
`synthetics.persistence.existingClaimName`	If mounting a volume, the user may provide a name for a PersistentVolumeClaim that already exists in the cluster. Presumes the existence of an corresponding PersistentVolume.
`synthetics.persistence.existingVolumeName`	If mounting a volume and not providing a PersistentVolumeClaim, the user must at minimum provide a PersistentVolume name. Helm will generate a PersistentVolumeClaim.
`synthetics.persistence.storageClass`	The name of the StorageClass for the generated PersistentVolumeClaim. This should match the StorageClassName on the existing PV. If not providers, Kubernetes will use the default storage class if present.
`synthetics.persistence.size`	The size of the volume for the generated PersistentVolumeClaim. Format: `10Gi`. Default 2Gi.
`global.checkTimeout`	The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes). Default: 180 seconds
`image.repository`	The container to pull. Default: `docker.io/newrelic/synthetics-job-runner`
`image.pullPolicy`	The pull policy. Default: `IfNotPresent`
`podSecurityContext`	Set a custom security context for the synthetics-job-manager pod.
`ping-runtime.enabled`	Whether or not the persistent ping runtime should be deployed. This can be disabled if you do not use ping monitors. Default: `true`
`ping-runtime.replicaCount`	The number of ping runtime containers to deploy. Increase the replicaCount to scale the deployment based on your ping monitoring needs. Default: `1`
`ping-runtime.image.repository`	The container image to pull for the ping runtime. Default: `docker.io/newrelic/synthetics-ping-runtime`
`ping-runtime.image.pullPolicy`	The pull policy for the ping-runtime container. Default: `IfNotPresent`
`node-api-runtime.enabled`	Whether or not the Node.js API runtime should be deployed. This can be disabled if you do not use scripted API monitors. Default: `true`
`node-api-runtime.parallelism`	The number of Node.js API runtime `CronJobs` to deploy. The maximum number of concurrent Node.js API jobs that will execute at any time. Additional details. Default: `1`
`node-api-runtime.completions`	The number of Node.js API runtime `CronJobs` to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. . Increase this setting if you notice periods of time with no API runtime jobs running. Additional details. Default: `6`
`node-api-runtime.image.repository`	The container image to pull for the Node.js API runtime. Default: `docker.io/newrelic/synthetics-node-api-runtime`
`node-api-runtime.image.pullPolicy`	The pull policy for the Node.js API runtime container. Default: `IfNotPresent`
`node-browser-runtime.enabled`	Whether or not the Node.js browser runtime should be deployed. This can be disabled if you do not use simple or scripted browser monitors. Default: `true`
`node-browser-runtime.parallelism`	The number of Chrome browser runtime `CronJobs` to deploy. The maximum number of concurrent Chrome browser jobs that will execute at any time. Additional details. Default: `1`
`node-browser-runtime.completions`	The number of Chrome browser runtime `CronJobs` to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. Increase this setting if you notice periods of time with no browser runtime jobs running. Additional details. Default: `6`
`node-browser-runtime.image.repository`	The container image to pull for the Node.js browser runtime. Default: `docker.io/newrelic/synthetics-node-browser-runtime`
`node-browser-runtime.image.pullPolicy`	The pull policy for the Node.js browser runtime container. Default: `IfNotPresent`

User-defined variables for scripted monitors

Private synthetics job managers let you configure environment variables for scripted monitors. These variables are managed locally on the SJM and can be accessed via $env.USER_DEFINED_VARIABLES. You can set user-defined variables in two ways. You can mount a JSON file or you can supply an environment variable to the SJM on launch. If both are provided, the SJM will only use values provided by the environment.

The user may create a JSON-formatted file and mount the volume where the file is located to a specified target path in the SJM container.

The file must have read permissions and contain a JSON-formatted map. Example user-defined variables file:

{
  "KEY": "VALUE",
  "user_name": "MINION",
  "my_password": "PASSW0RD123",
  "my_URL": "https://newrelic.com/",
  "ETC": "ETC"
}

Place the file in the source directory on the host. The SJM is expecting the file name to be user_defined_variables.json

Docker example:

The expected target directory is: /var/lib/newrelic/synthetics/variables/

bash

$docker run ... -v /variables:/var/lib/newrelic/synthetics/variables:rw ...

Kubernetes example:

The user has two options when providing a file to the SJM pod in Kubernetes. They may:

Pass in a local file.
Provide a PersistentVolume that includes the user_defined_variables.json.

Pass in a local file

This option creates a ConfigMap Kubernetes resource and mounts that to the SJM pod.

bash

$helm install newrelic/synthetics-job-manager ... --set-file "synthetics.userDefinedVariables.userDefinedFile=[local-path]/user_defined_variables.json" ...

Mount a `PersistentVolume`

This option requires the user to provide a PersistentVolume that includes the user_defined_variables.json file or a PersistentVolumeClaim to the same. For more details on helm chart installation using a PersistentVolume, follow the instructions at permanent data storage.

Once the user has prepared a PersistentVolume as described below, launch the SJM, setting the path where the user_defined_variables.json file is located, and set any other synthetics.persistence variables as necessary.

bash

$helm install newrelic/synthetics-job-manger ... --set synthetics.userDefinedVariables.userDefinedPath="variables"

The variables may be passed to their respective container system via environment variable.

Docker example:

Use the -e flag to set up an environment variable named USER_DEFINED_VARIABLES and give it the value of a JSON formatted map string.

bash

$docker run ... -e USER_DEFINED_VARIABLES='{"key":"value","name":"sjm"}' ...

Kubernetes example:

Use the --set-literal flag to pass in the JSON formatted string.

bash

$helm install newrelic/synthetics-job-manager ... --set-literal synthetics.userDefinedVariables.userDefinedJson='{"key":"value","name":"sjm"}' ...

Accessing user-defined environment variables from scripts

To reference a configured user-defined environment variable, use the reserved $env.USER_DEFINED_VARIABLES followed by the name of a given variable with dot notation (for example, $env.USER_DEFINED_VARIABLES.MY_VARIABLE).

Caution

User-defined environment variables are not sanitized from logs. Consider using the secure credentials feature for sensitive information.

Custom node modules

Custom node modules are provided in both CPM and SJM. They allow you to create a customized set of node modules and use them in scripted monitors (scripted API and scripted browser) for synthetic monitoring.

Set up your custom modules directory

Create a directory with a package.json file following npm official guidelines in the root folder. The SJM will install any dependencies listed in the package.json's dependencies field. These dependencies will be available when running monitors on the private synthetics job manager. See an example of this below.

Example

In this example, a custom module directory is used with the following structure:

/example-custom-modules-dir/
    ├── counter
    │   ├── index.js
    │   └── package.json
    └── package.json            ⇦ the only mandatory file

The package.json defines dependencies as both a local module (for example, counter) and any hosted modules (for example, smallest version 1.0.1):

{
    "name": "custom-modules",
    "version": "1.0.0",                                ⇦ optional
    "description": "example custom modules directory", ⇦ optional
    "dependencies": {
    "smallest": "1.0.1",                               ⇦ hosted module
    "counter": "file:./counter"                        ⇦ local module
    }
}

Add your custom modules directory to the SJM for Docker or Kubernetes

For Kubernetes, the directory at /var/lib/newrelic/synthetics/modules needs to exist on a PV prior to launching the SJM with custom modules enabled.

Tip

The PV access mode should be ReadWriteMany if you need to share storage across multiple pods.

One method is to create a pod that mounts the PV just for the purpose of copying your custom modules directory to the PV. The following example uses Amazon EFS with Amazon EKS:

Create the namespace, persistent volume, and persistent volume claim

Make sure you've already set up your EFS filesystem and installed the EFS CSI driver on your cluster. You will also need your EFS filesystem ID for the PV's spec.csi.volumeHandle.

bash

$kubectl apply -f - <<EOF
$apiVersion: v1
$kind: Namespace
$metadata:
$  name: newrelic
$
$---
$kind: StorageClass
$apiVersion: storage.k8s.io/v1
$metadata:
$  name: efs-sc
$provisioner: efs.csi.aws.com
$
$---
$apiVersion: v1
$kind: PersistentVolume
$metadata:
$  name: custom-modules-pvc
$spec:
$  capacity:
$    storage: 5Gi
$  volumeMode: Filesystem
$  accessModes:
$    - ReadWriteMany
$  persistentVolumeReclaimPolicy: Retain
$  storageClassName: efs-sc
$  csi:
$    driver: efs.csi.aws.com
$    volumeHandle: <your-efs-filesystem-id>
$
$---
$apiVersion: v1
$kind: PersistentVolumeClaim
$metadata:
$  name: custom-modules-pvc
$  namespace: newrelic
$spec:
$  accessModes:
$    - ReadWriteMany
$  storageClassName: efs-sc
$  resources:
$    requests:
$      storage: 5Gi
$EOF

Switch to the newrelic namespace in your ~/.kube/config.

bash

$kubectl config get-contexts
$kubectl config set-context YOUR_CONTEXT --namespace=newrelic
$kubectl config view --minify | grep namespace:

At this point, the PVC should be bound to the PV with RWX access mode.

bash

$kubectl get pv,pvc
NAME                                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/custom-modules-pvc   5Gi        RWX            Retain           Bound    newrelic/custom-modules-pvc   efs-sc         <unset>                          4m46s
NAME                                       STATUS   VOLUME               CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/custom-modules-pvc   Bound    custom-modules-pvc   5Gi        RWX            efs-sc         <unset>                 4m10s

Create `mount-custom-mods-pod` to copy your custom-modules directory

bash

$kubectl apply -f - <<EOF
$apiVersion: v1
$kind: Pod
$metadata:
$  name: mount-custom-mods-pod
$spec:
$  containers:
$  - name: mount-custom-mods-pod
$    image: nginx
$    resources:
$      requests:
$        memory: "64Mi"
$        cpu: "250m"
$      limits:
$        memory: "128Mi"
$        cpu: "500m"
$    volumeMounts:
$      - mountPath: "/var/lib/newrelic/synthetics/modules"
$        name: custom-modules-storage
$  volumes:
$  - name: custom-modules-storage
$    persistentVolumeClaim:
$      claimName: custom-modules-pvc
$EOF

At this point, the mount-custom-mods-pod should be created and configured to use the volume.

bash

$kubectl describe po mount-custom-mods-pod | grep -A4 Volumes:
Volumes:
  custom-modules-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  custom-modules-pvc
    ReadOnly:   false

Check events for any warnings related to the PV, PVC, or mount-custom-mods-pod.

bash

$kubectl get events --field-selector type=Warning --sort-by='.lastTimestamp'

Copy your custom-modules directory to the PV

It's not necessary to copy node_modules as it will be generated by the SJM on npm install.

bash

$cd custom-modules
$rm -rf node_modules && cd ..

Check that the mount-custom-mods-pod is running.

bash

$kubectl get po
NAME                    READY   STATUS    RESTARTS   AGE
mount-custom-mods-pod   1/1     Running   0          5m43s

Copy to the PV.

bash

$kubectl cp custom-modules newrelic/mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules

Check that /var/lib/newrelic/synthetics/modules/custom-modules/package.json exists on the PV.

bash

$kubectl exec -it mount-custom-mods-pod -- bash
root@mount-custom-mods-pod:/# cd /var/lib/newrelic/synthetics/modules/
root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# ls -l
total 4
drwxr-xr-x 2 root root 6144 Jun 29 03:49 custom-modules
root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# ls -l custom-modules/
total 4
-rw-r--r-- 1 501 staff 299 Jun 29 03:49 package.json

Launch the SJM with custom modules feature enabled

Set values for persistence.existingClaimName and customNodeModules.customNodeModulesPath either in the command line or in a YAML file during installation. The customNodeModules.customNodeModulesPath value should specify the subpath on the Persistent Volume where your custom modules files exist. For example:

bash

$helm upgrade --install synthetics-job-manager newrelic/synthetics-job-manager -n newrelic --set global.persistence.existingClaimName=custom-modules-pvc --set global.customNodeModules.customNodeModulesPath=custom-modules --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY
Release "synthetics-job-manager" does not exist. Installing it now.
NAME: synthetics-job-manager
LAST DEPLOYED: Fri Jun 28 16:53:28 2024
NAMESPACE: newrelic
STATUS: deployed
REVISION: 1
TEST SUITE: None

The custom-modules directory should now contain the installed packages in node_modules.

bash

$kubectl exec -it mount-custom-mods-pod -- bash
root@mount-custom-mods-pod:/# cd /var/lib/newrelic/synthetics/modules/
root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# ls -l custom-modules/
total 16
-rw-r--r--  1 root root   836 Jun 29 03:51 README
drwxr-xr-x 18 root root  6144 Jun 29 03:51 node_modules
-rw-r--r--  1  501 staff  299 Jun 29 03:49 package.json
-rw-r--r--  1 root root   190 Jun 29 03:51 package.json.shasum

If custom node modules are not detected, adjust permissions on the custom-modules directory and package.json file.

bash

$kubectl exec -it mount-custom-mods-pod -- bash
root@mount-custom-mods-pod:/# cd /var/lib/newrelic/synthetics/modules/
root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# chmod -R 777 custom-modules
root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# chown -R 2000:2000 custom-modules

To check if the modules were installed correctly or if any errors occurred, look for the following lines in the synthetics-job-manager container or pod logs:

2024-06-29 03:51:28,407{UTC} [main] INFO  c.n.s.j.p.options.CustomModules - Detected mounted path for custom node modules
2024-06-29 03:51:28,408{UTC} [main] INFO  c.n.s.j.p.options.CustomModules - Validating permission for custom node modules package.json file
2024-06-29 03:51:28,409{UTC} [main] INFO  c.n.s.j.p.options.CustomModules - Installing custom node modules...
2024-06-29 03:51:44,670{UTC} [main] INFO  c.n.s.j.p.options.CustomModules - Custom node modules installed successfully.

Now you can add "require('smallest');" into the script of monitors you send to this private location.

Change `package.json` for custom modules

In addition to local and hosted modules, you can utilize Node.js modules as well. To update the custom modules used by your SJM, make changes to the package.json file, and restart the SJM. During the reboot process, the SJM will recognize the configuration change and automatically perform cleanup and re-installation operations to ensure the updated modules are applied.

Caution

Local modules: While your package.json can include any local module, these modules must reside inside the tree under your custom module directory. If stored outside the tree, the initialization process will fail and you will see an error message in the docker logs after launching SJM.

Permanent data storage

Users may want to use permanent data storage to provide the user_defined_variables.json file or support custom node modules.

Docker

To set permanent data storage on Docker:

Create a directory on the host where you are launching the Job Manager. This is your source directory.
Launch the Job Manager, mounting the source directory to the target directory /var/lib/newrelic/synthetics.
Example:
bash
```
$docker run ... -v /sjm-volume:/var/lib/newrelic/synthetics:rw ...
```

Kubernetes

To set permanent data storage on Kubernetes, the user has two options:

Provide an existing PersistentVolumeClaim (PVC) for an existing PersistentVolume (PV), setting the synthetics.persistence.existingClaimName configuration value. Example:
bash
```
$helm install ... --set synthetics.persistence.existingClaimName=sjm-claim ...
```
Provide an existing PersistentVolume (PV) name, setting the synthetics.persistence.existingVolumeName configuration value. Helm will generate a PVC for the user. The user may optionally set the following values as well:

synthetics.persistence.storageClass: The storage class of the existing PV. If not provided, Kubernetes will use the default storage class.

synthetics.persistence.size: The size for the claim. If not set, the default is currently 2Gi.

bash

$helm install ... --set synthetics.persistence.existingVolumeName=sjm-volume --set synthetics.persistence.storageClass=standard ...

Sizing considerations for Kubernetes and Docker

Tip

Docker specific sizing considerations will be available soon.

If you're working in larger environments, you may need to customize the job manager configuration to meet minimum requirements to execute synthetic monitors efficiently. Many factors can impact sizing requirements for a synthetics job manager deployment, including:

If all runtimes are required based on expected usage
The number of jobs per minute by monitor type (ping, simple or scripted browser, and scripted API)
Job duration, including jobs that time out at around 3 minutes
The number of job failures. For job failures, automatic retries are scheduled when a monitor starts to fail to provide built-in 3/3 retry logic. These additional jobs add to the throughput requirements of the synthetic job manager.

In addition to the sizing configuration settings listed below, additional synthetics job managers can be deployed with the same private location key to load balance jobs across multiple environments.

Kubernetes

Each runtime used by the Kubernetes synthetic job manager can be sized independently by setting values in the helm chart.

Additional ping runtimes can be started to help execute ping monitor load by increasing the ping-runtime.replicaCount setting from the default value of 1.

The Node.js API and Node.js Browser runtimes are sized independently using a combination of the parallelism and completions settings. Ideal configurations for these settings will vary based on customer requirements.

The parallelism setting controls how many pods of a particular runtime run concurrently. The parallelism setting is the equivalent of the synthetics.heavyWorkers configuration in the containerized private minion (CPM). Ensure that your Kubernetes cluster has enough resources available to run this number of pods based on their resource request and limit values.

The completions setting controls how many pods of a particular runtime must complete before the CronJob can start another Kubernetes Job for that runtime. Note the difference between a Kubernetes Job (capital J) versus a synthetics monitor job. For improved efficiency, completions should be set to 6-10x the parallelism value. This can help to minimize the "nearing the end of completions" inefficiency where fewer than the parallelism number pods could end up running as the Kubernetes Job waits for all completions to finish.

When completions is greater than 1, pods with a "Completed" status will remain visible in the output of kubectl get pods -n YOUR_NAMESPACE until all completions defined in the Kubernetes Job have been met, for example 6/6 completions. Resources are released from the node when a pod has a status of Completed or Failed.

A Kubernetes Job age of 5 minutes (kubectl get jobs -n YOUR_NAMESPACE) is a conservative target to account for variability in how long it takes pods to complete and how many synthetics jobs need to run per minute (jobs rate). The following equations can be used as a starting point for completions and parallelism for each runtime. Adjustments may need to be made based on observations of private location queue growth.

completions = 300 / avg job duration (s)
parallelism = synthetics jobs per 5 minutes / completions

Different runtimes will likely have different synthetics job durations and rates. The following queries can be used to obtain average duration and rate for a private location.

-- non-ping average job duration by runtime type
FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' 
WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION' FACET type SINCE 1 hour ago

-- non-ping jobs per minute by runtime type
FROM SyntheticCheck SELECT rate(uniqueCount(id), 5 minutes) AS 'jobs per 5 minutes' 
WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION' FACET type SINCE 1 hour ago

Tip

The above queries are based on current results. If your private location does not have any results or the job manager is not performing at its best, query results may not be accurate. In that case, try a few different values for completions and parallelism until you see a kubectl get jobs -n YOUR_NAMESPACE duration of at least 5 minutes (enough completions) and the queue is not growing (enough parallelism).

Example	Description
`parallelism=1` `completions=1`	The runtime will execute 1 synthetics job per minute. After 1 job completes, the `CronJob` configuration will start a new job at the next minute. Throughput will be extremely limited with this configuration.
`parallelism=1` `completions=6`	The runtime will execute 1 synthetics job at a time. After the job completes, a new job will start immediately. After the `completions` setting number of jobs completes, the `CronJob` configuration will start a new Kubernetes Job and reset the completions counter. Throughput will be limited, but slightly better. A single long running synthetics job will block the processing of any other synthetics jobs of this type.
`parallelism=3` `completions=24`	The runtime will execute 3 synthetics jobs at once. After any of these jobs complete, a new job will start immediately. After the `completions` setting number of jobs completes, the `CronJob` configuration will start a new Kubernetes Job and reset the completions counter. Throughput is much better with this or similar configurations. A single long running synthetics job will have limited impact to the processing of other synthetics jobs of this type.

Example

Description

parallelism=1

completions=1

The runtime will execute 1 synthetics job per minute. After 1 job completes, the CronJob configuration will start a new job at the next minute. Throughput will be extremely limited with this configuration.

parallelism=1

completions=6

The runtime will execute 1 synthetics job at a time. After the job completes, a new job will start immediately. After the completions setting number of jobs completes, the CronJob configuration will start a new Kubernetes Job and reset the completions counter. Throughput will be limited, but slightly better. A single long running synthetics job will block the processing of any other synthetics jobs of this type.

parallelism=3

completions=24

The runtime will execute 3 synthetics jobs at once. After any of these jobs complete, a new job will start immediately. After the completions setting number of jobs completes, the CronJob configuration will start a new Kubernetes Job and reset the completions counter. Throughput is much better with this or similar configurations. A single long running synthetics job will have limited impact to the processing of other synthetics jobs of this type.

If synthetics jobs take longer to complete, fewer completions are needed to fill 5 minutes with jobs but more parallel pods will be needed. Similarly, if more synthetics jobs need to be processed per minute, more parallel pods will be needed. The parallelism setting directly affects how many synthetics jobs per minute can be run. Too small a value and the queue may grow. Too large a value and nodes may become resource constrained.

If your parallelism settings is working well to keep the queue at zero, setting a higher value for completions than what is calculated from 300 / avg job duration can help to improve efficiency in a couple of ways:

Accommodate variability in job durations such that at least 1 minute is filled with synthetics jobs, which is the minimum CronJob duration.
Reduce the number of completions cycles to minimize the "nearing the end of completions" inefficiency where the next set of completions can't start until the final job completes.

It's important to note that the completions value should not be too large or the CronJob will experience warning events like the following:

8m40s       Warning   TooManyMissedTimes     cronjob/synthetics-node-browser-runtime                  too many missed start times: 101. Set or decrease .spec.startingDeadlineSeconds or check clock skew

Tip

Please keep in mind that New Relic is not liable for any modifications you make to the synthetics job manager files.

Configuration using environment variables

Docker environment configuration

Kubernetes environment configuration

User-defined variables for scripted monitors

Mounting JSON file

Pass in a local file

Mount a `PersistentVolume`

Passing as an environment variable

Accessing user-defined environment variables from scripts

Caution

Custom node modules

Set up your custom modules directory

Example

Add your custom modules directory to the SJM for Docker or Kubernetes

Docker

Kubernetes

Tip

Create the namespace, persistent volume, and persistent volume claim

Create `mount-custom-mods-pod` to copy your custom-modules directory

Copy your custom-modules directory to the PV

Launch the SJM with custom modules feature enabled

Change `package.json` for custom modules

Caution

Permanent data storage

Docker

Kubernetes

Sizing considerations for Kubernetes and Docker

Tip

Kubernetes

Tip

Tip

Synthetics job manager configuration

Configuration using environment variables .css-21sua1{background:none;border:none;width:0;padding:0;}

Kubernetes environment configuration

User-defined variables for scripted monitors

Mounting JSON file

Passing as an environment variable

Accessing user-defined environment variables from scripts

Caution

Custom node modules

Set up your custom modules directory

Example

Add your custom modules directory to the SJM for Docker or Kubernetes

Docker

Kubernetes

Change package.json for custom modules

Caution

Permanent data storage

Docker

Kubernetes

Sizing considerations for Kubernetes and Docker

Tip

Kubernetes

Tip

Tip

Configuration using environment variables

Change `package.json` for custom modules