Synthetics job manager configuration

This doc will guide you through configuring your synthetics job manager by showing you how to:

• Set up custom modules for scripted API or scripted browser monitors.
• Provide user-defined variables in your configuration.
• Update other environment-variables when launching your synthetics job manager.

User-defined variables for scripted monitors

Private synthetics job managers let you configure environment variables for scripted monitors. These variables are managed locally on the SJM and can be accessed via $env.USER_DEFINED_VARIABLES. You can set user-defined variables in two ways. You can mount a JSON file or you can supply an environment variable to the SJM on launch. If both are provided, the SJM will only use values provided by the environment.

{
              "KEY": "VALUE",
              "user_name": "MINION",
              "my_password": "PASSW0RD123",
              "my_URL": "https://newrelic.com/",
              "ETC": "ETC"
            }

docker run ... -v /variables:/var/lib/newrelic/synthetics/variables:rw ...

### Mount a PersistentVolume
This option requires the user to provide a PersistentVolume that includes the user_defined_variables.json file or a PersistentVolumeClaim to the same. For more details on helm chart installation using a PersistentVolume, follow the instructions at [permanent data storage](/docs/synthetics/synthetic-monitoring/private-locations/job-manager-configuration#permanent-data-storage).
    
Once the user has prepared a PersistentVolume as described below, launch the SJM, setting the path where the user_defined_variables.json file is located and setting any other `synthetics.persistence` variables as necessary.

docker run ... -e USER_DEFINED_VARIABLES='{"key":"value","name":"sjm"}' ...

helm install newrelic/synthetics-job-manager ... --set-literal synthetics.userDefinedVariables.userDefinedJson='{"key":"value","name":"sjm"}' ...

Accessing user-defined environment variables from scripts

To reference a configured user-defined environment variable, use the reserved $env.USER_DEFINED_VARIABLES followed by the name of a given variable with dot notation.

For example, $env.USER_DEFINED_VARIABLES.MY_VARIABLE

Custom node modules

Custom node modules are provided in both CPM and SJM. They allow you to create a customized set of node modules and use them in scripted monitors (scripted API and scripted browser) for synthetic monitoring.

To set up the modules:

1. Create a directory with a package.json file following npm official guidelines in the root folder. The SJM will install any dependencies listed in the package.json's dependencies field. These dependencies will be available when running monitors on the private synthetics job manager. See an example of this below.

/example-custom-modules-dir/
      ├── counter
      │   ├── index.js
      │   └── package.json
      └── package.json            ⇦ the only mandatory file

{
      "name": "custom-modules",
      "version": "1.0.0",           ⇦ optional
      "description": "example custom modules directory", ⇦ optional
      "dependencies": {
        "smallest": "1.0.1",          ⇦ hosted module
        "counter": "file:./counter" ⇦ local module
      }
    }

2. Once you create the custom modules directory and the package.json, apply it to your SJM for Docker and Kubernetes.

   Docker

   For Docker, launch SJM mounting the directory at /var/lib/newrelic/synthetics/modules. For example:

   docker run ... -v /example-custom-modules-dir:/var/lib/newrelic/synthetics/modules:rw ...

   Copy

   Kubernetes

   Complete the following:

   1. Launch the SJM, setting a value for the persistence.customModules configuration value either in the command line or in a YAML file during installation. The value should specify the subpath on your synthetics job manager Persistent Volume where your custom modules files exist. For example:

      helm install ... --set persistence.customModules=<custom-modules-subpath> ...

      Copy

   2. Make sure that your custom modules directory is available on the Minion Pod. You can use kubectl cp as one method to copy the directory from your host to the Minion. For example:

      kubectl cp /example-custom-modules-dir <namespace>/<pod_name>:/var/lib/newrelic/synthetics/modules

      Copy

3. To check if the modules were installed correctly or if any errors occurred, review the SJM logs for the section titled "... Initialization of Custom Modules ...". These logs will include the npm installation logs, providing information regarding the installation process and any potential errors encountered.

Now you can add "require('smallest');" into the script of monitors you send to this private location.

Change package.json for custom modules

In addition to local and hosted modules, you can utilize Node.js modules as well. To update the custom modules used by your SJM, make changes to the package.json file, and restart the SJM. During the reboot process, the SJM will recognize the configuration change and automatically perform cleanup and re-installation operations to ensure the updated modules are applied.

Permanent data storage

Users may want to use permanent data storage to provide the user_defined_variables.json file or support custom node modules (not yet available to private Synthetics Job Managers).

Docker

To set permanent data storage on Docker:

1. Create a directory on the host where you are launching the Job Manager. This is your source directory.
2. Launch the Job Manager, mounting the source directory to the target directory /var/lib/newrelic/synthetics.

Example:

docker run ... -v /sjm-volume:/var/lib/newrelic/synthetics:rw ...

Kubernetes

To set permanent data storage on Kubernetes, the user has two options:

1. Provide an existing PersistentVolumeClaim (PVC) for an existing PersistentVolume (PV), setting the synthetics.persistence.existingClaimName configuration value.

Example:

helm install ... --set synthetics.persistence.existingClaimName=sjm-claim ...

2. Provide an existing PersistentVolume (PV) name, setting the synthetics.persistence.existingVolumeName configuration value. Helm will generate a PVC for the user.

The user may optionally set the following values as well:

• synthetics.persistence.storageClass: the storage class of the existing PV. If not provided, Kubernetes will use the default storage class.
• synthetics.persistence.size: the size for the claim. If not set, the default is currently 2Gi.

helm install ... --set synthetics.persistence.existingVolumeName=sjm-volume --set synthetics.persistence.storageClass=standard ...

Environment variables

Environmental variables allow you to fine-tune the synthetics job manager configuration to meet your specific environmental and functional needs.

Sizing considerations for Kubernetes and Docker

If you're working in larger environments, you may need to customize the job manager configuration to meet minimum requirements to execute synthetic monitors efficiently. Many factors can impact sizing requirements for a synthetics job manager deployment, including:

• If all runtimes are required based on expected usage
• The number of jobs per minute by monitor type (ping, simple or scripted browser, and scripted API)
• Job duration, including jobs that time out at around 3 minutes
• The number of job failures. For job failures, automatic retries are scheduled when a monitor starts to fail to provide built-in 3/3 retry logic. These additional jobs add to the throughput requirements of the synthetic job manager.

In addition to the sizing configuration settings listed below, additional synthetics job managers can be deployed with the same private location key to load balance jobs across multiple environments.

Kubernetes

Each runtime used by the Kubernetes synthetic job manager can be sized independently by setting values in the helm chart.

Additional ping runtimes can be started to help execute ping monitor load by increasing the ping-runtime.replicaCount setting from the default value of 1.

The Node.js API and Node.js Browser runtimes are sized independently using a combination of the parallelism and completions settings. Ideal configurations for these settings will vary based on customer requirements.

The parallelism setting controls how many pods of a particular runtime run concurrently. The parallelism setting is the equivalent of the synthetics.heavyWorkers configuration in the containerized private minion (CPM). Ensure that your Kubernetes cluster has enough resources available to run this number of pods based on their resource request and limit values.

The completions setting controls how many pods of a particular runtime must complete before the CronJob can start another Kubernetes Job for that runtime. Note the difference between a Kubernetes Job (capital J) versus a synthetics monitor job. For improved efficiency, completions should be set to 6-10x the parallelism value. This can help to minimize the "nearing the end of completions" inefficiency where fewer than the parallelism number pods could end up running as the Kubernetes Job waits for all completions to finish.

When completions is greater than 1, pods with a "Completed" status will remain visible in the output of kubectl get pods -n YOUR_NAMESPACE until all completions defined in the Kubernetes Job have been met, for example 6/6 completions. Resources are released from the node when a pod has a status of Completed or Failed.

A Kubernetes Job age of 5 minutes (kubectl get jobs -n YOUR_NAMESPACE) is a conservative target to account for variability in how long it takes pods to complete and how many synthetics jobs need to run per minute (jobs rate). The following equations can be used as a starting point for completions and parallelism for each runtime. Adjustments may need to be made based on observations of private location queue growth.

completions = 300 / avg job duration (s)
parallelism = synthetics jobs per 5 minutes / completions

Different runtimes will likely have different synthetics job durations and rates. The following queries can be used to obtain average duration and rate for a private location.

# non-ping average job duration by runtime type
FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION' FACET type SINCE 1 hour ago

# non-ping jobs per minute by runtime type
FROM SyntheticCheck SELECT rate(uniqueCount(id), 5 minutes) AS 'jobs per 5 minutes' WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION' FACET type SINCE 1 hour ago

If synthetics jobs take longer to complete, fewer completions are needed to fill 5 minutes with jobs but more parallel pods will be needed. Similarly, if more synthetics jobs need to be processed per minute, more parallel pods will be needed. The parallelism setting directly affects how many synthetics jobs per minute can be run. Too small a value and the queue may grow. Too large a value and nodes may become resource constrained.

If your parallelism settings is working well to keep the queue at zero, setting a higher value for completions than what is calculated from 300 / avg job duration can help to improve efficiency in a couple of ways:

• Accommodate variability in job durations such that at least 1 minute is filled with synthetics jobs, which is the minimum CronJob duration.
• Reduce the number of completions cycles to minimize the "nearing the end of completions" inefficiency where the next set of completions can't start until the final job completes.

It's important to note that the completions value should not be too large or the CronJob will experience warning events like the following:

8m40s       Warning   TooManyMissedTimes     cronjob/synthetics-node-browser-runtime                  too many missed start times: 101. Set or decrease .spec.startingDeadlineSeconds or check clock skew