As a note, New Relic is not liable for any modifications you make to the synthetics job manager files.
Environment variables
Environmental variables allow you to fine-tune the synthetics job manager configuration to meet your specific environmental and functional needs.
The variables are provided at startup using the -e, --env argument.
The following table shows all the environment variables that synthetics job manager supports. PRIVATE_LOCATION_KEY is required, and all other variables are optional.
Name
Description
PRIVATE_LOCATION_KEY
REQUIRED. Private location key, as found on the Private Location entity list.
DOCKER_API_VERSION
Format: "vX.Y" API version to be used with the given Docker service.
Default: v1.35.
DOCKER_HOST
Points the synthetics job manager to a given DOCKER_HOST. If absent, the default value is /var/run/docker.sock.
HORDE_API_ENDPOINT
For US-based accounts, the endpoint is: https://synthetics-horde.nr-data.net.
For EU-based accounts, the endpoint is: https://synthetics-horde.eu01.nr-data.net/
Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor.
DOCKER_REGISTRY
The Docker Registry domain where the runtime images are hosted. Use this to override docker.io as the default.
DOCKER_REPOSITORY
The Docker repository / organization where the runtime images are hosted. Use this to override newrelic as the default.
HORDE_API_PROXY_HOST
Proxy server host used for Horde communication. Format: "localhost".
HORDE_API_PROXY_PORT
Proxy server port used for Horde communication. Format: 8888.
HORDE_API_PROXY_USERNAME
Proxy server username used for Horde communication. Format: "username".
HORDE_API_PROXY_PW
Proxy server password used for Horde communication. Format: "password".
HORDE_API_PROXY_ACCEPT_SELF_SIGNED_CERT
Accept self signed proxy certificates for the proxy server connection used for Horde communication? Acceptable values: true
CHECK_TIMEOUT
The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes).
Default: 180 seconds
LOG_LEVEL
Default: INFO.
Additional options: WARN, ERROR
HEAVYWEIGHT_WORKERS
The number of concurrent heavyweight jobs (Browser/Scripted Browser and Scripted API) that can run at one time.
Default: Available CPUs - 1.
DESIRED_RUNTIMES
An array that may be used to run specific runtime images. Format: ['newrelic/synthetics-ping-runtime:latest','newrelic/synthetics-node-api-runtime:latest','newrelic/synthetics-node-browser-runtime:latest']
Default: all latest runtimes.
VSE_PASSPHRASE
If set, enables verified script execution and uses this value as a passphrase.
The variables are provided at startup using the --set argument.
The following list shows all the environment variables that synthetics job manager supports. synthetics.privateLocationKey is required, and all other variables are optional.
A number of additional advanced settings are available and fully documented in our Helm chart README
Name
Description
synthetics.privateLocationKey
REQUIRED if synthetics.privateLocationKeySecretName is not set. Private location key of the private location, as found on the private location web page.
synthetics.privateLocationKeySecretName
REQUIRED if synthetics.privateLocationKey is not set. Name of the Kubernetes secret that contains the key privateLocationKey, which contains the authentication key associated with your synthetics private location.
replicaCount
Number of replicas to maintain with your installation
Default: 1.
imagePullSecrets
The name of the secret object used to pull an image from a specified container registry.
fullnameOverride
Name override used for your Deployment, replacing the default.
appVersionOverride
Release version of synthetics-job-manager to use instead of the version specified in chart.yml.
synthetics.logLevel
Default: INFO.
Additional options: WARN, ERROR
synthetics.hordeApiEndpoint
For US-based accounts, the endpoint is: https://synthetics-horde.nr-data.net.
For EU-based accounts, the endpoint is: https://synthetics-horde.eu01.nr-data.net/
Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor.
synthetics.minionDockerRunnerRegistryEndpoint
The Docker Registry and Organization where the Minion Runner image is hosted. Use this to override quay.io/newrelic as the default (for example, docker.io/newrelic)
synthetics.vsePassphrase
If set, it enables verified script execution, and uses this value as a passphrase.
synthetics.vsePassphraseSecretName
If set, enables verified script execution and uses this value to retrieve the passphrase from a Kubernetes secret with a key called vsePassphrase.
synthetics.apiProxyHost
Proxy server used for Horde communication. Format: "host".
synthetics.apiProxyPort
Proxy server port used for Horde communication. Format: port.
synthetics.hordeApiProxySelfSignedCert
Accept self signed certificates when using a proxy server for Horde communication. Acceptable values: true.
synthetics.hordeApiProxyUsername
Proxy server username for Horde communication. Format: "username"
synthetics.hordeApiProxyPw
Proxy server password for Horde communication. Format: "password".
global.checkTimeout
The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes).
Default: 180 seconds
image.repository
The container to pull.
Default: docker.io/newrelic/synthetics-job-runner
image.pullPolicy
The pull policy.
Default: IfNotPresent
podSecurityContext
Set a custom security context for the synthetics-job-manager pod.
ping-runtime.enabled
Whether or not the persistent ping runtime should be deployed. This can be disabled if you do not use ping monitors.
Default: true
ping-runtime.replicaCount
The number of ping runtime containers to deploy. Increase the replicaCount to scale the deployment based on your ping monitoring needs.
Whether or not the Node.js API runtime should be deployed. This can be disabled if you do not use scripted API monitors.
Default: true
node-api-runtime.parallelism
The number of Node.js API runtime CronJobs to deploy. The maximum number of concurrent Node.js API jobs that will execute at any time. Additional details.
Default: 1
node-api-runtime.completions
The number of Node.js API runtime CronJobs to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. . Increase this setting if you notice periods of time with no API runtime jobs running. Additional details.
Default: 6
node-api-runtime.image.repository
The container image to pull for the Node.js API runtime.
The pull policy for the Node.js API runtime container.
Default: IfNotPresent
node-browser-runtime.enabled
Whether or not the Node.js browser runtime should be deployed. This can be disabled if you do not use simple or scripted browser monitors.
Default: true
node-browser-runtime.parallelism
The number of Chrome browser runtime CronJobs to deploy. The maximum number of concurrent Chrome browser jobs that will execute at any time. Additional details.
Default: 1
node-browser-runtime.completions
The number of Chrome browser runtime CronJobs to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. Increase this setting if you notice periods of time with no browser runtime jobs running. Additional details.
Default: 6
node-browser-runtime.image.repository
The container image to pull for the Node.js browser runtime.
The pull policy for the Node.js browser runtime container.
Default: IfNotPresent
Sizing considerations for Kubernetes and Docker
Tip
Docker specific sizing considerations will be available soon.
If you're working in larger environments, you may need to customize the job manager configuration to meet minimum requirements to execute synthetic monitors efficiently. Many factors can impact sizing requirements for a synthetics job manager deployment, including:
If all runtimes are required based on expected usage
The number of jobs per minute by monitor type (ping, simple or scripted browser, and scripted API)
Job duration, including jobs that time out at around 3 minutes
The number of job failures. For job failures, automatic retries are scheduled when a monitor starts to fail to provide built-in 3/3 retry logic. These additional jobs add to the throughput requirements of the synthetic job manager.
In addition to the sizing configuration settings listed below, additional synthetics job managers can be deployed with the same private location key to load balance jobs across multiple environments.
Kubernetes
Each runtime used by the Kubernetes synthetic job manager can be sized independently by setting values in the helm chart.
Additional ping runtimes can be started to help execute ping monitor load by increasing the ping-runtime.replicaCount setting from the default value of 1.
The Node.js API and Node.js Browser runtimes are sized independently using a combination of the parallelism and completions settings. Ideal configurations for these settings will vary based on customer requirements.
The parallelism setting controls how many pods of a particular runtime run concurrently. The parallelism setting is the equivalent of the synthetics.heavyWorkers configuration in the containerized private minion (CPM). Ensure that your Kubernetes cluster has enough resources available to run this number of pods based on their resource request and limit values.
The completions setting controls how many pods of a particular runtime must complete before the CronJob can start another Kubernetes Job for that runtime. Note the difference between a Kubernetes Job (capital J) versus a synthetics monitor job. For improved efficiency, completions should be set to 6-10x the parallelism value. This can help to minimize the "nearing the end of completions" inefficiency where fewer than the parallelism number pods could end up running as the Kubernetes Job waits for all completions to finish.
When completions is greater than 1, pods with a "Completed" status will remain visible in the output of kubectl get pods -n YOUR_NAMESPACE until all completions defined in the Kubernetes Job have been met, for example 6/6 completions. Resources are released from the node when a pod has a status of Completed or Failed.
A Kubernetes Job age of 5 minutes (kubectl get jobs -n YOUR_NAMESPACE) is a conservative target to account for variability in how long it takes pods to complete and how many synthetics jobs need to run per minute (jobs rate). The following equations can be used as a starting point for completions and parallelism for each runtime. Adjustments may need to be made based on observations of private location queue growth.
completions = 300 / avg job duration (s)
parallelism = synthetics jobs per 5 minutes / completions
Different runtimes will likely have different synthetics job durations and rates. The following queries can be used to obtain average duration and rate for a private location.
# non-ping average job duration by runtime type
FROM SyntheticCheck SELECT average(duration)AS'avg job duration'WHEREtype!='SIMPLE'AND location ='YOUR_PRIVATE_LOCATION' FACET type SINCE 1hour ago
# non-ping jobs per minute by runtime type
FROM SyntheticCheck SELECT rate(uniqueCount(id),5 minutes)AS'jobs per 5 minutes'WHEREtype!='SIMPLE'AND location ='YOUR_PRIVATE_LOCATION' FACET type SINCE 1hour ago
Tip
The above queries are based on current results. If your private location does not have any results or the job manager is not performing at its best, query results may not be accurate. In that case, try a few different values for completions and parallelism until you see a kubectl get jobs -n YOUR_NAMESPACE duration of at least 5 minutes (enough completions) and the queue is not growing (enough parallelism).
Example
Description
parallelism=1
completions=1
The runtime will execute 1 synthetics job per minute. After 1 job completes, the CronJob configuration will start a new job at the next minute. Throughput will be extremely limited with this configuration.
parallelism=1
completions=6
The runtime will execute 1 synthetics job at a time. After the job completes, a new job will start immediately. After the completions setting number of jobs completes, the CronJob configuration will start a new Kubernetes Job and reset the completions counter. Throughput will be limited, but slightly better. A single long running synthetics job will block the processing of any other synthetics jobs of this type.
parallelism=3
completions=24
The runtime will execute 3 synthetics jobs at once. After any of these jobs complete, a new job will start immediately. After the completions setting number of jobs completes, the CronJob configuration will start a new Kubernetes Job and reset the completions counter. Throughput is much better with this or similar configurations. A single long running synthetics job will have limited impact to the processing of other synthetics jobs of this type.
If synthetics jobs take longer to complete, fewer completions are needed to fill 5 minutes with jobs but more parallel pods will be needed. Similarly, if more synthetics jobs need to be processed per minute, more parallel pods will be needed. The parallelism setting directly affects how many synthetics jobs per minute can be run. Too small a value and the queue may grow. Too large a value and nodes may become resource constrained.
If your parallelism settings is working well to keep the queue at zero, setting a higher value for completions than what is calculated from 300 / avg job duration can help to improve efficiency in a couple of ways:
Accommodate variability in job durations such that at least 1 minute is filled with synthetics jobs, which is the minimum CronJob duration.
Reduce the number of completions cycles to minimize the "nearing the end of completions" inefficiency where the next set of completions can't start until the final job completes.
It's important to note that the completions value should not be too large or the CronJob will experience warning events like the following:
8m40s Warning TooManyMissedTimes cronjob/synthetics-node-browser-runtime too many missed start times: 101. Set or decrease .spec.startingDeadlineSeconds or check clock skew