--- title: Synthetics job manager configuration source: https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/job-manager-configuration --- This doc will guide you through configuring your [synthetics job manager](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/install-job-manager) by showing you how to: - Use [environment variables](#environment-variables) to configure your synthetics job manager. - Set up [custom modules](#custom-modules) for [scripted API](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/scripting-monitors/write-synthetic-api-tests/) or [scripted browser](https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/scripting-monitors/write-scripted-browsers) monitors. - Provide [user-defined variables](#user-defined-vars) in your configuration. ## Configuration using environment variables [#environment-variables] Environmental variables allow you to fine-tune the synthetics job manager configuration to meet your specific environmental and functional needs. **Docker environment configuration** The variables are provided at startup using the `-e, --env` argument. The following table shows all the environment variables that synthetics job manager supports. `PRIVATE_LOCATION_KEY` is required, and all other variables are optional. | Name | Description | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PRIVATE_LOCATION_KEY` | **Required.** Private location key, as found on the Private Location entity list. | | `DOCKER_API_VERSION` | Format: `"vX.Y"` API version to be used with the given Docker service. Default: `v1.35.` | | `DOCKER_HOST` | Points the synthetics job manager to a given `DOCKER_HOST`. If absent, the default value is `/var/run/docker.sock.` | | `HORDE_API_ENDPOINT` | For US-based accounts (default), the endpoint is: `https://synthetics-horde.nr-data.net.` For [EU-based](https://docs.newrelic.com/docs/using-new-relic/welcome-new-relic/get-started/introduction-eu-region-data-center#partner-hierarchy) accounts, the endpoint is: `https://synthetics-horde.eu01.nr-data.net/` For JP-based accounts, the endpoint is: `https://synthetics-horde.jp.nr-data.net/` Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor. | | `DOCKER_REGISTRY` | The Docker Registry domain where the runtime images are hosted. Use this to override `docker.io` as the default. | | `DOCKER_REPOSITORY` | The Docker repository or organization where the runtime images are hosted. Use this to override `newrelic` as the default. | | `HORDE_API_PROXY_HOST` | Proxy server host used for Horde communication. Format: `"localhost"`. | | `HORDE_API_PROXY_PORT` | Proxy server port used for Horde communication. Format: `8888`. | | `HORDE_API_PROXY_USERNAME` | Proxy server username used for Horde communication. Format: `"username"`. | | `HORDE_API_PROXY_PW` | Proxy server password used for Horde communication. Format: `"password"`. | | `HORDE_API_PROXY_ACCEPT_SELF_SIGNED_CERT` | Accept self signed proxy certificates for the proxy server connection used for Horde communication? Acceptable values: `true` | | `CHECK_TIMEOUT` | The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes). Default: 180 seconds | | `LOG_LEVEL` | Default: `INFO.` Additional options: `WARN`, `ERROR`, `DEBUG` | | `HEAVYWEIGHT_WORKERS` | The number of concurrent heavyweight jobs (Browser/Scripted Browser and Scripted API) that can run at one time. Default: Available CPUs - 1. | | `DESIRED_RUNTIMES` | An array that may be used to run specific runtime images. Format: ['newrelic/synthetics-ping-runtime:latest','newrelic/synthetics-node-api-runtime:latest','newrelic/synthetics-node-browser-runtime:latest'] Default: all latest runtimes. | | `VSE_PASSPHRASE` | If set, enables **verified script execution** and uses this value as a **passphrase**. | | `USER_DEFINED_VARIABLES` | A locally hosted set of user defined key value pairs. | | `ENABLE_WASM` | If set, enables webassembly for node browser runtime. To use webassembly, your synthetics job manager minimum version should be release-367 or higher and node browser runtime version should be 2.3.21 or higher. | **Podman environment configuration** The variables are provided at startup using the `-e, --env` argument. The following table displays all the environment variables that synthetics job manager supports. `PRIVATE_LOCATION_KEY` is required, and all other variables are optional. To run the synthetics job manager in a Podman environment, the minimum version should be release-418 or higher. | Name | Description | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PRIVATE_LOCATION_KEY` | **Required.** Private location key, as found on the Private Location entity list. | | `HORDE_API_ENDPOINT` | For US-based accounts (default), the endpoint is: `https://synthetics-horde.nr-data.net.` For [EU-based](https://docs.newrelic.com/docs/using-new-relic/welcome-new-relic/get-started/introduction-eu-region-data-center#partner-hierarchy) accounts, the endpoint is: `https://synthetics-horde.eu01.nr-data.net/` For JP-based accounts, the endpoint is: `https://synthetics-horde.jp.nr-data.net/` Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor. | | `PODMAN_API_SERVICE_HOST` | The host entry added to the Pod created where the SJM is going to run. Use this to override `podman.service` as the default. | | `PODMAN_API_SERVICE_PORT` | The port at which the Podman LibPod RESTful API service is running in the instance. Use this to override `8000` as the default. | | `PODMAN_API_VERSION` | The specific version of the Podman LibPod RESTful API being used. Use this to override `v5.0.0` as the default. | | `PODMAN_POD_NAME` | The name of the pod in which the SJM container is run. Use this to override `SYNTHETICS` as the default. | | `DOCKER_REGISTRY` | The Docker Registry domain where the runtime images are hosted. Use this to override `docker.io` as the default. | | `DOCKER_REPOSITORY` | The Docker repository or organization where the runtime images are hosted. Use this to override `newrelic` as the default. | | `HORDE_API_PROXY_HOST` | Proxy server host used for Horde communication. Format: `"localhost"`. | | `HORDE_API_PROXY_PORT` | Proxy server port used for Horde communication. Format: `8888`. | | `HORDE_API_PROXY_USERNAME` | Proxy server username used for Horde communication. Format: `"username"`. | | `HORDE_API_PROXY_PW` | Proxy server password used for Horde communication. Format: `"password"`. | | `HORDE_API_PROXY_ACCEPT_SELF_SIGNED_CERT` | Accept self signed proxy certificates for the proxy server connection used for Horde communication? Acceptable values: `true` | | `CHECK_TIMEOUT` | The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes). Default: 180 seconds | | `LOG_LEVEL` | Default: `INFO.` Additional options: `WARN`, `ERROR`, `DEBUG` | | `HEAVYWEIGHT_WORKERS` | The number of concurrent heavyweight jobs (Browser/Scripted Browser and Scripted API) that can run at one time. Default: Available CPUs - 1. | | `DESIRED_RUNTIMES` | An array that may be used to run specific runtime images. Format: ['newrelic/synthetics-ping-runtime:latest','newrelic/synthetics-node-api-runtime:latest','newrelic/synthetics-node-browser-runtime:latest'] Default: all latest runtimes. | | `VSE_PASSPHRASE` | If set, enables **verified script execution** and uses this value as a **passphrase**. | | `USER_DEFINED_VARIABLES` | A locally hosted set of user defined key value pairs. | | `ENABLE_WASM` | If set, enables webassembly for node browser runtime. To use webassembly, your synthetics job manager minimum version should be release-367 or higher and node browser runtime version should be 2.3.21 or higher. | **Kubernetes environment configuration** The variables are provided at startup using the `--set` argument. The following list shows all the environment variables that synthetics job manager supports. `synthetics.privateLocationKey` is required, and all other variables are optional. A number of additional advanced settings are available and fully documented in [our Helm chart README](https://github.com/newrelic/helm-charts/blob/master/charts/synthetics-job-manager/README.md) | Name | Description | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `synthetics.privateLocationKey` | **Required if `synthetics.privateLocationKeySecretName` is not set**. [Private location key](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/install-job-manager/#private-location-key) of the private location, as found on the private location web page. | | `synthetics.privateLocationKeySecretName` | **Required if `synthetics.privateLocationKey` is not set**. Name of the Kubernetes secret that contains the key `privateLocationKey`, which contains the authentication key associated with your synthetics private location. | | `imagePullSecrets` | The name of the secret object used to pull an image from a specified container registry. | | `fullnameOverride` | Name override used for your Deployment, replacing the default. | | `appVersionOverride` | Release version of synthetics-job-manager to use instead of the version specified in [chart.yml](https://github.com/newrelic/helm-charts/blob/master/charts/synthetics-job-manager/Chart.yaml). | | `synthetics.logLevel` | Default: `INFO.` Additional options: `WARN`, `ERROR` | | `synthetics.hordeApiEndpoint` | For US-based accounts (default), the endpoint is: `https://synthetics-horde.nr-data.net.` For [EU-based](https://docs.newrelic.com/docs/using-new-relic/welcome-new-relic/get-started/introduction-eu-region-data-center#partner-hierarchy) accounts, the endpoint is: `https://synthetics-horde.eu01.nr-data.net/` For JP-based accounts, the endpoint is: `https://synthetics-horde.jp.nr-data.net/` Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor. | | `synthetics.minionDockerRunnerRegistryEndpoint` | The Docker Registry and Organization where the Minion Runner image is hosted. Use this to override `quay.io/newrelic` as the default (for example, `docker.io/newrelic`) | | `synthetics.vsePassphrase` | If set, it enables **verified script execution**, and uses this value as a **passphrase**. | | `synthetics.vsePassphraseSecretName` | If set, enables verified script execution and uses this value to retrieve the passphrase from a Kubernetes secret with a key called `vsePassphrase`. | | `synthetics.enableWasm` | If set, enables webassembly for node browser runtime. To use webassembly, your synthetics job manager minimum version should be release-367 or higher and node browser runtime version should be 2.3.21 or higher. | | `synthetics.apiProxyHost` | Proxy server used for Horde communication. Format: `"host"`. | | `synthetics.apiProxyPort` | Proxy server port used for Horde communication. Format: `port`. | | `synthetics.hordeApiProxySelfSignedCert` | Accept self signed certificates when using a proxy server for Horde communication. Acceptable values: `true`. | | `synthetics.hordeApiProxyUsername` | Proxy server username for Horde communication. Format: `"username"` | | `synthetics.hordeApiProxyPw` | Proxy server password for Horde communication. Format: `"password"`. | | `synthetics.userDefinedVariables.userDefinedJson` | A JSON string of user-defined variables. The user may access these variables in their script. Format: `'{"key":"value","key2":"value2"}'`. | | `synthetics.userDefinedVariables.userDefinedFile` | A path local to the user to a JSON file containing user-defined variables. This is passed in via `--set-file` and cannot be set in the Values file. | | `synthetics.userDefinedVariables.userDefinedPath` | A path on the user's provided PersistentVolume to the user_defined_variables.json file. User must provide a PersistentVolume or PersistentVolumeClaim if this variable is populated. | | `synthetics.persistence.existingClaimName` | If mounting a volume, the user may provide a name for a PersistentVolumeClaim that already exists in the cluster. Presumes the existence of an corresponding PersistentVolume. | | `synthetics.persistence.existingVolumeName` | If mounting a volume and not providing a PersistentVolumeClaim, the user must at minimum provide a PersistentVolume name. Helm will generate a PersistentVolumeClaim. | | `synthetics.persistence.storageClass` | The name of the StorageClass for the generated PersistentVolumeClaim. This should match the StorageClassName on the existing PV. If not providers, Kubernetes will use the default storage class if present. | | `synthetics.persistence.size` | The size of the volume for the generated PersistentVolumeClaim. Format: `10Gi`. Default 2Gi. | | `global.checkTimeout` | The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes). Default: 180 seconds | | `image.repository` | The container to pull. Default: `docker.io/newrelic/synthetics-job-runner` | | `image.pullPolicy` | The pull policy. Default: `IfNotPresent` | | `podSecurityContext` | Set a custom security context for the synthetics-job-manager pod. | | `ping-runtime.enabled` | Whether or not the persistent ping runtime should be deployed. This can be disabled if you do not use ping monitors. Default: `true` | | `ping-runtime.replicaCount` | The number of ping runtime containers to deploy. Increase the replicaCount to scale the deployment based on your ping monitoring needs. Default: `1` | | `ping-runtime.image.repository` | The container image to pull for the ping runtime. Default: `docker.io/newrelic/synthetics-ping-runtime` | | `ping-runtime.image.pullPolicy` | The pull policy for the ping-runtime container. Default: `IfNotPresent` | | `node-api-runtime.enabled` | Whether or not the Node.js API runtime should be deployed. This can be disabled if you do not use scripted API monitors. Default: `true` | | `node-api-runtime.parallelism` | The number of Node.js API runtime `CronJobs` to deploy. The maximum number of concurrent Node.js API jobs that will execute at any time. [Additional details](#kubernetes-sizing). Default: `1` | | `node-api-runtime.completions` | The number of Node.js API runtime `CronJobs` to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. . Increase this setting if you notice periods of time with no API runtime jobs running. [Additional details](#kubernetes-sizing). Default: `6` | | `node-api-runtime.image.repository` | The container image to pull for the Node.js API runtime. Default: `docker.io/newrelic/synthetics-node-api-runtime` | | `node-api-runtime.image.pullPolicy` | The pull policy for the Node.js API runtime container. Default: `IfNotPresent` | | `node-browser-runtime.enabled` | Whether or not the Node.js browser runtime should be deployed. This can be disabled if you do not use simple or scripted browser monitors. Default: `true` | | `node-browser-runtime.parallelism` | The number of Chrome browser runtime `CronJobs` to deploy. The maximum number of concurrent Chrome browser jobs that will execute at any time. [Additional details](#kubernetes-sizing). Default: `1` | | `node-browser-runtime.completions` | The number of Chrome browser runtime `CronJobs` to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. Increase this setting if you notice periods of time with no browser runtime jobs running. [Additional details](#kubernetes-sizing). Default: `6` | | `node-browser-runtime.image.repository` | The container image to pull for the Node.js browser runtime. Default: `docker.io/newrelic/synthetics-node-browser-runtime` | | `node-browser-runtime.image.pullPolicy` | The pull policy for the Node.js browser runtime container. Default: `IfNotPresent` | **OpenShift environment configuration** The variables are provided at startup using the `--set` argument. The following list shows all the environment variables that synthetics job manager supports. `synthetics.privateLocationKey` is required, and all other variables are optional. A number of additional advanced settings are available and fully documented in [our Helm chart README](https://github.com/newrelic/helm-charts/blob/master/charts/synthetics-job-manager/README.md) | Name | Description | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `synthetics.privateLocationKey` | **Required**. [Private location key](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/install-job-manager/#private-location-key), as found on the private location entity list. | | `imagePullSecrets` | The name of the secret object used to pull an image from a specified container registry. | | `fullnameOverride` | Name override used for your Deployment, replacing the default. | | `appVersionOverride` | Release version of synthetics-job-manager to use instead of the version specified in [chart.yml](https://github.com/newrelic/helm-charts/blob/master/charts/synthetics-job-manager/Chart.yaml). | | `synthetics.logLevel` | Default: `INFO.` Additional options: `WARN`, `ERROR` | | `synthetics.hordeApiEndpoint` | For US-based accounts (default), the endpoint is: `https://synthetics-horde.nr-data.net.` For [EU-based](https://docs.newrelic.com/docs/using-new-relic/welcome-new-relic/get-started/introduction-eu-region-data-center#partner-hierarchy) accounts, the endpoint is: `https://synthetics-horde.eu01.nr-data.net/` For JP-based accounts, the endpoint is: `https://synthetics-horde.jp.nr-data.net/` Ensure your synthetics job manager can connect to the appropriate endpoint in order to serve your monitor. | | `synthetics.vsePassphrase` | If set, it enables **verified script execution**, and uses this value as a **passphrase**. | | `synthetics.vsePassphraseSecretName` | If set, enables verified script execution and uses this value to retrieve the passphrase from a Kubernetes secret with a key called `vsePassphrase`. | | `synthetics.enableWasm` | If set, enables webassembly for node browser runtime. To use webassembly, your synthetics job manager minimum version should be release-367 or higher and node browser runtime version should be 2.3.21 or higher. | | `synthetics.apiProxyHost` | Proxy server used for Horde communication. Format: `"host"`. | | `synthetics.apiProxyPort` | Proxy server port used for Horde communication. Format: `port`. | | `synthetics.hordeApiProxySelfSignedCert` | Accept self signed certificates when using a proxy server for Horde communication. Acceptable values: `true`. | | `synthetics.hordeApiProxyUsername` | Proxy server username for Horde communication. Format: `"username"` | | `synthetics.hordeApiProxyPw` | Proxy server password for Horde communication. Format: `"password"`. | | `synthetics.userDefinedVariables.userDefinedJson` | A JSON string of user-defined variables. The user may access these variables in their script. Format: `'{"key":"value","key2":"value2"}'`. | | `synthetics.userDefinedVariables.userDefinedFile` | A path local to the user to a JSON file containing user-defined variables. This is passed in via `--set-file` and cannot be set in the Values file. | | `synthetics.userDefinedVariables.userDefinedPath` | A path on the user's provided `PersistentVolume` to the` user_defined_variables.json` file. User must provide a `PersistentVolume` or `PersistentVolumeClaim` if this variable is populated. | | `global.persistence.existingClaimName` | If mounting a volume, the user may provide a name for a `PersistentVolumeClaim` that already exists in the cluster. Presumes the existence of an corresponding `PersistentVolume`. | | `global.persistence.existingVolumeName` | If mounting a volume and not providing a `PersistentVolumeClaim`, the user must at minimum provide a `PersistentVolume` name. Helm will generate a `PersistentVolumeClaim`. | | `global.persistence.storageClass` | The name of the `StorageClass` for the generated `PersistentVolumeClaim`. This should match the `StorageClassName` on the existing PV. If not providers, **Kubernetes** will use the default storage class if present. | | `global.persistence.size` | The size of the volume for the generated `PersistentVolumeClaim`. Format: `10Gi`. Default `2Gi`. | | `global.checkTimeout` | The maximum amount of seconds that your monitor checks are allowed to run. This value must be an integer between 0 seconds (excluded) and 900 seconds (included) (for example, from 1 second to 15 minutes). Default: 180 seconds | | `image.repository` | The container to pull. Default: `docker.io/newrelic/synthetics-job-runner` | | `image.pullPolicy` | The pull policy. Default: `IfNotPresent` | | `podSecurityContext` | Set a custom security context for the `synthetics-job-manager` pod. | | `ping-runtime.enabled` | Whether or not the persistent ping runtime should be deployed. This can be disabled if you do not use ping monitors. Default: `true` | | `ping-runtime.replicaCount` | The number of ping runtime containers to deploy. Increase the `replicaCount` to scale the deployment based on your ping monitoring needs. Default: `1` | | `ping-runtime.image.repository` | The container image to pull for the ping runtime. Default: `docker.io/newrelic/synthetics-ping-runtime` | | `ping-runtime.image.pullPolicy` | The pull policy for the ping-runtime container. Default: `IfNotPresent` | | `node-api-runtime.enabled` | Whether or not the Node.js API runtime should be deployed. This can be disabled if you do not use scripted API monitors. Default: `true` | | `node-api-runtime.parallelism` | The number of Node.js API runtime `CronJobs` to deploy. The maximum number of concurrent Node.js API jobs that will execute at any time. [Additional details](#kubernetes-sizing). Default: `1` | | `node-api-runtime.completions` | The number of Node.js API runtime `CronJobs` to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. . Increase this setting if you notice periods of time with no API runtime jobs running. [Additional details](#kubernetes-sizing). Default: `6` | | `node-api-runtime.image.repository` | The container image to pull for the Node.js API runtime. Default: `docker.io/newrelic/synthetics-node-api-runtime` | | `node-api-runtime.image.pullPolicy` | The pull policy for the Node.js API runtime container. Default: `IfNotPresent` | | `node-browser-runtime.enabled` | Whether or not the Node.js browser runtime should be deployed. This can be disabled if you do not use simple or scripted browser monitors. Default: `true` | | `node-browser-runtime.parallelism` | The number of Chrome browser runtime `CronJobs` to deploy. The maximum number of concurrent Chrome browser jobs that will execute at any time. [Additional details](#kubernetes-sizing). Default: `1` | | `node-browser-runtime.completions` | The number of Chrome browser runtime `CronJobs` to complete per minute. Increase this setting along with parallelism to improve throughput. This should be increased any time parallelism is increased and completions should always be at least greater than or equal to parallelism. Increase this setting if you notice periods of time with no browser runtime jobs running. [Additional details](#kubernetes-sizing). Default: `6` | | `node-browser-runtime.image.repository` | The container image to pull for the Node.js browser runtime. Default: `docker.io/newrelic/synthetics-node-browser-runtime` | | `node-browser-runtime.image.pullPolicy` | The pull policy for the Node.js browser runtime container. Default: `IfNotPresent` | ## User-defined variables for scripted monitors [#user-defined-vars] Private synthetics job managers let you configure environment variables for scripted monitors. These variables are managed locally on the SJM and can be accessed via `$env.USER_DEFINED_VARIABLES`. You can set user-defined variables in two ways. You can mount a JSON file or you can supply an environment variable to the SJM on launch. If both are provided, the SJM will only use values provided by the environment. **Mounting JSON file** The user may create a JSON-formatted file and mount the volume where the file is located to a specified target path in the SJM container. The file must have read permissions and contain a JSON-formatted map. Example user-defined variables file: ````json { "KEY": "VALUE", "user_name": "MINION", "my_password": "PASSW0RD123", "my_URL": "https://newrelic.com/", "ETC": "ETC" } ``` Place the file in the source directory on the host. The SJM is expecting the file name to be user_defined_variables.json Docker example: The expected target directory is: `/var/lib/newrelic/synthetics/variables/` ```sh docker run ... -v /variables:/var/lib/newrelic/synthetics/variables:rw ... ``` Podman example: In case of SELinux, mount the volume additionally with `:z` or `:Z`. For more information, refer [Podman documentation.](https://docs.podman.io/en/latest/markdown/podman-run.1.html#volume-v-source-volume-host-dir-container-dir-options) The expected target directory is: `/var/lib/newrelic/synthetics/variables/` ```sh podman run ... -v /variables:/var/lib/newrelic/synthetics/variables:rw,z ... ``` Kubernetes example: The user has two options when providing a file to the SJM pod in Kubernetes. They may: * Pass in a local file. * Provide a PersistentVolume that includes the `user_defined_variables.json`. ### Pass in a local file This option creates a ConfigMap Kubernetes resource and mounts that to the SJM pod. ```sh helm install newrelic/synthetics-job-manager ... --set-file "synthetics.userDefinedVariables.userDefinedFile=[local-path]/user_defined_variables.json" ... ``` ### Mount a `PersistentVolume` This option requires the user to provide a `PersistentVolume` that includes the `user_defined_variables.json` file or a `PersistentVolumeClaim` to the same. For more details on helm chart installation using a `PersistentVolume`, follow the instructions at [permanent data storage](/docs/synthetics/synthetic-monitoring/private-locations/job-manager-configuration#permanent-data-storage). Once the user has prepared a `PersistentVolume` as described below, launch the SJM, setting the path where the `user_defined_variables.json` file is located, and set any other `synthetics.persistence` variables as necessary. ```sh helm install newrelic/synthetics-job-manger ... --set synthetics.userDefinedVariables.userDefinedPath="variables" ``` ```` **Passing as an environment variable** The variables may be passed to their respective container system via environment variable. Docker example: Use the `-e` flag to set up an environment variable named `USER_DEFINED_VARIABLES` and give it the value of a JSON formatted map string. ````sh docker run ... -e USER_DEFINED_VARIABLES='{"key":"value","name":"sjm"}' ... ``` Podman example: Use the `-e` flag to set up an environment variable named `USER_DEFINED_VARIABLES` and give it the value of a JSON formatted map string. ```sh podman run ... -e USER_DEFINED_VARIABLES='{"key":"value","name":"sjm"}' ... ``` Kubernetes example: Use the `--set-literal` flag to pass in the JSON formatted string. ```sh helm install newrelic/synthetics-job-manager ... --set-literal synthetics.userDefinedVariables.userDefinedJson='{"key":"value","name":"sjm"}' ... ``` ```` ### Accessing user-defined environment variables from scripts [#env-vars-scripts] To reference a configured user-defined environment variable, use the reserved `$env.USER_DEFINED_VARIABLES` followed by the name of a given variable with dot notation (for example, `$env.USER_DEFINED_VARIABLES.MY_VARIABLE`). > #### ⚠️ CAUTION > > User-defined environment variables are not sanitized from logs. Consider using the [secure credentials](https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/using-monitors/secure-credentials-store-credentials-information-scripted-browsers) feature for sensitive information. ## Custom node modules [#custom-modules] Custom node modules are provided in the SJM. They allow you to create a customized set of [node modules](https://docs.npmjs.com/about-packages-and-modules) and use them in scripted monitors (scripted API and scripted browser) for synthetic monitoring. ### Set up your custom modules directory Create a directory with a `package.json` file following [npm official guidelines](https://docs.npmjs.com/files/package.json) in the root folder. The SJM will install any dependencies listed in the package.json's `dependencies` field. These dependencies will be available when running monitors on the private synthetics job manager. See an example of this below. #### Example In this example, a custom module directory is used with the following structure: ``` /example-custom-modules-dir/ ├── counter │ ├── index.js │ └── package.json └── package.json ⇦ the only mandatory file ``` The `package.json` defines `dependencies` as both a local module (for example, `counter`) and any hosted modules (for example, `smallest` version `1.0.1`): ```json { "name": "custom-modules", "version": "1.0.0", ⇦ optional "description": "example custom modules directory", ⇦ optional "dependencies": { "smallest": "1.0.1", ⇦ hosted module "counter": "file:./counter" ⇦ local module } } ``` ### Add your custom modules directory to the SJM for Docker, Podman, or Kubernetes **Docker** For Docker, launch SJM mounting the directory at `/var/lib/newrelic/synthetics/modules`. For example: ````sh docker run ... -v /example-custom-modules-dir:/var/lib/newrelic/synthetics/modules:rw ... ``` ```` **Podman** For podman, launch SJM mounting the directory at `/var/lib/newrelic/synthetics/modules`. In case of SELinux, mount the volume additionally with with `:z` or `:Z`. For more information, refer [Podman documentation](https://docs.podman.io/en/latest/markdown/podman-run.1.html#volume-v-source-volume-host-dir-container-dir-options). For example: ````sh podman run ... -v /example-custom-modules-dir:/var/lib/newrelic/synthetics/modules:rw,z ... ``` ```` **Kubernetes** For Kubernetes, the directory at `/var/lib/newrelic/synthetics/modules` needs to exist on a PV prior to launching the SJM with custom modules enabled. > #### 💡 TIP > > The PV access mode should be ReadWriteMany if you need to share storage across multiple pods. One method is to create a pod that mounts the PV just for the purpose of copying your custom modules directory to the PV. The following example uses Amazon EFS with Amazon EKS: #### Create the namespace, persistent volume, and persistent volume claim 1. Make sure you've already set up your EFS filesystem and installed the [EFS CSI driver](https://github.com/kubernetes-sigs/aws-efs-csi-driver) on your cluster. You will also need your EFS filesystem ID for the PV's `spec.csi.volumeHandle`. ```sh kubectl apply -f - < --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: custom-modules-pvc namespace: newrelic spec: accessModes: - ReadWriteMany storageClassName: efs-sc resources: requests: storage: 5Gi EOF ``` 2. Switch to the `newrelic` namespace in your `~/.kube/config`. ```sh kubectl config get-contexts kubectl config set-context YOUR_CONTEXT --namespace=newrelic kubectl config view --minify | grep namespace: ``` 3. At this point, the PVC should be bound to the PV with RWX access mode. ```sh kubectl get pv,pvc [output] NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE [output] persistentvolume/custom-modules-pvc 5Gi RWX Retain Bound newrelic/custom-modules-pvc efs-sc 4m46s [output] [output] NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE [output] persistentvolumeclaim/custom-modules-pvc Bound custom-modules-pvc 5Gi RWX efs-sc 4m10s ``` #### Create `mount-custom-mods-pod` to copy your custom-modules directory ```sh kubectl apply -f - < #### ⚠️ CAUTION > > Local modules: While your `package.json` can include any local module, these modules must reside inside the tree under your custom module directory. If stored outside the tree, the initialization process will fail and you will see an error message in the [docker logs](https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/private-locations/job-manager-maintenance-monitoring#monitor-docker-logs) after launching SJM. ## Configure custom CA certificates for private locations [#custom-ca-certificates] If your synthetic monitors need to connect to internal HTTPS endpoints secured by a private Certificate Authority (CA), you can mount your custom CA certificates directly into the SJM. The SJM automatically detects these certificates and passes them to each runtime container at startup. This ensures your monitors trust your internal endpoints without requiring you to build or maintain custom runtime images. ### Certificate requirements To ensure the SJM loads your certificates successfully, they must meet the following criteria: - **Format:** Certificates must be in PEM format with a `.pem` file extension. - **Permissions:** Files must be readable by UID 2000 (the non-root user that runs the runtime container). - **Certificate type:** Only CA certificates are loaded. Any leaf certificates (server or client certificates) that lack the `CA:TRUE` basic constraint extension are automatically ignored. ### Deployment environments Select your container orchestration platform below to mount your certificates. **Docker** Mount the local host directory containing your PEM files to the SJM container path `/var/lib/newrelic/synthetics/certs` as a read-only volume. ````sh docker run \ -v /PATH/TO/YOUR_CA_CERTS:/var/lib/newrelic/synthetics/certs:ro \ newrelic/synthetics-job-manager ``` **How this works:** The SJM detects this mount at startup and propagates the certificates as read-only to every dynamic container it spawns. ```` **Podman** For Podman environments—especially those running with SELinux enabled—you must append the `:z` or `:Z` flag to the volume mount to configure the correct security context. ````sh podman run \ -v /PATH/TO/YOUR_CA_CERTS:/var/lib/newrelic/synthetics/certs:ro,z \ newrelic/synthetics-job-manager ``` ```` **Kubernetes (Helm)** When deploying SJM to a Kubernetes cluster via Helm, you can manage your custom CA bundle as a generic Kubernetes secret. **1. Create the Kubernetes secret** Store your PEM certificate file(s) in a secret within your New Relic agent namespace: ````sh kubectl create secret generic my-ca-bundle \ --from-file=/PATH/TO/YOUR_CA_CERTS/ \ -n newrelic ``` **2. Update your Helm values** Reference the newly created secret in your custom `values.yaml` configuration: ```yaml global: customCertificates: volume: secret: secretName: my-ca-bundle ``` **How this works:** At startup, the SJM mounts this secret and automatically passes it down to all newly spawned execution pods. ```` ## mTLS client certificates [#mtls-client-certificates] If your monitors connect to internal endpoints that enforce mutual TLS (mTLS) — requiring the client to present a certificate — you can provide client certificates mapped to target hostnames. Monitors present the correct certificate automatically. No script changes are required. **Requirements:** - Certificate and key files must be in PEM format - A `cert-map.json` file mapping hostnames to certificate and key files is required **cert-map.json format:** ```json { "api.internal.company.com": { "cert": "client.crt", "key": "client.key" }, "*.payments.internal": { "cert": "payments.crt", "key": "payments.key" }, "default": { "cert": "fallback.crt", "key": "fallback.key" } } ``` Hostname matching priority: exact match → wildcard (`*.domain.com`) → `default` catch-all → no certificate (non-mTLS connections are unaffected). **Docker** Mount a directory containing `cert-map.json` and the referenced certificate and key files: ````sh docker run \ -v /path/to/ca-certs:/var/lib/newrelic/synthetics/certs:ro \ -v /path/to/client-certs:/var/lib/newrelic/synthetics/client-cert:ro \ newrelic/synthetics-job-manager ``` If the mTLS server hostnames are not resolvable via DNS inside the runtime containers, add hostname mappings using `RUNTIME_EXTRA_HOSTS`: ```sh docker run \ -e 'RUNTIME_EXTRA_HOSTS=api.internal.company.com:10.0.0.1,svc.internal:10.0.0.2' \ -v /path/to/client-certs:/var/lib/newrelic/synthetics/client-cert:ro \ newrelic/synthetics-job-manager ``` ```` **Podman** Mount the client certificate directory. For SELinux add `:z` or `:Z`: ````sh podman run \ -v /path/to/client-certs:/var/lib/newrelic/synthetics/client-cert:ro,z \ newrelic/synthetics-job-manager ``` ```` **Kubernetes** Create a Kubernetes Secret containing `cert-map.json` and all referenced certificate and key files: ````sh kubectl create secret generic my-mtls-certs \ --from-file=cert-map.json=/path/to/cert-map.json \ --from-file=client.crt=/path/to/client.crt \ --from-file=client.key=/path/to/client.key \ -n newrelic ``` Reference the secret in your Helm values: ```yaml global: clientCertificates: volume: secret: secretName: my-mtls-certs ``` The secret is mounted read-only into all runtime pods automatically. ```` ## Certificates support limitations [#certificates-support-limitations] The following limitations apply to both custom CA certificates and mTLS client certificates unless noted: - **Restart required for cert changes** — Certificates are loaded at container startup. After adding, removing, or rotating certificates, restart the job manager container (or the runtime pods in Kubernetes) for changes to take effect. - **Supported runtimes** — Custom certificate features apply to scripted API and scripted browser monitors only. Ping monitors are not affected. - **Unique Subject CN per certificate (mTLS only)** — Each client certificate must have a unique Subject Common Name (CN). Chrome uses the CN to identify and auto-select the correct certificate per hostname. ## Permanent data storage [#permanent-data-storage] Users may want to use permanent data storage to provide the `user_defined_variables.json` file or support custom node modules. **For Docker** To set permanent data storage on Docker: 1. Create a directory on the host where you are launching the Job Manager. This is your source directory. 2. Launch the Job Manager, mounting the source directory to the target directory `/var/lib/newrelic/synthetics`. Example: ```sh docker run ... -v /sjm-volume:/var/lib/newrelic/synthetics:rw ... ``` **For Podman** To set permanent data storage on Podman: 1. Create a directory on the host where you are launching the Job Manager. This is your source directory. 2. Launch the Job Manager, mounting the source directory to the target directory `/var/lib/newrelic/synthetics`. Example: ```sh podman run ... -v /sjm-volume:/var/lib/newrelic/synthetics:rw,z ... ``` **For Kubernetes** ### Kubernetes To set permanent data storage on Kubernetes, the user has two options: 1. Provide an existing PersistentVolumeClaim (PVC) for an existing PersistentVolume (PV), setting the `synthetics.persistence.existingClaimName` configuration value. Example: ```sh helm install ... --set synthetics.persistence.existingClaimName=sjm-claim ... ``` 2. Provide an existing PersistentVolume (PV) name, setting the `synthetics.persistence.existingVolumeName` configuration value. Helm will generate a PVC for the user. The user may optionally set the following values as well: - `synthetics.persistence.storageClass`: The storage class of the existing PV. If not provided, Kubernetes will use the default storage class. - `synthetics.persistence.size`: The size for the claim. If not set, the default is currently 2Gi. ```sh helm install ... --set synthetics.persistence.existingVolumeName=sjm-volume --set synthetics.persistence.storageClass=standard ... ``` ## Sizing considerations [#vm-sizing] **For Docker and Podman** To ensure your private location runs efficiently, you must provision enough CPU resources on your host to handle your monitoring workload. Many factors impact sizing, but you can quickly estimate your needs. You'll need **1 CPU core for each heavyweight monitor** (i.e., simple browser, scripted browser, or scripted API monitor). Below are two formulas to help you calculate the number of cores you need, whether you're diagnosing a current setup or planning for a future one. ### Formula 1: Diagnosing an Existing Location If your current private location is struggling to keep up and you suspect jobs are queuing, use this formula to find out how many cores you actually need. It's based on the observable performance of your system. C\_{est} = (R\_{proc} + R\_{growth}) \\cdot D\_{avg,m} - C\_{est} = **Estimated CPU Cores**. - R\_{proc} = The **rate** of heavyweight jobs being **processed** per minute. - R\_{growth} = The **rate** your `jobManagerHeavyweightJobs` queue is **growing** per minute. - D\_{avg,m} = The **average duration** of heavyweight jobs in **minutes**. This formula calculates your true job arrival rate by adding the jobs your system _is processing_ to the jobs that are _piling up_ in the queue. Multiplying this total load by the average job duration tells you exactly how many cores you need to clear all the work without queuing. ### Formula 2: Forecasting a New or Future Location If you're setting up a new private location or planning to add more monitors, use this formula to forecast your needs ahead of time. C\_{est} = N\_{mon} \\cdot D\_{avg,m} \\cdot \\frac{1}P\_{avg,m} - C\_{est} = **Estimated CPU Cores**. - N\_{mon} = The total **number** of heavyweight **monitors** you plan to run. - D\_{avg,m} = The **average duration** of a heavyweight job in **minutes**. - P\_{avg,m} = The **average period** for heavyweight monitors in **minutes** (e.g., a monitor that runs every 5 minutes has P\_{avg,m} = 5). This calculates your expected workload from first principles: how many monitors you have, how often they run, and how long they take. **Important sizing factors** When using these formulas, remember to account for these factors: - **Job duration (D\_{avg,m}):** Your average should include jobs that **time out** (often 3 minutes), as these hold a core for their entire duration. - **Job failures and retries:** When a monitor fails, it's automatically retried. These retries are additional jobs that add to the total load. A monitor that consistently fails and retries **effectively multiplies its period**, significantly impacting throughput. - **Scaling out:** In addition to adding more cores to a host (scaling up), you can deploy additional synthetics job managers with the same private location key to load balance jobs across multiple environments (scaling out). It's important to note that a single Synthetics Job Manager (SJM) has a throughput limit of **approximately 15 heavyweight jobs per minute**. This is due to an internal threading strategy that favors the efficient competition of jobs across multiple SJMs over the raw number of jobs processed per SJM. If your calculations indicate a need for higher throughput, you must **scale out** by deploying additional SJMs. You can [check if your job queue is growing](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/job-manager-maintenance-monitoring/) to determine if more SJMs are needed. Adding more SJMs with the same private location key provides several advantages: - **Load balancing**: Jobs for the private location are distributed across all available SJMs. - **Failover protection**: If one SJM instance goes down, others can continue processing jobs. - **Higher total throughput**: The total throughput for your private location becomes the sum of the throughput from each SJM (e.g., two SJMs provide up to 30 jobs/minute). ### NRQL queries for diagnosis You can run these queries in the [query builder](https://docs.newrelic.com/docs/query-your-data/explore-query-data/get-started/introduction-querying-new-relic-data/) to get the inputs for the diagnostic formula. Make sure to set the time range to a long enough period to get a stable average. **1. Find the rate of jobs processed per minute (R\_{proc})**: This query counts the number of non-ping (heavyweight) jobs completed over the last day and shows the average rate per minute. ```sql FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) AS 'job rate per minute' WHERE location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE' SINCE 1 day ago ``` **2. Find the rate of queue growth per minute (R\_{growth})**: This query calculates the average per-minute growth of the `jobManagerHeavyweightJobs` queue on a time series chart. A line above zero indicates the queue is growing, while a line below zero means it's shrinking. ```sql FROM SyntheticsPrivateLocationStatus SELECT derivative(jobManagerHeavyweightJobs, 1 minute) AS 'queue growth rate per minute' WHERE name = 'YOUR_PRIVATE_LOCATION' TIMESERIES SINCE 1 day ago ``` > #### 💡 TIP > > Make sure to select the account where the private location exists. It's best to view this query as a time series because the derivative function can vary wildly. The goal is to get an estimate of the rate of queue growth per minute. Play with different time ranges to see what works best. **3. Find total number of heavyweight monitors (N\_{mon})**: This query finds the unique count of heavyweight monitors. ```sql FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'monitor count' WHERE location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE' SINCE 1 day ago ``` **4. Find average job duration in minutes (D\_{avg,m})**: This query finds the average execution duration of completed non-ping jobs and converts the result from milliseconds to minutes. `executionDuration` represents the time the job took to execute on the host. ```sql FROM SyntheticCheck SELECT average(executionDuration)/60e3 AS 'avg job duration (m)' WHERE location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE' SINCE 1 day ago ``` **5. Find average heavyweight monitor period (P\_{avg,m}):** If the private location's `jobManagerHeavyweightJobs` queue is growing, it isn't accurate to calculate the average monitor period from existing results. This will need to be estimated from the list of monitors on the [Synthetic Monitors](https://one.newrelic.com/synthetics) page. Make sure to select the correct New Relic account and you may need to filter by `privateLocation`. > #### 💡 TIP > > Synthetic monitors may exist in multiple sub accounts. If you have more sub accounts than can be selected in the query builder, choose the accounts with the most monitors. ### Note about ping monitors and the `pingJobs` queue **Ping monitors are different.** They are lightweight jobs that do not consume a full CPU core each. Instead, they use a separate queue (`pingJobs`) and run on a pool of worker threads. While they are less resource-intensive, a high volume of ping jobs, especially failing ones, can still cause performance issues. Keep these points in mind: - **Resource model:** Ping jobs utilize worker threads, not dedicated CPU cores. The core-per-job calculation does not apply to them. - **Timeout and retry:** A failing ping job can occupy a worker thread for up to **60 seconds**. It first attempts an HTTP HEAD request (30-second timeout). If that fails, it immediately retries with an HTTP GET request (another 30-second timeout). - **Scaling:** Although the sizing formula is different, the same principles apply. To handle a large volume of ping jobs and keep the `pingJobs` queue from growing, you may need to scale up and/or scale out. Scaling up means increasing cpu and memory resources per host or namespace. Scaling out means adding more instances of the ping runtime. This can be done by deploying more job managers on more hosts, in more namespaces, or even [within the same namespace](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/job-manager-configuration#scaling-out-with-multiple-sjm-instances). Alternatively, the `ping-runtime` in Kubernetes allows you to set [a larger number of replicas](https://github.com/newrelic/helm-charts/blob/41c03e287dafd41b9c914e5a6c720d5aa5c01ace/charts/synthetics-job-manager/values.yaml#L173) per deployment. **For Kubernetes and OpenShift** Each runtime used by the Kubernetes and OpenShift synthetic job manager can be sized independently by setting values in the [helm chart](https://github.com/newrelic/helm-charts/tree/master/charts/synthetics-job-manager). The [node-api-runtime](https://github.com/newrelic/helm-charts/tree/master/charts/synthetics-job-manager/charts/node-api-runtime) and [node-browser-runtime](https://github.com/newrelic/helm-charts/tree/master/charts/synthetics-job-manager/charts/node-browser-runtime) are sized independently using a combination of the `parallelism` and `completions` settings. - The `parallelism` setting controls how many pods of a particular runtime run concurrently. - The `completions` setting controls how many pods must complete before the `CronJob` starts another Kubernetes Job for that runtime. ### Best practices for sizing your deployment It's often not possible to precisely calculate the needed parallelism and completions values because the average duration as seen in New Relic might not be accurate, especially if the existing private location is not working well. Follow this practical approach to dial in parallelism and completions. The equations below can be used to get ballpark values to start from. **1. Estimate completions and parallelism** Do your best to estimate the average execution duration and number of jobs per 5 minutes. This provides you with a ballpark starting point for the next step, which will involve trial and error to tune the parallelism and completions values in a working cluster. Make sure to scale them proportionally, for example, going from the defaults of 1 and 6 to 10 and 60. **Estimated Completions:** This determines how long your 5-minute job load will take to complete. ```sql -- Get average execution duration in minutes FROM SyntheticCheck SELECT average(executionDuration / 60e3) AS 'Avg Duration (min)' WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION' SINCE 1 hour ago ``` Completions = \\frac{5}D\_{avg,m} Where D\_{avg,m} is your **average job execution duration in minutes**. **Estimated Parallelism:** This determines how many workers (pods) you need running concurrently to handle your 5-minute job load. ```sql -- Get jobs per 5 minutes FROM SyntheticCheck SELECT rate(uniqueCount(id), 5 minutes) AS 'Number of monitor jobs per 5 minutes' WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION' SINCE 1 hour ago ``` P\_{est} = \\frac{N_m}{Completions} Where N_m is your **number of jobs per 5 minutes**. This P\_{est} value is your **estimated parallelism**. **2. Perform a Helm deploy** Perform a Helm deploy with estimated parallelism and completions values, and your best guess for `ping-runtime.replicaCount` given the number of cpu cores per node and the number of ping monitors that need to run per minute. **3. Monitor queue growth** With the synthetic monitors configured to send jobs to the private location, check for queue growth on a timeseries line chart for `pingJobs` and `jobManagerHeavyweightJobs`. - If the `pingJobs` queue has a positive slope, increase `ping-runtime.replicaCount` and redeploy. - If the `jobManagerHeavyweightJobs` queue has a positive slope, increase `parallelism` and `completions` proportionally until the queue is no longer growing (negative slope). A negative slope indicates that the job manager has enough parallelism to handle the job demand. It will eventually reach zero with a negative slope. ```sql FROM SyntheticsPrivateLocationStatus SELECT average(jobManagerHeavyweightJobs) AS 'Heavyweight Queue Growth', average(pingJobs) AS 'Ping Queue Growth' WHERE name = 'YOUR_PRIVATE_LOCATION' SINCE 1 day ago TIMESERIES ``` **4. Tune based on pod running state** With the queue decreasing or at zero, check for `node-api-runtime` and `node-browser-runtime` pods that are in a "running" state for 10+ minutes. This indicates that parallelism is set too high and there are more pods than needed. To avoid wasting resources unnecessarily, decrease `parallelism` and `completions` to reduce the age of each "running" runtime pod. If targeting a Kubernets job age of 5 minutes, runtime pods should be in a running state for less than 5 minutes, meaning the pod was created, it quickly received a job to run and completed. **5. Scale out if necessary** If the queue is not decreasing, yet there are many pods in a "running" state for 10+ minutes, it's likely that the job manager is hitting its performance bottleneck. The next thing to do is decrease parallelism and scale out with one or more additional deployments. For example, with `parallelism: 100`, `completions: 600` the queue is still growing yet there are many pods in a "running" state for 10+ minutes, and the Kubernetes Job age is 20 minutes ... set `parallelism: 50`, `completions: 200` and scale horizontally (out) by adding 2 additional deployments. This yields a total of 150 parallel pods and should reduce the K8s job age to less than 20 minutes while also reducing the number of long-lived "running" pods. Aim for a K8s job age of 5-10 minutes. For more information on adding deployments, see [Scaling out with multiple SJM deployments](#scaling-out-with-multiple-sjm-deployments). > #### 💡 TIP > > You can use the following query to help determine if you need to scale out. > > **Note:** Monitors can exist in multiple sub-accounts. > > ```sql > -- monitors per minute per SJM > FROM SyntheticCheck SELECT > round(rate(uniqueCount(id), 1 minute)/uniqueCount(minionId),0.1) AS 'heavy jobs per minute per SJM', > uniqueCount(minionId) AS 'number of SJMs (namespaces)', > round(rate(uniqueCount(id), 1 minute),0.1) AS 'heavy jobs per minute total' > WHERE minionContainerSystem = 'KUBERNETES' AND minionDeploymentMode = 'private' AND location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE' FACET location SINCE 1 hour ago TIMESERIES > ``` > #### 💡 TIP > > Reducing the number of K8s job cycles can also improve performance. As each cycle reaches the set number of completions, there are fewer and fewer "running" pods to take on new Synthetics jobs. For example, with completions set to 200 and parallelism set to 50, we initially have 50 running pods, but this starts to decrease as we pass 150 completions. At 199 completions, only 1 running pod remains. > > Setting a larger value for completions is not a bad idea, but it can lead to warning events in K8s about `TooManyMissedTimes` for the cronjob. ### Scaling out with multiple SJM deployments To scale beyond the 15 jobs/minute throughput of a single SJM, you must install **multiple, separate SJM Helm releases**. > #### ⚠️ IMPORTANT > > Do not use replicas to scale the job manager pod. The SJM architecture requires a 1:1 relationship between a runtime pod and its parent SJM pod. If runtime pods send results back to the wrong SJM replica (e.g., through a Kubernetes service), those results will be lost. However, ping-runtime.replicaCount is okay to use. The correct strategy is to deploy multiple SJM instances, each as its own Helm release. Each SJM will compete for jobs from the same private location, providing load balancing, failover protection, and an increased total job throughput. #### Horizontal scaling strategy If you need to scale out, you can simplify maintenance by treating each SJM deployment as a fixed-capacity unit. 1. **Set Parallelism:** For _each_ SJM, set `parallelism` to the same maximum that a single SJM can handle without creating too many long-lived "running" runtime pods. This maximizes the potential throughput of each SJM without wasting resources. 2. **Set Completions:** For _each_ SJM, set `completions` to the same fixed value as well. Adjust as needed to target a 5 minute Kubernetes job age per runtime, i.e., node-browser-runtime and node-api-runtime. 3. **Install Releases:** Install as many separate Helm releases as you need to handle your total job demand, i.e., get the queue to zero or line chart to a negative slope. 4. **Monitor and Add:** Monitor the private location job queue. If it starts to grow (positive slope), simply install another Helm release (e.g., `sjm-delta`) using the same fixed configuration. By fixing parallelism and completions to static values, increasing or decreasing capacity becomes a simpler process of **adding or removing Helm releases**. This helps to avoid wasting cluster resources on a parallelism value that is higher than the SJM can effectively utilize. #### Installation example When installing multiple SJM releases, you must provide a **unique name for each release**. All instances must be configured with the **same private location key**. Setting the `fullnameOverride` is highly recommended to create shorter, more manageable resource names. For example, to install two SJMs named `sjm-alpha` and `sjm-beta` into the `newrelic` namespace (both using the same `values.yaml` with your fixed parallelism and completions): ```sh # Install the first SJM deployment helm upgrade --install sjm-alpha newrelic/synthetics-job-manager \ -n newrelic \ -f values.yaml \ --set fullnameOverride=sjm-alpha \ --set ping-runtime.fullnameOverride=sjm-alpha-ping \ --set node-api-runtime.fullnameOverride=sjm-alpha-api \ --set node-browser-runtime.fullnameOverride=sjm-alpha-browser ``` ```sh # Install the second SJM deployment to add capacity helm upgrade --install sjm-beta newrelic/synthetics-job-manager \ -n newrelic \ -f values.yaml \ --set fullnameOverride=sjm-beta \ --set ping-runtime.fullnameOverride=sjm-beta-ping \ --set node-api-runtime.fullnameOverride=sjm-beta-api \ --set node-browser-runtime.fullnameOverride=sjm-beta-browser ``` You can continue this pattern (`sjm-charlie`, `sjm-delta`, etc.) for as many SJMs as needed to keep the job queue from growing.