Troubleshoot private locations

If you encounter problems while setting up private locations for synthetic monitoring of New Relic, try these troubleshooting tips to identify common Synthetics Job Manager problems.

Job demand and consumption

Identify your current throughput and assess whether your job manager is able to keep up with the job load.

Jobs demand throughput per minute:
```
SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest 
WHERE type != 'SIMPLE' FACET location SINCE 2 days ago
```

Jobs per minute over time:

SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest 
WHERE type != 'SIMPLE' FACET location SINCE 2 weeks ago TIMESERIES

Identify queue growth:

SELECT derivative(checksPending, 1 minute) AS 'queue growth rate (per minute)' 
FROM SyntheticsPrivateLocationStatus WHERE name = 'YOUR_PRIVATE_LOCATION' SINCE 2 days ago TIMESERIES

Consumption rate:

SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest 
WHERE type != 'SIMPLE' FACET location SINCE 2 weeks ago TIMESERIES

Clean up failing monitors that are consuming resources:

SELECT 100*latest(minionJobsFailed)/latest(minionJobsReceived) AS 'job failure rate (%)' 
FROM SyntheticsPrivateMinion FACET minionLocation SINCE 2 weeks ago TIMESERIES MAX

Memory usage troubleshooting

Recommended memory allocation per synthetics job manager:

Docker: 3.256 GiB per cpu core for Docker synthetics job manager
Kubernetes synthetics job manager (values are configurable):
- 1.6 GiB for job manager pod
- 1.0 GiB for ping runtime pod
- 2 GiB per Node.js API runtime pod
- 3 GiB per Node.js browser runtime pod
Target 60% max memory utilization

Helpful queries to identify current usage:

Memory usage per location:

SELECT latest(minionPhysicalMemoryUsedPercentage) FROM SyntheticsPrivateMinion
FACET minionLocation SINCE 2 weeks ago TIMESERIES MAX

Memory usage per hostname:

SELECT latest(minionPhysicalMemoryUsedPercentage) FROM SyntheticsPrivateMinion 
WHERE minionLocation = 'YOUR_PRIVATE_LOCATION' FACET minionHostname SINCE 2 weeks ago TIMESERIES MAX

Browser monitors crash due to shared memory sizing on Podman

Browser monitors running on a Podman-based private location might fail intermittently with an error like this:

NoSuchSessionError: invalid session id: session deleted as the browser has closed the connection, disconnected: Unable to receive message from renderer

This is typically a Chrome renderer process crash caused by insufficient shared memory (/dev/shm), not a scripting or network issue. Chrome relies heavily on shared memory for rendering and tends to exhaust it faster than Firefox, especially on pages that redirect across origins during login flows—each new origin spins up additional renderer processes that draw from the same /dev/shm pool.

How Podman pods allocate shared memory

When synthetics job manager runs in Podman with --pod, every container joined to that pod shares a single IPC namespace. The pod itself mounts /dev/shm once, at the size you created the pod with (64 MiB by default). You set that size at podman pod create time, and an individual container can't change or override it afterward.

This means the following common attempts to fix shared memory issues don't work once a container has joined a pod, because the pod's /dev/shm mount takes precedence over per-container settings:

Passing --shm-size on podman run instead of podman pod create
Bind-mounting with -v /dev/shm:/dev/shm on podman run

Diagnose

Before you change anything, run these checks to confirm the cause:

Check the --shm-size value in the pod's create command:
bash
```
$podman pod inspect YOUR_POD_NAME --format '{{.CreateCommand}}'
```
This prints the exact podman pod create command that created the pod. Look for --shm-size in that output. If it's missing, the pod is running on the 64 MiB default regardless of what you intended to set—this can happen if you didn't fully remove a previous pod with the same name before recreating it, or didn't restart a container in the pod afterward.

Find the browser runtime container's ID:

bash

$podman ps --filter pod=YOUR_POD_NAME --filter ancestor=newrelic/synthetics-node-browser-runtime --format '{{.ID}}'

Use this ID as YOUR_BROWSER_RUNTIME_CONTAINER_ID in the next step.

Check the shared memory actually available inside that container, ideally captured during or immediately before a failing run:
bash
```
$podman exec YOUR_BROWSER_RUNTIME_CONTAINER_ID df -h /dev/shm
```
Compare the output to the size you configured:
- If Size still shows 64M, the resize never took effect—recreate the pod as described in the next section.
- If Size shows your configured value (for example, 2.0G for a pod created with --shm-size=2g), check the Used and Avail columns. If Avail is near zero, the pool is exhausted—every container in the pod draws from this same shared allocation, so if multiple monitors run concurrently on the same private location, size --shm-size larger or reduce how many run at once. If Avail still has headroom, the pool isn't the cause of this particular crash and you should look elsewhere (for example, HEAVY_WORKER_MEMORY).

Fix

Remove the existing pod and recreate it with a larger --shm-size, then restart synthetics job manager so it rejoins the new pod:

bash

$podman pod stop YOUR_POD_NAME
$podman pod rm YOUR_POD_NAME
$podman pod create --network slirp4netns --name YOUR_POD_NAME --add-host=podman.service:HOST_IP_ADDRESS --shm-size=2g

Start with 2g, the value the install docs use. If the check in step 3 still shows the pool is exhausted, increase it further (for example, to 4g) before you reduce concurrency, and re-check df -h /dev/shm after each change.

Then start synthetics job manager again using the podman run command from the Podman start procedure—removing the pod also removes the containers that were running in it.

CPU usage troubleshooting

Identify potential periods of high cpu demand:

Overall CPU usage:

SELECT latest(minionProcessorsUsagePercentage) FROM SyntheticsPrivateMinion 
FACET minionLocation SINCE 2 weeks ago TIMESERIES MAX

CPU usage for a specified location:

SELECT latest(minionProcessorsUsagePercentage) FROM SyntheticsPrivateMinion 
WHERE minionLocation = 'YOUR_PRIVATE_LOCATION' FACET minionHostname SINCE 2 weeks ago TIMESERIES MAX

Network troubleshooting

Network access is red:

The Synthetics Job Manager makes a check to New Relic to verify public network access. If public network access is not available, the Network Access icon on the Synthetics Job Manager's Overview page will turn red, but the Synthetics Job Manager will otherwise be fully functional. If this is the expected behavior, you can safely ignore this error.

You may also see log entries like this:

2016-01-21 21:47:09,401 [dw-22 - GET /private-location/status] c.n.s.m.util.PrivateMinionUtils WARN Unable to detect Public Network Access trying to fetch https://newrelic.com/synthetics: java.net.ConnectException: Connection refused

New Relic access is red:

The Synthetics Job Manager must have access to https://synthetics-horde.nr-data.net in order to retrieve the list of jobs to run and publish results that you can view in your New Relic user interface. If your firewall rules don't permit direct access, you can configure proxy access. If necessary, add the DNS endpoint synthetics-horde.nr-data.net to your allow list.

Port conflicts running Docker containers on the host

If you're running multiple job manager containers on the same host, you'll have port conflicts and resource contention. To avoid this, make sure to run each job manager on a separate host.

Job demand and consumption .css-21sua1{background:none;border:none;width:0;padding:0;}