When using synthetic monitoring's private locations with New Relic's alerts, you can be notified if a location is under-provisioned, mis-configured, or generally misbehaving.
This guide will help you answer the following basic questions regarding private location health by using New Relic dashboards and NRQL alerts:
- Are my private job managers or minions online?
- Does my private location need more job managers or minions?
- Can I check the status of a specific job manager or minion directly?
Prerequisites
Before following the instructions in this guide, ensure you have:
- A synthetic private location
- At least one private minion installed at that location
- Checks scheduled to run at that location
- An alert policy for the private location, with a configured notification channel to notify your team when a violation occurs
The following Private Minion dashboard example JSON can be imported to your account using:
{ "name": "Synthetics Private Minions", "description": "Details on events from SyntheticsPrivateLocationStatus, SyntheticsPrivateMinion, SyntheticCheck, SyntheticRequest, NrAuditEvent, plus a page for Performance Analysis.", "permissions": "PUBLIC_READ_WRITE", "pages": [ { "name": "SyntheticsPrivateLocationStatus", "description": null, "widgets": [ { "title": "queue size by location", "layout": { "column": 1, "row": 1, "width": 3, "height": 5 }, "linkedEntityGuids": null, "visualization": { "id": "viz.bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT latest(checksPending) FACET name" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "private location queues", "layout": { "column": 4, "row": 1, "width": 9, "height": 5 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT latest(pingJobs),latest(jobManagerHeavyweightJobs),latest(minionHeavyweightJobs) FACET name TIMESERIES MAX LIMIT 40 SINCE 2 weeks ago" } ], "platformOptions": { "ignoreTimeRange": false }, "yAxisLeft": { "zero": true } } }, { "title": "queue size by location and type", "layout": { "column": 1, "row": 6, "width": 3, "height": 5 }, "linkedEntityGuids": null, "visualization": { "id": "viz.bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT latest(pingJobs),latest(jobManagerHeavyweightJobs),latest(minionHeavyweightJobs) FACET name " } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "rate of queue growth", "layout": { "column": 4, "row": 6, "width": 9, "height": 5 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT clamp_min(derivative(pingJobs,1 minute),0) AS 'ping',clamp_min(derivative(jobManagerHeavyweightJobs,1 minute),0) AS 'job manager heavy',clamp_min(derivative(minionHeavyweightJobs,1 minute),0) AS 'minion heavy' FACET name TIMESERIES MAX LIMIT 40 SINCE 2 weeks ago" } ], "platformOptions": { "ignoreTimeRange": false }, "yAxisLeft": { "zero": true } } }, { "title": "audit events for private locations", "layout": { "column": 1, "row": 11, "width": 12, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM NrAuditEvent SELECT * WHERE targetType = 'PRIVATE_LOCATION' LIMIT MAX" } ] } } ] }, { "name": "SyntheticsPrivateMinion", "description": null, "widgets": [ { "title": "private locations", "layout": { "column": 1, "row": 1, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'minions',latest(timestamp) WHERE minionLocation NOT LIKE 'AWS_%' FACET minionLocation" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "minion details", "layout": { "column": 4, "row": 1, "width": 9, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT latest(minionId),uniqueCount(minionId) - 1 AS 'restarts',latest(minionIpv4),latest(timestamp),latest(minionStartTimestamp),max(minionJobsQueued),max(minionJobsFinished),100*max(minionJobsFailed)/max(minionJobsQueued) AS 'job failure rate',latest(minionJobsRunning),max(minionJobsTimedout),max(minionJobsSkipped),max(minionJobsInternalEngineError),latest(minionWorkers),latest(minionProcessors),latest(minionSwapMemoryUsedBytes/pow(1024,3)) AS 'used swap (GiB)',latest(minionSwapMemoryTotalBytes/pow(1024,3)) AS 'total swap (GiB)',latest(minionPhysicalMemoryUsedBytes/pow(1024,3)) AS 'used memory (GiB)',latest(minionPhysicalMemoryTotalBytes/pow(1024,3)) AS 'total memory (GiB)',latest(minionPhysicalMemoryTotalBytes/pow(1024,3))/latest(minionWorkers) AS 'memory (GiB) per heavy worker' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionBuildVersion,minionBuildNumber,minionLocation,minionOsVersion,minionContainerSystemVersion,minionHostname LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "minion instances", "layout": { "column": 1, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'minions' WHERE minionLocation NOT LIKE 'AWS_%'" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "restarts", "layout": { "column": 3, "row": 4, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.area" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) WHERE minionLocation NOT LIKE 'AWS_%' FACET minionId TIMESERIES LIMIT 50" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Minion container runtimes", "layout": { "column": 9, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'container versions' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionContainerSystemVersion LIMIT MAX" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Minion builds", "layout": { "column": 1, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'build numbers' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionBuildVersion,minionBuildNumber LIMIT MAX" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Minion OSes", "layout": { "column": 5, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'os versions' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionOsVersion LIMIT MAX" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Minion IPs", "layout": { "column": 9, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'IP addresses' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionIpv4 LIMIT MAX" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "CPU utilization (%)", "layout": { "column": 1, "row": 10, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT average(minionProcessorsUsagePercentage) AS 'avg',max(minionProcessorsUsagePercentage) AS 'max' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "memory utilization (%)", "layout": { "column": 4, "row": 10, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT average(minionPhysicalMemoryUsedPercentage) AS 'average',max(minionPhysicalMemoryUsedPercentage) AS 'max' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "used vs free memory (GiB)", "layout": { "column": 7, "row": 10, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.area" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionPhysicalMemoryUsedBytes) / pow(1024,3) AS 'used',(max(minionPhysicalMemoryTotalBytes)-max(minionPhysicalMemoryUsedBytes)) / pow(1024,3) AS 'free' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "used vs free swap (GiB)", "layout": { "column": 10, "row": 10, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.area" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionSwapMemoryUsedBytes) / pow(1024,3) AS 'used',(max(minionSwapMemoryTotalBytes)-max(minionSwapMemoryUsedBytes)) / pow(1024,3) AS 'free' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "jobs running", "layout": { "column": 1, "row": 13, "width": 4, "height": 9 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsRunning) AS 'concurrent',clamp_min(derivative(minionJobsFinished,1 minute),0) AS 'finished per minute' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs finished", "layout": { "column": 5, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsFinished) - min(minionJobsFinished) AS 'total jobs' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job failure rates", "layout": { "column": 9, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT (max(minionJobsFailed/minionJobsReceived))*100 AS 'failed rate',(max(minionJobsInternalEngineError/minionJobsReceived))*100 AS 'IEE rate' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "yAxisLeft": { "zero": true } } }, { "title": "jobs skipped", "layout": { "column": 5, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsSkipped) - min(minionJobsSkipped) AS 'skipped jobs' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionHostname TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "yAxisLeft": { "zero": false } } }, { "title": "count of Internal Engine Errors (IEE) vs failed jobs", "layout": { "column": 9, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsInternalEngineError) AS 'IEE',max(minionJobsFailed) AS 'failed jobs' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": false } } }, { "title": "jobs timed out", "layout": { "column": 5, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsTimedout) - min(minionJobsTimedout) AS 'timed out jobs' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionHostname TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "yAxisLeft": { "zero": true } } }, { "title": "IEE as a percentage of failed jobs", "layout": { "column": 9, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsInternalEngineError/minionJobsFailed)*100 AS 'IEE as a percentage of failed jobs' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false } } } ] }, { "name": "SyntheticCheck", "description": null, "widgets": [ { "title": "minion IDs by location", "layout": { "column": 1, "row": 1, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT latest(minion) WHERE location NOT LIKE 'AWS_%' FACET locationLabel,minionId LIMIT MAX" } ] } }, { "title": "minion count", "layout": { "column": 7, "row": 1, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(minionId) WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by minion", "layout": { "column": 9, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET minion WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "avg monitor frequency by type (m)", "layout": { "column": 1, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "yAxisLeft": { "zero": true } } }, { "title": "avg monitor frequency by location (m)", "layout": { "column": 5, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET locationLabel TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "avg monitor frequency by status (m)", "layout": { "column": 7, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by monitor", "layout": { "column": 9, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET monitorId WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "monitors by location", "layout": { "column": 1, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET locationLabel" } ] } }, { "title": "monitors by location", "layout": { "column": 5, "row": 7, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET locationLabel TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by location", "layout": { "column": 7, "row": 7, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' FACET locationLabel TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by location", "layout": { "column": 9, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET locationLabel WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "monitors by runtime type", "layout": { "column": 1, "row": 10, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET runtimeType" } ] } }, { "title": "monitors by runtime type", "layout": { "column": 5, "row": 10, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET runtimeType TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by runtime type", "layout": { "column": 7, "row": 10, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' FACET runtimeType TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by runtime type", "layout": { "column": 9, "row": 10, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET runtimeType WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "monitors by type", "layout": { "column": 1, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET type" } ] } }, { "title": "monitors by type", "layout": { "column": 5, "row": 13, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by type", "layout": { "column": 7, "row": 13, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by type", "layout": { "column": 9, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET type WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "monitors by status", "layout": { "column": 1, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET result" } ] } }, { "title": "monitors by status", "layout": { "column": 5, "row": 16, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by status", "layout": { "column": 7, "row": 16, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by status", "layout": { "column": 9, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET result WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "successful monitors by type", "layout": { "column": 1, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type" } ] } }, { "title": "successful monitors by type", "layout": { "column": 5, "row": 19, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration for successes by type", "layout": { "column": 7, "row": 19, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration for successes by type", "layout": { "column": 9, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET type WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "failed monitors by type", "layout": { "column": 1, "row": 22, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type" } ] } }, { "title": "failed monitors by type", "layout": { "column": 5, "row": 22, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration for failures by type", "layout": { "column": 7, "row": 22, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration for failures by type", "layout": { "column": 9, "row": 22, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(duration) AS 'abnormal job duration' FACET type WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" } } }, { "title": "ping monitors by location", "layout": { "column": 1, "row": 25, "width": 4, "height": 6 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'monitors with results',uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg job frequency (m)',average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' AND type = 'SIMPLE' FACET locationLabel" } ] } }, { "title": "top 5 ping errors", "layout": { "column": 5, "row": 25, "width": 8, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT count(*),latest(monitorId) AS 'latest monitor id',latest(type),average(duration) AS 'avg job duration',max(timestamp) AS 'last occurred' WHERE type = 'SIMPLE' AND result = 'FAILED' AND location NOT LIKE 'AWS_%' FACET error LIMIT 5" } ] } }, { "title": "ping duration with monitor count", "layout": { "column": 5, "row": 28, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration',uniqueCount(monitorId) AS 'monitor count' WHERE type = 'SIMPLE' AND location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "ping timeout vs other error durations", "layout": { "column": 7, "row": 28, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) WHERE type = 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET cases(WHERE error NOT LIKE '%timeout%' AS 'other error duration', WHERE error LIKE '%timeout%' AS 'timeout duration') TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "ping jobs per minute", "layout": { "column": 9, "row": 28, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' AND type = 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "ping timeout rate", "layout": { "column": 11, "row": 28, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT percentage(count(*), WHERE error LIKE '%timeout%' OR error LIKE '%timed-out%') WHERE location NOT LIKE 'AWS_%' AND type = 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "non-ping monitors by location", "layout": { "column": 1, "row": 31, "width": 4, "height": 6 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'monitors with results',uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg job frequency (m)',average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' AND type != 'SIMPLE' FACET locationLabel" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "top 5 non-ping errors", "layout": { "column": 5, "row": 31, "width": 8, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT count(*),latest(monitorId) AS 'latest monitor id',latest(type),average(duration) AS 'avg job duration',max(timestamp) AS 'last occurred' WHERE type != 'SIMPLE' AND result = 'FAILED' AND location NOT LIKE 'AWS_%' FACET error LIMIT 5" } ] } }, { "title": "non-ping duration with monitor count", "layout": { "column": 5, "row": 34, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg job duration',uniqueCount(monitorId) AS 'monitor count' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "non-ping timeout vs other error durations", "layout": { "column": 7, "row": 34, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET cases(WHERE error NOT LIKE '%timeout%' AS 'other error duration', WHERE error LIKE '%timeout%' AS 'timeout duration') TIMESERIES AUTO" } ], "platformOptions": { "ignoreTimeRange": false }, "units": { "unit": "MS" }, "yAxisLeft": { "zero": true } } }, { "title": "non-ping jobs per minute", "layout": { "column": 9, "row": 34, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' AND type != 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "non-ping timeout rate", "layout": { "column": 11, "row": 34, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT percentage(count(*), WHERE error LIKE '%timeout%' OR error LIKE '%timed-out%') WHERE location NOT LIKE 'AWS_%' AND type != 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs per minute by minion", "layout": { "column": 1, "row": 37, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' FACET minion TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "errors per minute by type", "layout": { "column": 7, "row": 37, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' AND error != '' AND error IS NOT NULL FACET type,result TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs per minute by type", "layout": { "column": 1, "row": 41, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "error counts by class", "layout": { "column": 7, "row": 41, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT count(*) WHERE location NOT LIKE 'AWS_%' AND result != 'SUCCESS' FACET capture(error, r'(?P<firstWord>\\S+)\\s.+') AS 'error class' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "monitor details", "layout": { "column": 1, "row": 45, "width": 12, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(id) AS 'results',uniqueCount(location)/rate(uniqueCount(id), 1 minute) AS 'avg job frequency (m)',average(duration) AS 'avg job duration' WHERE location NOT LIKE 'AWS_%' FACET monitorId,type,apiVersion,runtimeType LIMIT 50" } ] } } ] }, { "name": "SyntheticRequest", "description": null, "widgets": [ { "title": "locations", "layout": { "column": 1, "row": 1, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET locationLabel LIMIT 10" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "top 5 error response codes", "layout": { "column": 6, "row": 1, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(responseCode) AS 'responses' WHERE responseCode NOT IN (200,201,202,203,204,300,301,302,303,304) AND responseCode IS NOT NULL AND location NOT LIKE 'AWS_%' FACET responseCode,responseStatus TIMESERIES AUTO LIMIT 5" } ], "yAxisLeft": { "zero": true } } }, { "title": "Average duration (ms) by location", "layout": { "column": 1, "row": 4, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(durationDNS) AS 'DNS',average(durationSSL) AS 'SSL',average(durationConnect) AS 'Connect',average(durationSend) AS 'Send',average(durationWait) AS 'Wait',average(durationBlocked) AS 'Blocked',average(durationReceive) AS 'Receive',average(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET locationLabel" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "max duration (ms) by URL", "layout": { "column": 6, "row": 4, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(durationDNS) AS 'DNS',max(durationSSL) AS 'SSL',max(durationConnect) AS 'Connect',max(durationSend) AS 'Send',max(durationWait) AS 'Wait',max(durationBlocked) AS 'Blocked',max(durationReceive) AS 'Receive',max(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET URL" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Max duration (ms) by location", "layout": { "column": 1, "row": 7, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(durationDNS) AS 'DNS',max(durationSSL) AS 'SSL',max(durationConnect) AS 'Connect',max(durationSend) AS 'Send',max(durationWait) AS 'Wait',max(durationBlocked) AS 'Blocked',max(durationReceive) AS 'Receive',max(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET locationLabel" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Average duration (ms) by URL", "layout": { "column": 6, "row": 7, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(durationDNS) AS 'DNS',average(durationSSL) AS 'SSL',average(durationConnect) AS 'Connect',average(durationSend) AS 'Send',average(durationWait) AS 'Wait',average(durationBlocked) AS 'Blocked',average(durationReceive) AS 'Receive',average(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET URL" } ] } }, { "title": "request rate", "layout": { "column": 1, "row": 10, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT rate(uniqueCount(id),1 minute) AS 'request rate' WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "request rate by job", "layout": { "column": 6, "row": 10, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT rate(uniqueCount(id),1 minute) AS 'request rate' WHERE location NOT LIKE 'AWS_%' FACET id TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs per minion", "layout": { "column": 1, "row": 13, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT uniqueCount(id) AS 'jobs' WHERE location NOT LIKE 'AWS_%' FACET locationLabel,minion,minionId" } ] } }, { "title": "response status", "layout": { "column": 6, "row": 13, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET responseStatus LIMIT 10" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "content type", "layout": { "column": 9, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET contentType LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "incomplete jobs", "layout": { "column": 1, "row": 16, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT percentage(uniqueCount(id),WHERE domComplete IS NULL) AS 'percent incomplete',uniqueCount(id) AS 'jobs' WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET monitorName,monitorId,runtimeType LIMIT 20" } ] } }, { "title": "response codes", "layout": { "column": 6, "row": 16, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET responseCode LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "content category", "layout": { "column": 9, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET contentCategory LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "percent of incomplete requests", "layout": { "column": 1, "row": 19, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT percentage(count(*),WHERE domComplete IS NULL) AS 'percent incomplete',count(*) AS 'requests',latest(responseCode),latest(responseStatus) WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET domain" } ] } }, { "title": "complete vs incomplete jobs", "layout": { "column": 6, "row": 19, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT uniqueCount(id) AS 'number of jobs' WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET cases(WHERE domComplete IS NULL AS 'incomplete',WHERE domComplete IS NOT NULL AS 'complete')" } ] } }, { "title": "top 5 domains for incomplete jobs", "layout": { "column": 9, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) AS 'requests',latest(URL) WHERE domComplete IS NULL AND isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET domain LIMIT 5" } ] } }, { "title": "page load duration for complete vs incomplete jobs", "layout": { "column": 1, "row": 22, "width": 3, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(onPageLoad/1e3) WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET cases(WHERE domComplete IS NULL AS 'incomplete',WHERE domComplete IS NOT NULL AS 'complete') TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "page load duration for complete vs incomplete jobs by monitor", "layout": { "column": 4, "row": 22, "width": 9, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(onPageLoad/1e3) WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET cases(WHERE domComplete IS NULL AS 'incomplete',WHERE domComplete IS NOT NULL AS 'complete'),monitorName TIMESERIES MAX LIMIT MAX" } ], "yAxisLeft": { "zero": true } } }, { "title": "domains", "layout": { "column": 1, "row": 26, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "request type", "layout": { "column": 5, "row": 26, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET verb LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non HTTP 200 by domain", "layout": { "column": 9, "row": 26, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE responseCode != 200 AND location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "hosts", "layout": { "column": 1, "row": 29, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET host LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "requests by URL", "layout": { "column": 5, "row": 29, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET hierarchicalURL LIMIT 10" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non HTTP 200 by URL", "layout": { "column": 9, "row": 29, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE responseCode != 200 AND location NOT LIKE 'AWS_%' FACET hierarchicalURL LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "average request and response size by URL", "layout": { "column": 1, "row": 32, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(requestBodySize),average(requestHeaderSize),average(responseBodySize),average(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET URL LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "average request and response size by domain", "layout": { "column": 7, "row": 32, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(requestBodySize),average(requestHeaderSize),average(responseBodySize),average(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "max request and response size by URL", "layout": { "column": 1, "row": 35, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(requestBodySize),max(requestHeaderSize),max(responseBodySize),max(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET URL LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Max Request and Response size by Domain", "layout": { "column": 7, "row": 35, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(requestBodySize),max(requestHeaderSize),max(responseBodySize),max(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ] } }, { "title": "audit events for monitors", "layout": { "column": 1, "row": 38, "width": 12, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM NrAuditEvent SELECT * WHERE targetType = 'MONITOR' LIMIT MAX" } ] } } ] }, { "name": "CPM Performance Analysis", "description": null, "widgets": [ { "title": "", "layout": { "column": 1, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.markdown" }, "rawConfiguration": { "text": "See [this Google sheet](https://docs.google.com/spreadsheets/d/1k2Aw11r6-S8pIpXQUINQZ0T5qlxS8iSwSo9t9GKDvzc/edit?usp=sharing) to assist in calculating how many workers will be needed based on the values on this page. This will help you to assess the required size of each host (scaling up) and how many hosts you need (scaling out).\n\nFor more info: \nhttps://discuss.newrelic.com/t/relic-solution-scaling-and-rightsizing-for-the-cpm/123999" } }, { "title": "count of heavy worker threads and cpu cores", "layout": { "column": 5, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": [ "MzI1NjA5M3xWSVp8REFTSEJPQVJEfDU3MjU1Mjk" ], "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT latest(minionWorkers),latest(minionProcessors),latest(timestamp) FACET minionId,minionLocation WHERE minionLocation NOT LIKE 'AWS_%' SINCE 5 minutes ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "number of minions (1 per host or node)", "layout": { "column": 9, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'number of minions' FACET minionLocation WHERE minionLocation NOT LIKE 'AWS_%' SINCE 5 minutes ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping monitors", "layout": { "column": 1, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'non-ping monitors' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' FACET locationLabel SINCE 2 days ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping jobs per minute", "layout": { "column": 5, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) AS 'jobs per minute' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' FACET locationLabel SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping avg success duration", "layout": { "column": 9, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'avg success duration' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET locationLabel SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping avg failure duration", "layout": { "column": 1, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) AS 'failure duration' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET locationLabel SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping job failure rate", "layout": { "column": 5, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT percentage(count(*), WHERE result = 'FAILED') WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' FACET locationLabel SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } } ] }, { "name": "SJM Performance Analysis", "description": null, "widgets": [ { "title": "", "layout": { "column": 1, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.markdown" }, "rawConfiguration": { "text": "See [this Google sheet](https://docs.google.com/spreadsheets/d/1Se-xAuZd8TJitNMNviiEUHTG08rKDc0O4Hkl63DgDvE/edit?usp=sharing) to assist in calculating values for Parallelism and Completions on node-api-runtime and node-browser-runtime pods.\n\nFor more info: \nhttps://discuss.newrelic.com/t/relic-solution-scaling-and-rightsizing-for-the-cpm/123999" } }, { "title": "minionIds and cpu cores", "layout": { "column": 5, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": [ "MzI1NjA5M3xWSVp8REFTSEJPQVJEfDU3MjU1Mzc" ], "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT latest(minionProcessors),latest(timestamp) FACET minionId,minionLocation WHERE minionLocation NOT LIKE 'AWS_%' SINCE 5 minutes ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "number of job managers (1 per host or node)", "layout": { "column": 9, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'number of job managers' FACET minionLocation WHERE minionLocation NOT LIKE 'AWS_%' SINCE 5 minutes ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "jobs per minute", "layout": { "column": 1, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE runtimeType IS NOT NULL AND location NOT LIKE 'AWS_%' FACET runtimeType SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "queue growth per minute", "layout": { "column": 5, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT clamp_min(derivative(checksPending,1 minute),0) FACET name TIMESERIES 3 hours SLIDE BY 1 hour SINCE 1 week ago" } ], "platformOptions": { "ignoreTimeRange": false }, "yAxisLeft": { "zero": true } } }, { "title": "avg success duration", "layout": { "column": 9, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) WHERE runtimeType IS NOT NULL AND location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET runtimeType SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "avg failure duration", "layout": { "column": 1, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(duration) WHERE runtimeType AND location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET runtimeType SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "job failure rate", "layout": { "column": 5, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT percentage(count(*), WHERE result = 'FAILED') WHERE runtimeType IS NOT NULL AND location NOT LIKE 'AWS_%' FACET runtimeType SINCE 1 hour ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "number of monitors", "layout": { "column": 9, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE runtimeType IS NOT NULL AND location NOT LIKE 'AWS_%' FACET runtimeType SINCE 2 days ago" } ], "platformOptions": { "ignoreTimeRange": false } } } ] } ]}
Steps to import:
- Copy the Dashboard JSON and paste into a text editor.
- Replace
"accountId": 0,
with your New Relic account ID for each occurrence in the JSON code. - Copy the Dashboard JSON from your text editor and import using one of the methods described above.
- Edit any charts that you'd like to use facet filtering with.
Tip
If your private locations exist in a parent account and Synthetics monitors in a sub account, insert the parent account ID for NRQL queries that use SyntheticPrivateLocationStatus
and SyntheticsPrivateMinion
, and the sub account ID for queries that use SyntheticCheck
and SyntheticRequest
.
Are my private job managers or minions online?
To answer this question, you can rely on attributes from the SyntheticsPrivateMinion
event. Private synthetics job managers and minions send this event to New Relic every 30 seconds. A simple way to check if your job managers or minions are online is to compare the unique count of minion IDs with the number of job managers or minions you expect to be online.
To understand how many job managers or minions are reporting, run this example NRQL query:
SELECT uniqueCount(minionId) FROM SyntheticsPrivateMinion WHERE minionLocation = '1-acme_okc_dc-309'
Using this query, you can create an alert condition to notify your team when fewer job managers or minions are reporting than expected. This condition is configured with a static threshold of 2 units
, which means you will receive an alert if any of your job managers or minions are offline.
You can verify that the alert policy works as expected by manually stopping one of your minions. Then, when the alert violation occurs, you will be notified by any notification channels that have been set up. Once the job manager or minion is restarted and it comes back online, the alert will recover.
There are more robust ways to check whether job managers or minions are functioning correctly, but this query and condition simply and successfully handle the case where a machine fails, is accidentally decommissioned, or the job manager or minion process crashes. It also ensures that the job manager or minion can communicate with New Relic.
Does my private location need more job managers or minions?
To answer this question, you can use the checksPending
attribute of the SyntheticsPrivateLocationStatus
event. The checksPending
attribute reflects the number of monitor checks that are scheduled (or "queued") but have yet to be accepted by a synthetics job manager or minion in the designated location. For a location with scheduled checks and no job managers or minions, this graph would grow linearly up and to the right.
Additional attributes are available to identify what job types are causing the checksPending
attribute to grow and where to focus your troubleshooting efforts:
pingJobs
can be ran by both synthetics job manager and containerized private minion (CPM)jobManagerHeavyweightJobs
are non-ping jobs waiting to run on runtimes supported by synthetics job managerminionHeavyweightJobs
are non-ping jobs waiting to run on runtimes supported by containerized private minion (CPM)
This metric is more complicated to monitor than uniqueCount(minionId)
because a high value does not necessarily mean the location is in a bad state. As long as the metric is not growing linearly up and to the right (and checks are being run on schedule), the location is in a good state.
This use case is perfect for anomaly NRQL alert conditions, which allow you to monitor the deviation of a metric rather than its static value. For example:
SELECT average(checksPending) FROM SyntheticsPrivateLocationStatus WHERE name = '1-acme_tokyo_dc-512'
To test this alert condition, schedule one-minute, browser-based monitors to run from your location. Browser-based jobs consume more resources than ping jobs, which is why they are a better fit for load simulation. New Relic will quickly notify you of a growing number of pending checks.
After doubling the number of job managers or minions to handle the load, the alert recovers. For example, using the Synthetics private location dashboard example, notice the growth and decline of pending checks over the course of the incident and recovery. By using the NRQL condition, New Relic will notify you if and when the location needs more minion capacity.
Can I check the status of a specific minion directly?
You can also check how a minion is operating by contacting it directly. You can use a set of HTTP endpoints exposed by the minion to determine what the application is doing. In order to access these endpoints, bind ports 8080
and 8180
to ports on the host. For example, for Docker, use docker run -p 80:8080 -p 81:8180 ...
):
:8080/status/check
: Details about internal health-checks the minion performs; HTTP 200 means "healthy.":8080/status
: Details about a minion's status; the same data is then reported as aSyntheticsPrivateMinion
event.:8180/
: JVM application admin endpoints; an advanced view of a minion's internal state.
This approach is not as automated or flexible as the checksPending
example. However, if you have total network connectivity failure, this manual approach can help troubleshoot the situation.