Varnish Cache monitoring integration

The Varnish Cache on-host integration collects and sends inventory and metrics from your Varnish Cache environment to New Relic so you can monitor its health. We collect metrics at the instance, lock, memory pool, storage, and backend levels.

Read on to install the integration, and to see what data we collect.

Compatibility and requirements

Our integration is compatible with Varnish Cache 1.0 or higher.

Before installing the integration, make sure that you meet the following requirements:

Install the infrastructure agent.
Linux distribution or Windows version compatible with our infrastructure agent.

Quick start

Instrument your Varnish Cache environment quickly and send your telemetry data with guided install. Our guided install creates a customized CLI command for your environment that downloads and installs the New Relic CLI and the infrastructure agent.

Ready to get started? Click one of these button to try it out.

Guided install

EU Guided install

Our guided install uses the infrastructure agent to set up the Varnish Cache integration. Not only that, it discovers other applications and log sources running in your environment and then recommends which ones you should instrument.

The guided install works with most setups. But if it doesn't suit your needs, you can find other methods below to get started monitoring your Varnish Cache environment.

Install and activate

To install the Varnish Cache integration:

Install the infrastructure agent, and replace the INTEGRATION_FILE_NAME variable with nri-varnish.
Change directory to the integrations folder:
```
cd /etc/newrelic-infra/integrations.d
```

Copy of the sample configuration file:

sudo cp varnish-config.yml.sample varnish-config.yml

Edit the varnish-config.yml file as described in the configuration settings.

Download the nri-varnish .MSI installer image from:
https://download.newrelic.com/infrastructure_agent/windows/integrations/nri-varnish/nri-varnish-amd64.msi
To install from the Windows command prompt, run:
bash
```
$msiexec.exe /qn /i PATH\TO\nri-varnish-amd64.msi
```
In the Integrations directory, C:\Program Files\New Relic\newrelic-infra\integrations.d\, create a copy of the sample configuration file by running:
bash
```
$copy varnish-config.yml.sample varnish-config.yml
```
Edit the varnish-config.yml file as described in the configuration settings.

Additional notes:

Advanced: It's also possible to install the integration from a tarball file. This gives you full control over the installation and configuration process.
On-host integrations do not automatically update. For best results, regularly update the integration package and the infrastructure agent.

Did this doc help with your installation?

Configure the integration

An integration's YAML-format configuration is where you can place required login credentials and configure how data is collected. Which options you change depend on your setup and preference.

The configuration file has common settings applicable to all integrations like interval, timeout, inventory_source. To read all about these common settings, refer to our Configuration Format document.

Important

If you are still using our legacy configuration/definition files please refer to this document for help.

Specific settings related to Varnish are defined using the env section of the configuration file. These settings control the connection to your Varnish instance, as well as other security settings and features. The list of valid settings is described in the following section.

Varnish Cache instance settings

The Varnish Cache integration collects both metrics(M) and inventory(I) information. Check the Applies To column below to find which settings can be used for each specific collection:

Setting	Description	Default	Applies To
INSTANCE_NAME	User defined name to identify data from this instance in New Relic. Required.	N/A	M/I
PARAMS_CONFIG_FILE	The location of the `varnish.params` config file. If this argument is omitted, the following locations will be checked: `/etc/default/varnish/varnish.params` `/etc/sysconfig/varnish/varnish.params` Note: The location and name of the Varnish configuration file may vary. For details, see Different locations of the Varnish configuration file. For Varnish 6 and above, this parameter is not required and the integration should be set up for metrics collection only. See the example for Varnish 6.	N/A	I
VARNISH_NAME	Name used when executing the `varnishd` daemon with a custom `-n` flag. Optional.	N/A	M
METRICS	Set to `true` to enable metrics-only collection.	`false`
INVENTORY	Set to `true` to enable inventory-only collection.	`false`

The varnish-config.yml commands accept the following arguments:

The values for these settings can be defined in several ways:

Adding the value directly in the config file. This is the most common way.
Replacing the values from environment variables using the {{}} notation. This requires infrastructure agent v1.14.0+. Read more here.
Using secrets management. Use this to protect sensitive information, such as passwords that would be exposed in plain text on the configuration file. For more information, see Secrets management.

Labels/Custom attributes

Environment variables can be used to control config settings, such as your , and are then passed through to the infrastructure agent. For instructions on how to use this feature, see Configure the infrastructure agent. You can further decorate your metrics using labels. Labels allow you to add key/value pairs attributes to your metrics which you can then use to query, filter or group your metrics on.
Our default sample config file includes examples of labels but, as they are not mandatory, you can remove, modify or add new ones of your choice.

labels:
   env: production
   role: varnish

Example configuration

Example varnish-config.yml file configuration:

This is the very basic configuration to collect metrics and inventory :

integrations:
  - name: nri-varnish
    env:
      INSTANCE_NAME: new_relic
      PARAMS_CONFIG_FILE: /etc/default/varnish/varnish.params
    interval: 15s
    labels:
      env: production
      role: varnish
    inventory_source: config/varnish

This is a basic configuration for Varnish 6 or above. Only metrics will be collected because, starting in Varnish 6, the params file was deprecated.

integrations:
  - name: nri-varnish
    env:
      INSTANCE_NAME: new_relic
      METRICS: true
    interval: 15s
    labels:
      env: production
      role: varnish
    inventory_source: config/varnish

For more about the general structure of on-host integration configuration, see Configuration.

Find and use data

To find your integration data in New Relic, go to one.newrelic.com > All capabilities > Infrastructure > Third-party services and select one of the Varnish Cache integration links.

In New Relic, Varnish Cache data is attached to the following event type:

VarnishSample
VarnishLockSample
VarnishStorageSample
VarnishMempoolSample
VarnishBackendSample

For more on how to find and use your data, see Understand integration data.

Metric data

The Varnish Cache integration collects the following metric data attributes. Each metric name is prefixed with a category indicator and a period, such as bans. or main..

Tip

A number of metrics are calculated as rates (per second) instead of totals as the metric names might suggest. For more details on which metrics are calculated as rates, refer to the spec.csv file.

Varnish sample metrics

These attributes can be found by querying the VarnishSample event types.

Metric	Description
`backend.connectionBusy`	Number of times the maximum connection has been reached.
`backend.connectionFails`	Number of failed connections to the backed.
`backend.connectionRecycles`	Number of backend connections that have been recycled.
`backend.connectionRetries`	Number of backend connections that have been retried.
`backend.connectionReuses`	Number of backend connections reuses.
`backend.connectionSuccess`	Number of successful backend connections,
`backend.connectionUnHealthy`	Number of backend connections that were not attempted due to ‘unhealthy’ backend status.
`backend.fetches`	Total number of backend fetches initiated.
`backend.requests`	Total number of backend connection requests made.
`bans.added`	Counter of bans added to ban list.
`bans.completed`	Number of bans marked ‘completed'.
`bans.cutoffLurkerKilled`	Number of objects killed by bans for cutoff (lurker).
`bans.deleted`	Counter of bans deleted from ban list.
`bans.dups`	Count of bans replaced by later identical bans.
`bans.fragmentationInBytes`	Extra bytes in persisted ban lists due to fragmentation.
`bans.lookupKilled`	Number of objects killed by bans during object lookup.
`bans.lookupTestsTested`	Count of how many tests and objects have been tested against each other during lookup.
`bans.lurkerCon`	Number of times the ban-lurker had to wait for lookups.
`bans.lurkerKilled`	Number of objects killed by the ban-lurker.
`bans.lurkerTested`	Count of how many bans and objects have been tested against each other by the ban-lurker.
`bans.lurkerTestsTested`	Count of how many tests and objects have been tested against each other during by the ban-lurker.
`bans.obj`	Number of bans using `obj.*` variables. These bans can possibly be washed by the ban-lurker.
`bans.persistedInBytes`	Bytes used by the persisted ban lists.
`bans.req`	Number of bans which use `req.*` variables. These bans can not be washed by the ban-lurker.
`bans.tested`	Count of how many bans and objects have been tested against each other during hash lookup.
`cache.graceHits`	Count of cache hits with grace. A cache hit with grace is a cache hit where the object is expired. These hits also included in the `cache_hit` counter.
`cache.hits`	Number of times an object has been delivered to a client without fetching it from a backend server.
`cache.misses`	Number of times the object was fetched from the backend before delivering it to the client.
`cache.missHits`	Number of times a hit object was returned for a miss response.
`cache.passHits`	Number of times a hit object was returned for a pass response.
`esi.errors`	Edge Side Includes (ESI) parsing errors (unlock).
`esi.warnings`	Edge Side Includes (ESI) parse warnings (unlock).
`fetch.bad`	The `beresp.body` length/fetch could not be determined.
`fetch.chuncked`	The `beresp.body` chunked.
`fetch.contentLength`	The `beresp.body` with content-length.
`fetch.eof`	The `beresp.body` with EOF.
`fetch.failed`	The `beresp` failed.
`fetch.head`	The `beresp` with no body because the request is HEAD.
`fetch.noBody`	The `beresp` with no body.
`fetch.noBody1xx`	The `beresp` with no body because of 1XX response.
`fetch.noBody204`	The `beresp` with no body because of 204 response.
`fetch.noBody304`	The `beresp` with no body because of 304 response.
`fetch.noThreadFail`	The `beresp` fetch failed, no thread available.
`hcb.inserts`	Number of critical bit tree-based hash (HCB) inserts.
`hcb.lock`	Number of HCB lookups with lock.
`hcb.noLock`	Number of HCB lookups without lock.
`lru.limited`	Number of times more storage space was needed, but limit was reached.
`lru.moved`	Number of move operations done on the LRU list.
`lru.nuked`	Number of least recently used (LRU) objects forcefully evicted from storage to make room for a new object.
`main.backends`	Number of backends.
`main.bans`	Count of bans.
`main.busyKilled`	Number of requests killed after sleep on busy objhdr.
`main.busySleep`	Number of requests sent to sleep on busy objhdr.
`main.busyWakeup`	Number of requests woken after sleep on busy objhdr.
`main.expired`	Number of expired objects.
`main.expiredMailed`	Number of objects mailed to expiry thread.
`main.expiredReceived`	Number of objects received by expiry thread.
`main.gunzip`	Number of gunzip operations.
`main.gunzipTest`	Number of test gunzip operations.
`main.gzip`	Number of gzip operations.
`main.objectcores`	Number of objectcore structs made.
`main.objectheads`	Number of objected structs made.
`main.objects`	Number of object structs made.
`main.passedRequests`	Total pass-ed requests seen.
`main.pipeSessions`	Total pipe sessions seen.
`main.pools`	Number of thread pools.
`main.purgeObjects`	Number of purged objects.
`main.purgeOperations`	Number of purge operations executed.
`main.reqDropped`	Number of requests dropped.
`main.sessions`	Total number of sessions seen.
`main.sessQueueLength`	Length of session queue waiting for threads.
`main.summs`	Number of times per-thread statistics were summed into the global counters.
`main.syntheticResponses`	Total synthethic responses made.
`main.threads`	Total number of threads.
`main.threadsCreated`	Total number of threads created in all pools.
`main.threadsDestroyed`	Total number of threads destroyed in all pools.
`main.threadsFailed`	Number of times creating a thread failed.
`main.threadsLimited`	Number of times more threads were needed, but limit was reached in a thread pool.
`main.unresurrectedObjects`	Number of unresurrected objects.
`main.uptimeInMilliseconds`	The child process uptime, in milliseconds.
`main.vclAvailable`	Number of Varnish Configuration Languages (VCL) available.
`main.vclDiscarded`	Number of discarded VCLs.
`main.vclFails`	Number of VCL failures.
`main.vclLoaded`	Number of loaded VCLs in total.
`main.vmodsLoaded`	Number of loaded Varnish modules (VMOD).
`mgt.childDied`	Number of times the child process has died due to signals.
`mgt.childDump`	Number of times the child process has produced core dumps.
`mgt.childExit`	Number of times the child process has been cleanly stopped.
`mgt.childPanic`	Number of times the management process has caught a child panic.
`mgt.childStart`	Number of times the child process has been started.
`mgt.childStop`	Number of times the child process has been cleanly stopped.
`mgt.uptimeInMilliseconds`	The management process uptime, in milliseconds.
`net.400Errors`	Number of client requests received, subject to 400 errors.
`net.417Errors`	Number of client requests received, subject to 417 errors
`net.httpOverflow`	Number of HTTP header overflows.
`net.pipe.inInBytes`	Total number of bytes forwarded from clients in pipe sessions.
`net.pipe.outInBytes`	Total number of bytes forwarded to clients in pipe sessions.
`net.pipereq.headerInBytes`	Total request bytes received for piped sessions.
`net.request.bodyInBytes`	Total request body transmitted, in bytes.
`net.request.headerInBytes`	Total request headers transmitted, in bytes.
`net.requests`	Number of good client requests received.
`net.response.bodyInBytes`	Total response body transmitted, in bytes.
`net.response.headerInBytes`	Total response headers transmitted, in bytes.
`sess.backendClose`	Number of session closes with the error `RESP_CLOSE`, (Backend/VCL requested close).
`sess.badClose`	Number of session closes with the error `Error RX_BAD`, (Received bad req/resp).
`sess.bodyFailClose`	Number of session closes with the error `Error RX_BODY`, (Failure receiving req.body).
`sess.clientClose`	Number of session closes with the error `REM_CLOSE`, (Client closed).
`sess.clientReqClose`	Number of session closes with the error `REQ_CLOSE`, (Client requested close).
`sess.closed`	Total number of sessions closed.
`sess.closedError`	Total number of sessions closed with errors.
`sess.dropped`	Number of sessions dropped for thread.
`sess.eofTxnClose`	Number of session closes with the error `TX_EOF`, (EOF transmission).
`sess.errorTxnClose`	Number of session closes with the error `TX_ERROR`, (Error transaction).
`sess.herd`	Number of times the `timeout_linger` triggered.
`sess.junkClose`	Number of session closes with the error `RX_JUNK`, (Received junk data).
`sess.overflowClose`	Number of session closes with the error `RX_OVERFLOW`, (Received buffer overflow).
`sess.overloadClose`	Number of session closes with the error `OVERLOAD`, (Out of some resource).
`sess.pipeOverflowClose`	Number of session closes with the error `PIPE_OVERFLOW`, (Session pipe overflow).
`sess.pipeTxnClose`	Number of session closes with the error `TX_PIPE`, (Piped transaction).
`sess.queued`	Number of sessions queued for thread.
`sess.readAhead`	Session Read Ahead.
`sess.requestHTTP10Close`	Number of session closes with the error `REQ_HTTP10`, (Proto < HTTP/1.1).
`sess.requestHTTP20Close`	Number of session closes with the error `REQ_HTTP20`, (HTTP2 not accepted).
`sess.shortRangeClose`	Number of session closes with the error `RANGE_SHORT`, (Insufficient data for range).
`sess.timeoutClose`	Number of session closes with the error `RX_TIMEOUT`, (Receive timeout).
`sess.vclFailClose`	Number of session closes with the error `VCL_FAILURE`, (VCL failure).
`session.connections`	Count of sessions successfully accepted.
`session.drops`	Count of sessions silently dropped due to lack of worker thread.
`session.fail`	Count of failures to accept TCP connection.
`shm.contentions`	Number of shared memory (SHM) MTX contentions.
`shm.cycles`	Number of SHM cycles through buffer.
`shm.flushes`	Number of SHM flushes due to overflow.
`shm.records`	Number of SHM records.
`shm.writes`	Number of SHM writes.
`workspace.backendOverflow`	Number of times we ran out of space in `workspace_backend`.
`workspace.clientOverflow`	Number of times we ran out of space in `workspace_client`.
`workspace.deliveryFail`	Delivery failed due to insufficient workspace.
`workspace.sessionOverflow`	Number of times we ran out of space in `workspace_session`.
`workspace.threadOverflow`	Number of times we ran out of space in `workspace_thread`.

Varnish lock sample metrics

These attributes can be found by querying the VarnishLockSample event type.

Metric	Description
`lock.created`	Count of created locks.
`lock.destroyed`	Count of destroyed locks.
`lock.locks`	Count of lock operations.

Varnish storage sample metrics

These attributes can be found by querying the VarnishStorageSample event type.

Metric	Description
`storage.allocFails`	Number of times the storage has failed to provide a storage segment.
`storage.allocInBytes`	Number of total bytes allocated by this storage.
`storage.allocOustanding`	Number of storage allocations outstanding.
`storage.allocReqs`	Number of times the storage has been asked to provide a storage segment.
`storage.availableInBytes`	Number of bytes left in the storage.
`storage.freeInBytes`	Number of total bytes returned to this storage.
`storage.outstandingInBytes`	Number of bytes allocated from the storage.

Varnish mempool sample metrics

These attributes can be found by querying the VarnishMempoolSample event type.

Metric	Description
`mempool.allocatedSizeInBytes`	Allocated size of memory pool, in bytes.
`mempool.allocs`	Memory pool allocations.
`mempool.frees`	Number of memory pools free.
`mempool.live`	Number of memory pools in use.
`mempool.pool`	Count in memory pool.
`mempool.ranDry`	Pool ran dry.
`mempool.recycles`	Recycled from pool.
`mempool.requestSizeInBytes`	Request size of memory pool, in bytes.
`mempool.surplus`	Too many for pool.
`mempool.timeouts`	Timed out from pool.
`mempool.tooSmall`	Too small to recycle.

Varnish backend sample metrics

These attributes can be found by querying the VarnishBackendSample event type.

Metric	Description
`backend.busyFetches`	Fetches not attempted due to backend being busy.
`backend.connections`	Number of concurrent connections to the backend.
`backend.connectionsFailed`	Number of backend connections failed.
`backend.connectionsNotAttempted`	Number of backend connection opens not attempted.
`backend.happy`	Happy health probes.
`backend.unhealtyFetches`	Fetches not attempted due to backend being unhealthy
`net.backend.pipeHeaderInBytes`	Total request bytes sent for piped sessions.
`net.backend.pipeInInBytes`	Total number of bytes forwarded from backend in pipe sessions.
`net.backend.pipeOutInBytes`	Total number of bytes forwarded to backend in pipe sessions.
`net.backend.requestBodyInBytes`	Total backend request body bytes sent.
`net.backend.requestHeaderInBytes`	Total backend request header bytes sent.
`net.backend.requests`	Number of backend requests sent,
`net.backend.responseBodyInBytes`	Total backend response body bytes received.
`net.backend.responseHeaderInBytes`	Total backend response header bytes received.

Inventory data

The Varnish Cache integration captures the configuration parameters. It parses the varnish.params configuration file for all parameters that are active.

The data is available on the Inventory page, under the config/varnish source. For more about inventory data, see Understand integration data.

Check the source code

This integration is open source software. That means you can browse its source code and send improvements, or create your own fork and build it.

Compatibility and requirements

Quick start

Install and activate

Linux installation

Windows installation

Did this doc help with your installation?

Configure the integration

Important

Varnish Cache instance settings

Labels/Custom attributes

Example configuration

Example configuration

Configuration for Varnish 6+

Find and use data

Metric data

Tip

Varnish sample metrics

Varnish lock sample metrics

Varnish storage sample metrics

Varnish mempool sample metrics

Varnish backend sample metrics

Inventory data

Check the source code

Varnish Cache monitoring integration

Compatibility and requirements .css-21sua1{background:none;border:none;width:0;padding:0;}

Quick start

Install and activate

Windows installation

Did this doc help with your installation?

Configure the integration

Important

Varnish Cache instance settings

Labels/Custom attributes

Example configuration

Example configuration

Configuration for Varnish 6+

Find and use data

Metric data

Tip

Varnish sample metrics

Varnish lock sample metrics

Varnish storage sample metrics

Varnish mempool sample metrics

Varnish backend sample metrics

Inventory data

Check the source code

Compatibility and requirements