Reduce data ingest
Our charts support setting an option to reduce the amount of data ingested at the cost of granularity. To enable it, set global.lowDataMode
to true
in the nri-bundle
chart.
lowDataMode
affects four specific components of the nri-bundle
chart outlined below.
New Relic Infrastructure
If lowDataMode
is enabled, the default scrape interval changes from 15s
to 30s
.
You can also specify a custom value for it using config.interval
, which will take preference over lowDataMode
.
Prometheus Agent Integration
If lowDataMode
is enabled, the metrics that are prefixed with the following are excluded by default as they're already collected and used by the New Relic Kubernetes Integration.
- kube_- container_- machine_- cadvisor_
New Relic Logging
If lowDataMode
is enabled, Labels and Annotations are not included in the logs forwarded to New Relic. This reduces the overall data ingest into New Relic.
The following fields are retained: cluster_name
, container_name
, namespace_name
, pod_name
, stream
, message
and log
.
Low Data Mode Log Example
The following log record contains all the attributes typically captured by the New Relic Logs Kubernetes plugin:
[ { "cluster_name": "api-test", "kubernetes": { "annotations": { "kubernetes.io/psp": "eks.privileged" }, "container_hash": "fryckbos/test@sha256:5b098eaf3c7d5b3585eb10cebee63665b6208bea31ef31a3f0856c5ffdda644b", "container_image": "fryckbos/test:latest", "container_name": "newrelic-logging", "docker_id": "134e1daf63761baa15e035b08b7aea04518a0f0e50af4215131a50c6a379a072", "host": "ip-192-168-17-123.ec2.internal", "labels": { "app": "newrelic-logging", "app.kubernetes.io/name": "newrelic-logging", "controller-revision-hash": "84db95db86", "pod-template-generation": "1", "release": "nri-bundle" }, "namespace_name": "nrlogs", "pod_id": "54556e3e-719c-46b5-af69-020b75d69bf1", "pod_name": "nri-bundle-newrelic-logging-jxnbj" }, "message": "[2021/09/14 12:30:49] [ info] [engine] started (pid=1)\n", "plugin": { "source": "kubernetes", "type": "fluent-bit", "version": "1.8.1" }, "stream": "stderr", "time": "2021-09-14T12:30:49.138824971Z", "timestamp": 1631622649138 }]
This is how the previous log record would look like after enabling lowDataMode
:
[ { "cluster_name": "api-test", "container_name": "newrelic-logging", "namespace_name": "nrlogs", "pod_name": "nri-bundle-newrelic-logging-jxnbj", "message": "[2021/09/14 12:30:49] [ info] [engine] started (pid=1)\n", "stream": "stderr", "timestamp": 1631622649138 }]
New Relic Pixie Integration
If lowDataMode
is enabled, the newrelic-pixie
integration performs heavier sampling on Pixie spans and reduces the collection interval from 10 seconds to 15 seconds.
lowDataMode
settings:
HTTP_SPAN_LIMIT: 750DB_SPAN_LIMIT: 250COLLECT_INTERVAL_SEC: 15
The default settings for these parameters and others can be found in the newrelic-pixie-integration Github repo.
To take full advantage of New Relic's kubernetes integration, you'll need a free New Relic account. If you haven't already, create yours below to start monitoring your data today.