You can configure the amount of memory Pixie uses. During the installation, use Helm to set the memory requests and limits, or to reduce the amount of memory Pixie uses for short-term data storage.
How does Pixie use memory?
The primary focus of the open source Pixie project is to build a real-time debugging platform. Pixie isn't intended to be a long-term durable storage solution and is best used in conjunction with New Relic. The New Relic integration queries Pixie every few minutes and persists a subset of Pixie's telemetry data in New Relic.
When you install the New Relic Pixie integration, a vizier-pem
agent is deployed to each node in your cluster via a DaemonSet. The vizier-pem
agents use memory for two main purposes:
- Collecting telemetry data: tracing application traffic or CPU profiles, amongst others. Those values must be stored in memory somewhere, as they're processed.
- Short-term storage of telemetry data: to power troubleshooting via the Live debugging with Pixie tab and as a temporary storage location for a subset of the telemetry data before it's stored in New Relic.
By default, vizier-pem
pods have a 2Gi
memory limit, and a 2Gi
memory request. They set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the data collection.
Why does Pixie's memory use increase after installation?
After installing Pixie, memory usage of the vizier-pem
pods increases as they begin storing telemetry data. Once you reach vizier-pem
's memory limit, old telemetry data is expired to make room for new data, and so memory utilization shouldn't increase any further.
Configuring Pixie's memory usage
For most clusters, we recommend using the default 2Gi
memory configuration. However, for certain low traffic clusters, Pixie can support a minimum memory limit of 1Gi
. To accommodate application pods, we recommend that no more than 25% of the nodes' total memory be allocated to Pixie. For example, if your nodes have a total memory of 4Gi
, you'll want to configure Pixie to use a 1Gi
memory limit.
Deploy Pixie with a particular memory limit
If you want to specify a different memory limit than the default 2Gi
for Pixie's vizier-pem
agents, you can add the following configuration parameter to your Helm chart during installation. For example, for a 1Gi
memory limit, you'd use:
--set pixie-chart.pemMemoryLimit=1Gi
Deploy Pixie with a particular memory request
By default, the vizier-pem
's memory request is the same as the limit. If you want to specify a different memory request, you can add the following configuration parameter to your Helm chart during installation. For example, for a 1Gi
memory request you'd use:
--set pixie-chart.pemMemoryRequest=1Gi
Important
The vizier-pem
memory request must be less than or equal to the vizier-pem
memory limit.
Deploy Pixie with a particular short-term data storage
By default, vizier-pem
pods set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the collection. For the default 2Gi
memory limit, this means that a vizier-pem
pod keeps 1.2Gi
memory for data storage.
If you want to specify a different amount for memory for short-term storage, you can add the following configuration parameter to your Helm chart during installation. For example, for 750MiB
of short-term data storage you'd use:
--set pixie-chart.dataCollectorParams.customPEMFlags.PL_TABLE_STORE_DATA_LIMIT_MB=750
Troubleshooting
For troubleshooting, see these docs: