You can configure the amount of memory Pixie uses. During the installation, use Helm to set the memory requests and limits, or to reduce the amount of memory Pixie uses for short-term data storage.
The primary focus of the open source Pixie project is to build a real-time debugging platform. Pixie isn't intended to be a long-term durable storage solution and is best used in conjunction with New Relic. The New Relic integration queries Pixie every few minutes and persists a subset of Pixie's telemetry data in New Relic.
When you install the New Relic Pixie integration, a
vizier-pem agent is deployed to each node in your cluster via a DaemonSet. The
vizier-pem agents use memory for two main purposes:
- Collecting telemetry data: tracing application traffic or CPU profiles, amongst other. Those values must be stored in memory somewhere, as they're processed.
- Short-term storage of telemetry data: to power troubleshooting via the Live debugging with Pixie tab; and as a temporary storage location for a subset of the telemetry data before it's stored in New Relic.
vizier-pem pods have a
2Gi memory limit, and a
2Gi memory request. They set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the data collection.
After installing Pixie, memory usage of the
vizier-pem pods increases as they begin storing telemetry data. Once you reach
vizier-pem's memory limit, old telemetry data is expired to make room for new data, and so memory utilization shouldn't increase any further.
For most clusters, we recommend using the default
2Gi memory configuration. However, for certain low traffic clusters, Pixie can support a minimum memory limit of
1Gi. To accommodate application pods, we recommend that no more than 25% of the nodes' total memory be allocated to Pixie. For example, if your nodes have a total memory of
4Gi, you'll want to configure Pixie to use a
1Gi memory limit.
If you want to specify a different memory limit than the default
2Gi for Pixie's
vizier-pem agents, you can add the following configuration parameter to your Helm chart during installation. For example, for a
1Gi memory limit, you'd use:
By default, the
vizier-pem's memory request is the same as the limit. If you want to specify a different memory request, you can add the following configuration parameter to your Helm chart during installation. For example, for a
1Gi memory request you'd use:
vizier-pem memory request must be less than or equal to the
vizier-pem memory limit.
vizier-pem pods set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the collection. For the default
2Gi memory limit, this means that a
vizier-pem pod keeps
1.2Gi memory for data storage.
If you want to specify a different amount for memory for short-term storage, you can add the following configuration parameter to your Helm chart during installation. For example, for
750MiB of short-term data storage you'd use:
For troubleshooting, see these docs: