You can configure the amount of memory Pixie uses. During the installation, use Helm to set the memory requests and limits, or to reduce the amount of memory Pixie uses for short-term data storage.
The primary focus of the open source Pixie project is to build a real-time debugging platform. Pixie isn't intended to be a long-term durable storage solution and is best used in conjunction with New Relic. The New Relic integration queries Pixie every few minutes and persists Pixie's telemetry data in New Relic.
When you install the New Relic Pixie integration, a
vizier-pem agent is deployed to each node in your cluster via a DaemonSet. The
vizier-pem agents use memory for two main purposes:
- Collecting telemetry data: tracing application traffic or CPU profiles, amongst other. Those values must be stored in memory somewhere, as they're processed.
- Short-term storage of telemetry data: before it's stored in New Relic.
vizier-pem pods have a
2Gi memory limit, and a
2Gi memory request. They set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the data collection.
After installing Pixie, memory usage of the
vizier-pem pods increases as they begin storing telemetry data. Once you reach
vizier-pem's memory limit, old telemetry data is expired to make room for new data, and so memory utilization shouldn't increase any further.
For most clusters, we recommend using the default
2Gi memory configuration. However, for certain low traffic clusters, Pixie can support a minimum memory limit of
1Gi. To accommodate application pods, we recommend that no more than 25% of the nodes' total memory be allocated to Pixie. For example, if your nodes have a total memory of
4Gi, you'll want to configure Pixie to use a
1Gi memory limit.
If you want to specify a different memory limit than the default
2Gi for Pixie's
vizier-pem agents, you can add the following configuration parameter to your Helm chart during installation. For example, for a
1Gi memory limit, you'd use:
By default, the
vizier-pem's memory request is the same as the limit. If you want to specify a different memory request, you can add the following configuration parameter to your Helm chart during installation. For example, for a
1Gi memory request you'd use:
vizier-pem memory request must be less than or equal to the
vizier-pem memory limit.
vizier-pem pods set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the collection. For the default
2Gi memory limit, this means that a
vizier-pem pod keeps
1.2Gi memory for data storage.
If you want to specify a different amount for memory for short-term storage, you can add the following configuration parameter to your Helm chart during installation. For example, for
750MiB of short-term data storage you'd use:
vizier-pem pods fail to schedule, this is usually because there are nodes that don't have the requested amount of memory.
Try reducing the memory request down as far as
1Gi. For particularly large or high traffic clusters,
1Gi may not be enough and some
vizier-pem OOMKills might occur. For these clusters, you can set a higher limit than request, such as
2Gi for the limit. We recommend
1Gi limit deployments for smaller clusters with less traffic.
Part of the benefit of deploying on Kubernetes is that Kubernetes will OOMKill pods that are using too many resources to protect the other pods in the cluster.
vizier-pem pods exceed the memory limit, they are OOMKilled. Kubernetes does this to protect the other applications, so it won't cause problems on other pods. For
vizier-pem pods that are regularly being OOMKilled, try deploying Pixie with more memory.