Sampler configuration examples

While the default sampling configuration works for all services, it may not be optimal. APM agents provide sampling configuration options so you can customize sampling to fit your service's needs. This page walks through some real-world examples of different types of services and how you can modify the sampling configuration to fit those needs.

Configuration examples on this page use YAML format (Java agent). For the equivalent syntax in your language and the complete parameter reference, see your agent's documentation:

• Java agent distributed tracing configuration
• Node.js agent distributed tracing configuration
• Python agent distributed tracing configuration

Prerequisites

This guide assumes familiarity with sampler contexts and types. Visit Configure Sampling page for a complete overview of sampler concepts.

Example 1: Poor distribution of endpoints captured due to rarely-called endpoints

The default sampler, the Adaptive sampler, samples about 10 traces per minute for most APM language agents. Since the Adaptive sampler chooses traces randomly, it is more likely to sample frequently-called endpoints than infrequently-called endpoints.

Solution: Increase adaptive sampling target

Increasing the adaptive_sampling_target samples more traces each minute, increasing the likelihood of capturing traces for infrequent endpoints.

distributed_tracing:
  sampler:
    adaptive_sampling_target: 20  # Increase from default 10 to 20

Example 2: Poor distribution of endpoints captured due to upstream sampling

With the default Adaptive sampler, the current service also samples any incoming trace that an upstream service already sampled. Sometimes, this leads to too many traces from endpoints with an upstream parent, and too few traces from other endpoints.

Solution: Reduce the sampling rate for traces sampled by an upstream service

Trace Id Ratio Based Sampler can be used to sample upstream-sampled traces only 10% of the time. This reduces the likelihood of filling up the span reservoir with traces that were sampled by an upstream service.

distributed_tracing:
  sampler:
    remote_parent_sampled:
      trace_id_ratio_based:
        ratio: 0.1

Example 3: Change trace distribution for all contexts

Example 2 - too many upstream-sampled traces - is a specific case of a more general problem: you want more control over the distribution of traces you see across all trace contexts.

Solution: Layer multiple ratio samplers

We can build upon the previous example by layering multiple trace ratio samplers. The sampling rate of each trace context is independent, so each ratio can be tuned to achieve the desired distribution of traces.

In this example, every trace context uses ratio-based sampling. The ratios are set to sample 25% of root traces (instead of a fixed number), 50% of traces sampled by an upstream service (instead of all of them), and 5% of traces not sampled by an upstream service (instead of none of them).

distributed_tracing:
  sampler:
    root:
      trace_id_ratio_based:
        ratio: 0.25
    remote_parent_sampled:
      trace_id_ratio_based:
        ratio: 0.5
    remote_parent_not_sampled:
      trace_id_ratio_based:
        ratio: 0.05

What's next

• See the complete sampler configuration reference for your agent: Java, Node.js, Python
• Understand sampling configuration for sampler types and adaptive sampling targets