In addition to duration and span attribute trace filters, trace observers use a random trace filter to include a representative sample of the traces that are observed. By default, 1% of all traces seen are kept via the random trace filter. You can configure this value anywhere from 0% to 100%, using increments as small as 0.0001%. Changes you make to the percent kept value are instantly recognized by the trace observer.
Overview
There are various use cases for when you might want to change the percent kept value. For each of the use cases below, you can make the change via the Infinite Tracing settings Nerdlet or by following the instructions in our GraphQL tutorial: Update the random sampler on a trace observer.
Short time window
Let's say you're launching a new feature and would like to see every single trace during the first 15 minutes of launch. Turn up the percent kept to 100%, and you'll see everything. Turn it back down when you're done.
Show me everything
If you want to go exploring through all your data, all the time, turn the percent kept to 100%, and you'll see everything.
Show me all signal and no noise
You may want to increase the signal-to-noise ratio by viewing only the traces that our algorithms have deemed special. Turn the percent kept to 0%, and you'll only see duration outliers and those traces matching your span attribute filter rules.
Show me more (or less)
Our default of 1% is just a starting point. If you have low trace throughput without many outliers or errors, seeing just 1 in 100 traces might not feel like enough. On the other end of the scale, if you have a lot of data, rest easy knowing you can knock down the filter by four orders of magnitude.