Enable distributed tracing for your Go applications
Distributed tracing allows you to see the entire journey of your requests throughout a distributed system. For the Go agent, we offer two types of distributed tracing (for more details, see How span sampling works):
Standard (head-based sampling): Before any traces arrive, we determine a set percentage of traces to accept and analyze. This gives you a solid starting point to see how tracing can help you. It is turned on by default in Go agents 3.16.0 and higher.
Infinite Tracing (tail-based sampling): Our cloud-based service accepts all your traces and then sorts through them to find the most important. Infinite tracing analyzes all of your traces and gives you configuration options to sample the traces that matter most to you.
Whether you just want to try out standard distributed tracing (head-based sampling) or also want to set up Infinite Tracing (tail-based sampling), you need to start by setting up standard tracing.
Standard distributed tracing
This is the best approach to set up standard distributed tracing if you haven't installed any APM agents for your services yet.
Tip
When you install the New Relic Go agent, standard distributed tracing is turned on by default. If you prefer to turn it off, see our configuration guide.
Identify services
Figure out which services touch your request so you can instrument each of them to send trace data to New Relic.
Instrument each service with an APM agent
For each service involved in your transactions, you'll perform separate installations of the agent. If some of your services use other languages, simply repeat the installation steps for those languages.
Tip
The Go agent requires you to manually instrument your Go services, unlike the auto-instrumentation of the other New Relic agents. This means you need to add some lines to your code to use the Go agent. You'll learn about this when you start the installation below.
To start the installation routine, click the tile below. When you're finished installing each agent, return here to see tips for viewing your traces.
After you instrument each of your services with APM agents, generate some traffic in your application so we can capture some traces. Here are some ways to view your traces in the UI:
Here's one way you can see traces for a particular service:
In the left pane's Monitor section, click Distributed tracing.
For details, click on an individual trace. If Group similar traces is on in the top menu, click on a trace group, and then click on an individual trace.
If you don't see the traces you want, you can filter by the trace.id.
For details, click on an individual trace. If Group similar traces is on in the top menu, click on a trace group, and then click on an individual trace.
If you don't see the traces you want, you can filter by the trace.id.
You can bring your logs and tracing details together to make troubleshooting easier and faster. With logs in context, you can see log messages alongside traces in the New Relic UI.
Standard distributed tracing for APM agents captures up to 10% of your traces, but if you want us to analyze all your data and find the most relevant traces, you can set up Infinite Tracing.
Complete the setup for standard distributed tracing
The Infinite Tracing setup builds on standard distributed tracing. So, make sure you've completed the steps above, and then continue with the trace observer setup.
Set up the trace observer
The trace observer is a New Relic AWS-based service that collects and analyzes all your traces. Follow the instructions in Set up trace observer. When you're done, return here with your trace observer information and continue with the next step to configure the agent.
Configure the agent for Infinite Tracing
Infinite Tracing configuration settings include the standard distributed tracing plus information about the trace observer:
If you need help with proxy configuration, see Proxy support.
(Optional) Customize Infinite Tracing
After you add the agent configuration settings, you should start seeing data in the New Relic UI. After you spend some time analyzing your data, you may want to adjust some of the features of Infinite Tracing:
All installations of the Go agent and distributed tracing require some manual instrumentation using the settings listed in Go agent configuration settings. Still, you may need to do some additional configuration to optize your setup. Here are some guidelines for instrumenting transactions and HTTP requests.
If you are using Go's http.ServeMux and want to enable New Relic's distributed tracing, your Go application must be instrumented with New Relic's WrapHandle and WrapHandleFunc wrappers. These wrappers automatically start and end transactions with the request and response writer, which will automatically add the correct distributed tracing headers. For more on how header propagation works, see How distributed tracing works.
Here is an example of code before instrumentation:
http.HandleFunc("/users", usersHandler)
And here is an example of that same code after instrumentation:
In order to have your outbound HTTP requests eligible for distributed tracing, create an external segment.
The easiest way to create an external segment for your outbound HTTP request is to use the newrelic.NewRoundTripper method. Here is an example of making a request to http://api.example.com that includes the outgoing distributed tracing headers:
If you have a more complex request that uses the Go standard library's http.Request, use the newrelic.StartExternalSegment method to ensure your outbound request is eligible for distributed tracing:
// Distributed tracing headers are not added to the outgoing request.
// Use newrelic.NewRoundTripper or newrelicc.StartExternalSegment instead.
defer newrelic.ExternalSegment{
StartTime: txn.StartSegmentNow(),
URL: url,
}.End()
return http.Get(url)
}
The distributed trace payload contains information that allows New Relic to stitch together transactions occurring in multiple services into a complete transaction trace. If New Relic-monitored services are not sending trace context to each other, it will result in incomplete trace details.
For general instructions on how to use the API calls below to implement distributed tracing, first see Use distributed tracing APIs.
If you want to...
Use this
Create a payload to be sent to the called service.
