A DevOps transformation requires a cultural shift so that teams can build new skills and motivations for the type of cross-team work required in a true DevOps practice. The transformation can be difficult when the people involved do not see the benefits of change as a clear objective.
Service level objectives (SLOs) provide a powerful mechanism to codify the goals of a DevOps team in a way that can be measured and shared. They also provide clear boundaries on service expectations that help teams achieve greater velocity and freedom in experimenting with new approaches.
This tutorial defines SLOs for successful service delivery objectives and utilize New Relic instrumentation to surface the current performance metrics relative to those objectives. Measurable SLOs and visibility into your current progress against those SLOs ensure that you will be able to properly assess future optimization efforts.
An SLO is an agreed upon means of measuring the performance of your service. The SLO defines a target value of a specified quantitative measure, which is called a service level indicator (SLI); for example:
SLOs clarify a target value for SLIs; for example:
- Average response time should be less than 200 ms
- 95% of requests should be completed within 250 ms
- Availability of the service should be 99.99%
Logically group SLOs together to provide an overall boolean indicator of whether or not the service is meeting expectations. For example, a helpful SLO for alerting purposes could be:
95% of requests completed within 250 ms AND availability is 99.99%
Service level components
HTTP status codes
< 1% HTTP 500s over 30 days
For every additional .1% of HTTP 500s, 5% refund of total contract
Value stream mapping can be a useful exercise to work through before setting SLOs. Work with your teams to clarify key components of your service and the appropriate metrics. Use these inputs as starting points for this tutorial.
- Learn about SLOs, SLIs, and SLAs from the Google Cloud Platform blog.
- Learn how New Relic has applied SLOs and SLIs into its reliability practices form this SREcon18 Americas presentation (approximately 21 minutes).
Start defining SLOs for your application by first taking an inventory of the services that your application provides to both your internal and external customers.
- Draft a list of services. Make the scope of services you consider as comprehensive as possible.
- Engage your team members and other stakeholders to validate the list for completeness.
- Segment your application stack to understand the potential components that might require SLOs.
For example, most applications can be segmented as:
- Application (backend/microservices)
- Dependency services (such as the message queue)
- Underlying servers
This example lists components that would benefit from SLOs:
Building a definitive list of services that require an SLO can be challenging, because an application often consists of many endpoints with complex interdependencies.
Begin your SLO journey with pragmatism. Start by defining a broader, simpler set of SLOs that are driven by what your customers care about most and what your team can control. As your teams better align around SLOs, you can then begin to fine-tune and add more complexity.
Once you have an inventory of services, begin to gather the information you need to define the SLOs for those services. Interviews with customers that depend on your services are often valuable for understanding service expectations. For example, to define SLOs for internal teams, New Relic, ask questions such as:
- If possible, can you broadly categorize the types of requests we can expect from you and your service?
- To what extent do you or your service depend on timely responses to requests?
- Are there requests for which response time is not critical?
- How does your service handle unavailable dependencies or data?
- What is the maximum amount of unavailable data that your service can handle?
- At what threshold does your service fail if a request takes too long?
- What are acceptable rates of errors?
- What would a SLA look like between our product and yours?
Existing usage data can also be a helpful research input.
Using the research on customer expectations that you gathered, draft a focused set of SLOs. New Relic recommends setting SLOs against one or more of the following SLIs:
Also, consider instrumenting and tracking the following SLIs:
- Throughput (peak and trough)
- Database call count and duration
- DNS and SSL timing
- DOM processing and page rendering
- Mean-time-to-detection (MTTD)
For a more comprehensive list of potential areas to measure, see Measuring DevOps.
Recommendation: To determine if your application is performing to customer expectations:
- Consider combining multiple SLIs (for example, availability and response time) into one SLO.
- Aim to define a consistent set of conditions across all of the services in your list.
- Consult your team and stakeholders to validate that the SLOs you set are reasonable, consistently attainable (even if you are not currently meeting them), and aligned to customer expectations.
After you finish this step, you should have a set of well-defined SLOs and SLIs.
Now you're ready to deploy agents or monitors to establish a performance baseline for the SLIs you created. With proper instrumentation in place, you'll have visibility into the performance indicators that matter for your team and your customers. In addition, you'll also have a clear understanding of how to meet your SLOs.
- Identify the service components your team will optimize.
- Verify which application tiers meet New Relic monitoring requirements.
- To ensure you have robust baselines from which to work, determine the level of instrumentation that is possible (or allowed) within your organization.
It's best practice to instrument everything you can, but there may be situations where instrumentation isn't viable. In that case, you should determine what other data is available or can be created. For example, you can gather logs to query them for SLO data and set alerts for them.
one.newrelic.com > Logs: Use the New Relic log management UI to leverage your logs.
If the application has a web front end in these situations, use New Relic synthetic monitors. Our synthetic monitors offer non-agent monitoring while still providing the ability to establish a baseline.
To instrument the example applications and components in this tutorial, use these New Relic features:
After you deploy the agents and monitors, use service maps to review the default metrics that New Relic captures. For example, a typical service map show many of the common SLIs that application teams rely on, including response time, Apdex, throughput, and error rate metrics from APM. It also shows page load time, Ajax response, throughput, and error rate from browser monitoring.
rpm.newrelic.com > Service maps: To explore detailed granularity for any area, use the service map's nodes.
To close any remaining gaps in visibility for your SLIs, use custom instrumentation. New Relic provides several avenues for adding custom instrumentation, including:
- Making API calls to agents from inside your source code
- Packaging XML-based custom instrumentation modules with deployed applications
- Adding UI-based instrumentation without a code deploy
In addition, you can add custom attributes to each transaction event that match application performance factors to critical business information. Then you can track those attributes in Insights dashboards. For more information, see the custom instrumentation documentation for your application:
Once you implement the appropriate instrumentation, it's easy to visualize your service level indicators with New Relic dashboards, which provide a single location to query and view all the data that New Relic tools gather.
The metrics you capture will become your application's baseline. Share dashboards with your application team and stakeholders to provide visibility into what is happening with your application and to monitor future performance.