Establish objectives and baselines: define team SLOs

An optimized cloud native environment requires a cultural shift—one that enables teams to build new skills and motivations for the type of cross-team work required in distributed, modern cloud environments. The transformation can be difficult when the people involved do not clearly see the benefits of change.

Service level objectives (SLOs) provide a powerful mechanism to codify team goals in a way that stakeholders can measure and share. They also provide clear boundaries on service expectations that help teams achieve greater velocity and freedom in experimenting with new approaches.

This tutorial defines SLOs for successful service delivery objectives and provides guidance on how to utilize New Relic instrumentation to surface current performance metrics relative to those objectives. Measurable SLOs, and visibility into your current progress towards those SLOs, help to ensure that you can properly assess future optimization efforts.

Service level components

An SLO is an agreed upon means of measuring the performance of your service. The SLO defines a target value of a specified quantitative measure, which is called a service level indicator (SLI); for example:

  • Average response time

  • Response time percentile

  • Application availability

SLOs clarify a target value for SLIs; for example:

  • Average response time should be less than 200 ms

  • 95% of requests should be completed within 250 ms

  • Availability of the service should be 99.99%

Logically group SLOs together to provide an overall boolean indicator of whether or not the service is meeting expectations. For example, a helpful SLO for alerting purposes could be:

95% of requests completed within 250 ms AND availability is 99.99%

Service level components Example values

SLI (Indicator)

HTTP status codes

SLO (Objective)

< 1% HTTP 500s over 30 days

SLA (Agreement)

For every additional .1% of HTTP 500s, 5% refund of total contract

Resources

Value stream mapping can be a useful exercise to work through before setting SLOs. Work with your teams to clarify key components of your service and appropriate metrics. Use these inputs as starting points for this tutorial.

In addition:

1. Build an inventory of services requiring SLOs

Start defining SLOs for your application by first taking an inventory of the services that your application provides to both your internal and external customers.

  1. Draft a list of services. Make the scope of services you consider as comprehensive as possible.

  2. Engage your team members and other stakeholders to validate the list for completeness.

  3. Segment your application stack to understand the potential components that might require SLOs.

For example, most applications can be segmented as:

  • Application (backend/microservices)

  • Dependency services (such as the message queue)

  • Database

  • Website

  • Underlying servers

Example SLOs

This example lists components that would benefit from SLOs:

Customer type Component name Owner Language stack Operating system

External

Service 1

John D.

Java

RHEL 6

Internal

Service 2

Jane A.

.NET

Win2003 R2

Internal

ActiveMQ

John D.

Java

AIX

External

Website

Jane A.

Classic ASP

Win2000

Internal

MS SQL

Dave Z.

n/a

Win2003 R2

Building a definitive list of services that require SLOs can be challenging, because an application often consists of many endpoints with complex interdependencies.

Begin your SLO journey with pragmatism. Start by defining a broader, simpler set of SLOs that are driven by what your customers care about most and what your team can control. As your teams align around a defined set of SLOs, you can then begin to fine-tune and add more complexity.

2. Research customer expectations for SLOs

Once you have an inventory of services, begin to gather the information you need to define the SLOs for those services. Interviews with customers that depend on your services are often valuable for understanding service expectations. For example, to define SLOs for internal teams, ask questions like:

  • If possible, can you broadly categorize the types of requests we can expect from you and your service?

  • To what extent do you or your service depend on timely responses to requests?

  • Are there requests for which response time is not critical?

  • How does your service handle unavailable dependencies or data?

  • What is the maximum amount of unavailable data that your service can handle?

  • At what threshold does your service fail if a request takes too long?

  • What are acceptable rates of errors?

  • What would a SLA look like between our product and yours?

Existing usage data can also be a helpful research input.

3. Define SLOs

Using the research on customer expectations that you gathered, draft a focused set of SLOs. New Relic recommends setting SLOs against one or more of the following SLIs:

  • Application availability percentage

  • Average response time

  • Response percentile

  • Error rate

  • Apdex value

Also, consider instrumenting and tracking the following SLIs:

  • Throughput (peak and trough)

  • Database call count and duration

  • DNS and SSL timing

  • DOM processing and page rendering

  • Mean-time-to-detection (MTTD)

For a more comprehensive list of potential areas to measure, see our article on Measuring DevOps.

Recommendation: To determine if your application is performing to customer expectations:

  1. Consider combining multiple SLIs (for example, availability and response time) into one SLO.

  2. Aim to define a consistent set of conditions across all of the services in your list.

  3. Consult your team and stakeholders to validate that the SLOs you set are reasonable, consistently attainable (even if you are not currently meeting them), and aligned to customer expectations.

After you finish this step, you should have a set of well-defined SLOs and SLIs.

4. Determine what can be instrumented

Now you are ready to deploy agents or monitors to establish a performance baseline for the SLIs you created. With proper instrumentation in place, you will have visibility into the performance indicators that matter for your team and your customers. You will also have a clear understanding of how to meet your SLOs.

  1. Identify the service components your team will optimize.

  2. Verify which application tiers meet New Relic monitoring requirements.

  3. To ensure you have robust baselines from which to work, determine the level of instrumentation that is possible (or allowed) within your organization.

There may be situations where instrumentation of the current on-premise environment is not viable. For example, firewalls with certain settings may not permit New Relic agents to transmit data.

If the application has a web front end in these situations, use New Relic Synthetics. Synthetics offers non-agent monitoring while still providing the ability to establish a baseline.

To instrument the example applications and components in this tutorial, use these New Relic products:

Examples:

Customer type Component name Tier owner Language stack Server OS New Relic Product

External

Service 1

John D.

Java

RHEL 6

APM, Infrastructure, Synthetics

Internal

Service 2

Jane A.

.Net

Win2003 R2

APM, Infrastructure

Internal

ActiveMQ

John D.

Java

AIX

APM, Plugins

External

Website

Jane A.

Classic ASP

Win2000

Synthetics

Internal

MS SQL

Dave Z.

NA

Win2003 R2

Infrastructure, On-host integration

5. Review the default metrics

After you deploy the agents and monitors, use service maps to review the default metrics that New Relic captures. For example, a typical service map shows many common SLIs that application teams rely on, including response time, Apdex, throughput, and error rate metrics from APM. It also shows page load time, Ajax response, throughput, and error rate from Browser.

6. Set up custom instrumentation

Use custom instrumentation to close any remaining visibility gaps visibility for your SLIs. New Relic provides several options for adding custom instrumentation, including:

  • Making API calls to agents from inside your source code

  • Packaging XML-based custom instrumentation modules with deployed applications

  • Adding UI-based instrumentation without a code deploy

In addition, you can add custom attributes to each transaction event— matching application performance factors to critical business information, and then tracking those attributes in Insights dashboards.

For more information, see the custom instrumentation documentation for your application:

7. Create dashboards to track SLIs

Once you implement the appropriate instrumentation, it is easy to visualize your service level indicators with New Relic Insights dashboards. Insights provides a single location to query and view all the data that New Relic products gather. For example, Insights helps visualize the following data gathered from New Relic products:

Each event has an event type, a timestamp, and one or more key attributes. For more information about the data that New Relic receives from your app, see Data collection.

To query and view data from the SLIs you selected for baselining:

  1. Use the Insights Data explorer to create widgets for your dashboards.

  2. Create dashboards that include widgets for SLI baseline data.

  3. Use these widgets and dashboards to establish team dashboards that you can share and use to conduct operations reviews.

Establish objectives: Insights
insights.newrelic.com > (select a dashboard): Query your SLI baseline data, create dashboards with widgets to visualize the data, then share with your team and stakeholders.

The metrics you capture will become your application's baseline. Share your Insights dashboard with your application team and stakeholders to provide visibility into what is happening with your application and to monitor future performance.

For more help

Recommendations for learning more: