Iterate and measure impact: track metrics before and after deployments

A properly instrumented system gives teams full visibility into the impact of the changes they make. Capturing tangible, measurable metrics from before and after each change allows teams to optimize changes in isolation, and reduce the impact to other ongoing changes.

Prerequisite

Before starting this tutorial, be sure to complete the Establish objectives and baselines tutorial and set the appropriate KPI targets for your applications.

1. Integrate measurements into your development process

With appropriate measurements incorporated into all phases of your development cycle, you can surface errors and performance issues before your customers uncover them. As your application teams plan their work, use your KPI dashboards in daily stand-ups and other planning meetings to analyze necessary debugging work, assess whether recent deployments were successful, and to prioritize other work efforts. When development and operations teams use dashboards during planning and discussions, they ensure that they’re consistently incorporating feedback about customer experiences and reliability risks into their development efforts.

For proper testing, ensure that instrumentation in your pre-production environments is in parity with your production environments. Use dashboards to compare the environments and verify that the code and infrastructure changes you make are correct across them, and eliminate any anomalies before pushing code to production.

Beyond using instrumentation to measure software performance, also use it to analyze team efficiency. For example, send Alerts data to Insights, and use NRQL to calculate mean time to repair (MTTR) by subtracting the difference in event timestamps as the current_state of each event changes from OPEN to ACKNOWLEDGED to CLOSED.

Or push events into Insights from a source code management (SCM) system like GitHub, and calculate the amount of time it takes a code change to go live by comparing the timestamp of a commit event to that of a deploy event. Plotted over time, this could become a KPI in an organization’s DevOps transformation.

2. Add automated deployment markers

It’s important to track deployments and how the impact of the code and infrastructure changes you make affect customer experience. New Relic APM’s deployment markers feature allows you to record deployments for each application. A deployment marker is an event indicating that a deployment happened, and it's paired with metadata available from your SCM system (such metadata typically includes the user, revision, change-log, etc.). APM displays a vertical line, or “marker”, on charts and graphs at the deployment event’s timestamp. When you hover over the line, APM displays the associated metadata for that deployment.

APM_web-transaction-times_Catalyst.png
rpm.newrelic.com/apm > (select an app) > Monitoring > Overview

APM also provides a chronological list of deployments of your application, and additional metrics, such as error rate and Apdex from the time of the deployment, are available as well.

APM_Deployments_Catalyst.png
rpm.newrelic.com/apm > (select an app) > Events > Deployments

Tracking deployments is an invaluable way to determine the root cause of immediate, long-term, or gradual degradations in your application.

New Relic recommends that you make POST requests to the New Relic REST API as the final step of a successful CI/CD deployment as described in the API documentation. The following tools have integrations or plugins available to help automate this:

3. Test your pipeline with Infrastructure

An important part of a successful DevOps transformation is a cultural shift toward smaller, more frequent changes to your code and infrastructure. After you complete the first two steps of this tutorial, begin to test and gather appropriate performance insights about your deployment pipeline to more clearly understand the impact of the changes you make.

Code changes should be as small as possible in terms of the number of lines of code and source files you change. Changes should also involve as few team members as possible. This makes it much easier to identify issue owners and determine root causes if errors occur.

Similarly, infrastructure changes should also have as small a footprint as possible to minimize the number of applications affected by each change. New Relic Infrastructure helps you see when an infrastructure change has caused a spike in issues, as shown in the following example:

For cloud infrastructure changes or larger code changes, consider using a blue/green deployment strategy. APM supports multiple app names which fits this model quite nicely.

To implement this strategy:

  1. Deploy a version of your application to existing infrastructure using a rollup app name that signifies it as the “blue” version (or control version).

  2. Deploy a new version of your application to a subset of the infrastructure (or an alternate set of infrastructure) with a rollup name that signifies this as the “green” version (or experimental version).

  3. Ensure that traffic is routed appropriately between both versions.

  4. Compare established KPI dashboards of both versions against one another.

  5. Optimize the new version accordingly to meet your KPI targets.

  6. When you’re satisfied with the application’s performance, deploy the new version across your infrastructure using the original app name, and consider this the new baseline for any future changes you'll make to the application.

For more help

Recommendations for learning more: