This guide walks you through establishing KPIs and using New Relic to improve and optimize your coding practices. It's part of our series on observability maturity.
High quality code is key to an organization's ability to innovate and grow. Poor quality code diverts engineering time that could be used for innovation, instead wasting it on troubleshooting and fixing. Poor quality code also has a strong negative impact on customer satisfaction and revenue.
Overall, a business's digital operations are only as stable as its code. Without a stable code base, engineering will never have time to meet even basic demand for new features, let alone move at the pace required to reach the leading edge.
This Development quality (DevQual) guide identifies specific key performance indicators and processes that drive stability through code quality improvement. DevQual is the first guide in the Innovation and growth value stream. DevQual is followed by the Release quality guide. Both use cases must be performed in order.
You're a good candidate for using this guide if:
- You're not measuring code quality.
- Your code quality is perceived as poor.
- You don't understand where your developers are spending their time.
- Your organization suffers from too many outages due to application defects.
DevQual leverages your existing development toolchain and Agile practices to reach the overall goal of reducing code defects, thus improving stability and customer satisfaction. Improved stability, in turn, frees up valuable developer time for innovation and growth.
Organizations that successfully implement this use case can then move on to improving business impact through the release quality use case, and then increase the velocity of innovation with the release cadence use case.
You'll use the development quality process to collect and measure the following KPIs:
- Build success
- Unit test success
- Code coverage
- Defect volume
- Code commit rate
- Pull request rate
- Merge rate
The stability KPIs will measure the quality of the submitted code. The velocity KPIs will measure the speed with which new code is written and committed.
These KPIs will help you to identify the sources of code defects and the areas that require the most developer effort so that you can focus your attention on reducing defects and more efficiently use your engineering talent. They'll also help you to understand if development velocity has any impact on code quality.
Detailed information on each metric follows.
Before you begin, if you don't have equivalent experience, you should complete the New Relic University (NRU) Overview Course. You should also have a basic understanding of:
As with any continuous improvement process, the first step of improving development quality (DevQual) is to establish the current state of your KPIs. To do so, perform the following tasks:
- Gather the required KPIs
- Implement the DevQual dashboards
- Establish a baseline
- Perform initial enablement
- Start continuous improvement process
We'll explain these steps in more detail.
As with any quality initative, we need to start by gathering our key performance indicators. In order to do this, you need to identify the specific technology platforms that support your development processes, such as source code repositories and build/test automation platforms. Once those platforms are identified, you'll need to identify methods for extracting each KPI's attributes and importing them into New Relic.
The KPIs and minimum-required attributes needed for this use case are listed in the key performance indicators section. Typically, you'll use your development toolchain's APIs to extract the KPIs and their attributes and then submit them to the New Relic platform using the custom events API.
Prior to starting any custom integration work, you should determine if any out-of-box integrations exist.
The DevQual dashboards are the primary technological drivers of this development quality improvement process. They'll show your current KPIs and help you to identify the areas which require improvement.
Sample DevQual dashboards can be found in the New Relic OMA resource center on GitHub.
The information you present in the dashboards is dependent on your development toolchain, so your dashboard will need to be modified from the sample.
The DevQual process requires enough data to form a baseline before you can perform the initial enablement. Your baseline should consist of a representative sample of development activity. Normally, this will be a minimum of two weeks, but it can be up to six weeks, depending on your current development pace. One easy way to do this is to align your baseline collection and evaluation cycle with your Agile sprints.
You should periodically ensure that DevQual event data is accumulating as expected.
You'll introduce development teams and other stakeholders to the baseline DevQual data and the ongoing continuous improvement process you'll be following.
The process consists of three activities:
- Review the DevQual KPIs and trends. You and the stakeholders will look at the DevQual KPIs and identify trends.
- Identfy achievements, challenges, and opportunities. In this phase, you'll identify areas where KPIs are improving (achievements) and areas where they're not improving (challenges). You'll then identify strategies and tactics for improving KPIs (opportunities) and determine how to implement them.
- Make technical recommendations. Here, you and the relevant stakeholders will identify and review technical recommendations, such as making changes to your development toolchain or observability strategies.
You can use the session template presentation to keep this part of the DevQual process organized.
This is the ongoing phase of the continuous improvement process. During this phase you will repeat the initial enablement steps to review your progress against your baseline and adjust your strategies and tactics so you deliver the required positive business outcome.
Each cycle of the improvment process should occur after several iterations of your development process. Typically, this is approximately every 2-3 weeks, at the mid-point and end of every Agile sprint.
During this phase you should:
- Report your KPIs each week to upper management to ensure that the stakeholder teams are appropriately prioritizing the work, and to show that progress towards the promised business outcomes are being reached.
- Record and retain your weekly KPIs over periods of months to years to establish a baseline and to show the rate of improvement.
You should keep in mind that this is a continuous improvment process. You will continue to collect and evaluate the KPIs over long periods of time to ensure you are meeting your DevQual goals.
As you establish the DevQual process, you will see increased application stability and a gradual reduction in the effort required to fix defects. In time, this will result in a reduction in your feature backlog.
After you are on the path to those goals, you should start to use the Release quality guide.