When people talk about artificial intelligence, they can mean different things. At New Relic, when we say AI, we mean the layer of your environment that uses a large language model (LLM) to generate a response when it receives an end user prompt. AI monitoring is an APM solution that gives you end-to-end visibility into your AI-powered app.
With AI monitoring, you can measure the performance of the engine powering your AI app, so that you can ensure your users have the best possible experience. To get started, all you need is to install one of our APM agents and enable AI monitoring.
Go to one.newrelic.com > AI monitoring > AI responses, then select a response row.
How does AI monitoring work?
To get started with AI monitoring, you'll install one of our APM agents to instrument your app. Instrumentation means that your app can be measured, which lets the agent capture data about app behaviors. Once instrumented, you need to enable AI monitoring on the configuration level.
Enabling AI monitoring allows the agent to recognize AI metadata associated with AI events. When your AI receives a prompt and returns a response, the agent can recognize data generated from external LLMs and vector stores, and parse out information about tokens usage.
Improve AI performance with AI monitoring
To overview your AI-powered app's performance: Go to one.newrelic.com > AI monitoring > AI responses.
AI monitoring can help you answer critical questions about AI app performance: are your end users waiting too long for a response? Is there a recent spike in token usage? Are there patterns of negative user feedback around certain topics? With AI monitoring, you can see data specific to the AI-layer:
- Identify errors in specific prompt and response interactions from the response table. If you're looking to make improvements to your AI models, learn how to analyze your model data in New Relic.
- If you're using different models across app environments, you can compare the cost and performance of your apps before deploying.
- Are you concerned about data compliance? Learn how to create drop filters to drop sensitive data before you send it to New Relic.
Get started with AI monitoring
Ready to get started? Make sure to confirm that you can instrument your AI library or framework. You may need to update the agent if you've already instrumented your app.
When you're ready, use our doc to manually install AI monitoring. This doc directs you to the relevant procedures for installing an APM agent, then walks you through configuring the agent for AI monitoring.