When people talk about artificial intelligence, they can mean different things. At New Relic, when we say AI, we mean the layer of your environment that uses a large language model (LLM) to generate a response when it receives an end user prompt. AI monitoring is an APM solution that gives you end-to-end visibility into your AI-powered app.
With AI monitoring, you can measure the performance of the engine powering your AI app, so that you can ensure your users have the best possible experience. To get started, all you need is to install one of our APM agents and enable AI monitoring.
Go to one.newrelic.com > AI monitoring > AI responses, then select a response row.
How does AI monitoring work?
To get started with AI monitoring, you'll install one of our APM agents to instrument your app. Instrumentation means that your app can be measured, which lets the agent capture data about app behaviors. Once instrumented, you need to enable AI monitoring on the configuration level.
Enabling AI monitoring allows the agent to recognize AI metadata associated with AI events. When your AI receives a prompt and returns a response, the agent can recognize data generated from external LLMs and vector stores, and parse out information about tokens usage.
Improve AI performance with AI monitoring
To overview your AI-powered app's performance: Go to one.newrelic.com > AI monitoring > AI responses.
AI monitoring can help you answer critical questions about AI app performance: are your end users waiting too long for a response? Is there a recent spike in token usage? Are there patterns of negative user feedback around certain topics? With AI monitoring, you can see data specific to the AI-layer:
- Identify errors in specific prompt and response interactions from the response table. If an error occurs, open the trace waterfall view to scope to the methods and calls your AI-powered app makes when generating a response.
- If your prompt engineers updated prompt parameters for your AI, you can track whether token usage spiked or dropped after the update. Use AI monitoring to help you make decisions that keep costs down.
- Maybe you're fine tuning your app in development, but you want cost and performance efficiency before it goes to production. If you're using different models in different app environments, you can compare the cost and performance of your apps before deploying.
Get started with AI monitoring
Ready to get started? Make sure to confirm that you can instrument your AI library or framework. You may need to update the agent if you've already instrumented your app.
When you're ready, use our doc to manually install AI monitoring. This doc directs you to the relevant procedures for installing an APM agent, then walks you through configuring the agent for AI monitoring.