Aporia is a fast, easy, and secure way for data science teams to monitor their machine learning models in production. With Aporia, teams can build their own customizable monitors in minutes to receive live alerts for early detection of issues like data drift, unexpected bias, data integrity issues, and performance degradation. Aporia also has an investigation toolbox to enable further investigation and root cause analysis. Learn more about Aporia.
Aporia now offers an integration with New Relic to provide full model management of the MLOps infrastructure, with customized dashboards within New Relic that show inferences investigation.
Aporia allows you to connect alerts generated by Aporia’s monitors to New Relic’s Incident Intelligence engine and the predictions data in order to create a comprehensive monitoring dashboard in New Relic for your models.
Login into Aporia’s console: On the navbar on the left, click on Integrations and choose New Relic.
Login into your New Relic account: Log into one.newrelic.com and click on Explorer. On the upper hand corner, on the main navigation menu, click on +Add more data.
Click on Aporia: In the search bar, type Aporia or scroll down to the MLOps Integration section and click on the Aporia icon.
Select your account ID: Select the account ID you want Aporia to integrate with.
Get a New Relic license key: Once you click on the Aporia icon, follow step one. Under Prediction data, click either Select, to choose an existing key, or Create API key.
Get an Auth Token: Optionally, you can also send metric alerts from Aporia to New Relic for intelligent correlation. To do this, copy the auth token by clicking on the Copy icon under Model quality metric alerts.
Paste the key and token in Aporia: Now go to Aporia’s dashboard and under the New Relic Integration screen and paste the key under New Relic insert token. If you also copied the Auth Token, you can paste this key under New Relic Incident Intelligence Token. Once pasted, click Save.
Verify the tokens: In the Aporia dashboard, click on the Verify tokens button to verify both the key and token are working properly. Green check marks or Red error marks should appear to indicate the status.
Now that you’ve integrated New Relic and Aporia, you can monitor your data using dashboards with automated charts created by Aporia.
Go to the integration dashboard: Once you’ve verified your tokens and confirmed the integration is set up correctly, return to the New Relic integration dashboard and click on See your data. This will redirect you to an automatically generated dashboard displaying data reported to Aporia in New Relic.
Analyze Aporia's dashboard. Aporia’s dashboard populated in New Relic contains 6 charts:
- The Most active models chart displays the different models which reported predictions in the selected timeframe.
- The Most active model versions chart displays the different versions which reported predictions in the selected timeframe.
- The Model inferences graph displays the number of unique predictions reported for each model and version.
- The Average numeric inferences chart displays the average value numeric predictions reported for each model and version.
- The Numeric inferences heatmaps chart displays a histogram of the numeric predictions reported for each model and version.
- The Categorical inferences charts display the different unique values and their frequencies of categorical predictions reported for each model and version.
- Filter data: Click on the … button and click on Edit. On the right navbar, under User as filter, enable Filter the current dashboard and click Save.
- Setup alert notifications: Once you've created some dashboards, you can get alerted on your data. To create a NRQL alerts condition from a chart, click the chart widget, then click Create alert condition. Once you've named and customized your condition, you can add it to an existing policy or create a new one.
- Get notified: Once you've created an alerts condition, you can choose how you want to be notified. See our docs on how to set up notification channels.
- Correlate your incidents: In addition to notifications, you can use Incident Intelligence to correlate your incidents. See our docs on how to correlate incidents using decisions.