New Relic has entered the MLOps observability space, and is now allowing data scientists and machine learning engineers to monitor and observe the performance and effectiveness of their machine-learning models in a production environment.
Once your production reaches a certain point in terms of the number of models it has to monitor, it becomes difficult to keep track of your data and model KPIs, resulting in even more complex systems. With bring your own data (BYO), after just a few minutes, you can quickly send your model’s inference and metric data to the New Relic platform, directly from a notebook or any other environment.
ml-performance-monitoring python package, based on
newrelic-telemetry-sdk-python, allows you to send your model’s features and prediction values, as well as custom metrics, by simply adding a few lines to your code.
Use the python package to send the following types of data to New Relic:
- Inference data: Stream your model’s feature and prediction values. Choose between:
“Online” instrumentation: Stream the values while the model is being invoked in production by easily wrapping your model through the
wrap_modelfunction. The data is streamed automatically on each invocation.
"Offline" instrumentation: Send the data (features and predictions) as a dataset (as an
np.array, or pandas dataframe).
Inference data is streamed as a custom event named "InferenceData".
Data metrics: Instead of sending all your raw inference data, select the aggregated statistics option over the features and predictions (for example,
percentile). They will be sent automatically as metrics.
Custom metrics: Calculate your own metrics, monitor your model performance or model data, and stream them to New Relic using the
record_metricsfunction. They will be sent as metrics.
Follow these steps to stream your data and view it on the New Relic platform:
Get your license key: Use the license key for the New Relic account you want your data to be associated with.
Stream your data to New Relic by using the new-relic-ml-performance-monitoring package. Follow the example below to see how to send data from your code.
View your data in the New Relic Platform:
SELECT * FROM InferenceData WHERE model_name=[model_name] SINCE 1 day ago
SELECT * FROM Metric WHERE model_name=[model_name] SINCE 1 day ago
Create your own dashboard: Build your own dashboard to visualize the metrics you sent, or view the distributions of your features and predictions. See an example dashboard below.
Explore entities: When you stream data to New Relic, an entity of the type
machine learning modelis automatically created, one per each model name. You can explore your model entities by selecting Explorer on New Relic One, and going to the Machine Learning section on the left navigation menu.
Run the following example to stream data and view an example dashboard on your New Relic account:
Run the example notebook: You'll have to define your ingest key as an environment variable, or send it as a parameter. (Here's an [example notebook].](https://github.com/newrelic-experimental/ml-performance-monitoring/tree/main/examples))