Getting started with bring your own data

This is a guide to getting started with New Relic's bring your own data. You'll learn how to install, run, and experiment with bring your own data , or BYOD, and start monitoring the performance of your machine learning models.

Quick start

Using BYOD makes it easy to monitor your machine learning models in 3 main steps:

# 1: Initialize the monitoring
ml_monitor = MLPerformanceMonitoring(...)

# 2: Add your algorithm
y = my_model.predict(X)

# 3: Record your data
ml_monitor.record_inference_data(X, y)

Use this example in colab and easily try an end-to-end example of model monitoring.

Installation

Installation is straightforward and similar to any python library

pip install git+https://github.com/newrelic-experimental/ml-performance-monitoring.git

Implementation

This guide will take you step by step for everything needed to start monitoring your Machine Learning models

1. Set Your Environment Variable

Get your (also referenced as ingest - license) and set it as environment variable: NEW_RELIC_INSERT_KEY. Click here for more details and instructions. Are you reporting data to the New Relic EU region? click here for more instructions.

2. Import package

from ml_performance_monitoring.monitor import MLPerformanceMonitoring

3. Create model monitor

metadata = {"environment": "notebook"}
model_version = "1.0"
features_columns, labels_columns = (
    ["feature_1", "feature_2", "feature_3", "feature_4"],
    ["target"],
)

ml_monitor = MLPerformanceMonitoring(
    insert_key=None,  # set the environment variable NEW_RELIC_INSERT_KEY or send your insert key here
    model_name="My stunning model",
    metadata=metadata,
    features_columns=features_columns,
    labels_columns=labels_columns,
    label_type="numeric",
    model_version=model_version
)

4. Run your model

y = my_model.predict(X)

5. Record

ml_performence_monitor_model.record_inference_data(X, y)

6. Monitor and alert

Done! To check your application go to one.newrelic.com and see real time data.

Examples

We created these notebooks in Google colab so you can try them out:

Try out an XGBoost model on California housing prices dataset. Open in colab.
Try out how to simulate 24 hours of model inference data using New Relic MLOps. Open in colab

EU Account Users

If you are using an EU account, send it as a parameter at the MLPerformanceMonitoring call if your environment variable is not set:

EVENT_CLIENT_HOST and METRIC_CLIENT_HOST
- US region account (default) -
  - EVENT_CLIENT_HOST: insights-collector.newrelic.com
  - METRIC_CLIENT_HOST: metric-api.newrelic.com
- EU region account -
  - EVENT_CLIENT_HOST: insights-collector.eu01.nr-data.net
  - METRIC_CLIENT_HOST: metric-api.eu.newrelic.com/metric/v1

It can also be sent as parameters at the MLPerformanceMonitoring call.