• Log inStart now

Bring your own data

Once your production environment reaches a certain point in terms of the number of models it has to monitor, it becomes difficult to keep track of your data and model KPIs, resulting in even more complex systems. With bring your own data (BYOD), after just a few minutes, you can quickly send your model’s inference and metric data to the New Relic platform, directly from a notebook or any other environment.

The ml-performance-monitoring python package, based on newrelic-telemetry-sdk-python, allows you to send your model’s features and prediction values, as well as custom metrics, by simply adding a few lines to your code.

Use the python package to send the following types of data to New Relic:

  1. Inference data: Stream your model’s feature and prediction values. Choose between:

    • “Online” instrumentation: Stream the values while the model is being invoked in production by easily wrapping your model through the wrap_model function. The data is streamed automatically on each invocation.

    • "Offline" instrumentation: Send the data (features and predictions) as a dataset (as an np.array, or pandas dataframe).

    Inference data is streamed as a custom event named InferenceData.

  2. Data metrics: Instead of sending all your raw inference data, select the aggregated statistics option over the features and predictions (for example, min, max, average, or percentile). They will be sent automatically as metrics.

  3. Custom metrics: Calculate your own metrics, monitor your model performance or model data, and stream them to New Relic using the record_metrics function. They will be sent as metrics.

How do I stream inference data and metrics to New Relic?

Follow these steps to stream your data and view it on the New Relic platform:

  1. Get your license key: Use the license key for the New Relic account you want your data to be associated with.

  2. Stream your data to New Relic with the new-relic-ml-performance-monitoring package. Follow the example below to see how to send data from your code.

  3. View your data in the New Relic Platform:

    • Query your data: Use the data explorer to view the data you sent, or use the following queries on the query builder:

      SELECT * FROM InferenceData WHERE model_name=[model_name] SINCE 1 day ago
      SELECT * FROM Metric WHERE model_name=[model_name] SINCE 1 day ago
    • Create your own dashboard: Build your own dashboard to visualize the metrics you sent, or view the distributions of your features and predictions. See an example dashboard below.

    • Explore entities: When you stream data to New Relic, an entity of the type machine learning model is automatically created, one per each model name. You can explore your model entities from one.newrelic.com > Model performance

Need an example for reference? Check out the following use case!

Run the following example to stream data and view an example dashboard on your New Relic account:

  1. Run the example notebook: You'll have to define your ingest key as an environment variable, or send it as a parameter. (Here's an example notebook.)

  2. View the data in the example dashboard: Import the example dashboard template JSON, and update YOUR_ACCOUNT_ID with your account ID.

Copyright © 2022 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.