Datarobot provides tools for deploying your machine-learning models into production. By integrating Datarobot with New Relic, you'll be able to instrument, analyze, troubleshoot, and optimize your machine-learning performance across your entire system. By rigorously observing your capabilities, you'll be able to react quickly to changes in the model's input or output and the relationship between the two.
Send your model performance metrics from Datarobot Insights to New Relic and you'll have real-time monitoring for your algorithms. You'll explore your metrics data with user-friendly charts and learn the state of your algorithms at a glance for faster and more efficient troubleshooting.
Integrate Datarobot with New Relic
First, Datarobot uses a Kafka topic to stream Insights from your machine-learning algorithm's performance metrics. Then, the New Relic connector (another algorithm) transforms the Kafka topic into a metrics data payload for a specific New Relic account.
Datarobot uses Kafka and Event Flows to send data to New Relic.
With Datarobot’s Event Flows, when a new message is set to your Kafka topic, your configured New Relic connector algorithm gets called. The connector transforms your metrics and sends them to your New Relic account.
Connect your Datarobot data to New Relic
By integrating incident intelligence with your Datarobot machine-learning models, you can monitor your machine learning model performance.
Start monitoring your Datarobot event flows with New Relic.
Get your API key: Go to one.newrelic.com > All capabilities > (account menu) > API keys and then create a user key for your account with a meaningful name. Make note of this name for later. For more on API keys, see our docs.
Create a dashboard: Go to one.newrelic.com > All capabilities > Dashboards, then click the Import dashboards button. Copy and paste the JSON code into the Paste your JSON field code.
Update the YOUR_ACCOUNT_ID values with your account ID.
{ "name": "Datarobot Dashboard for Default Metrics", "description": null, "permissions": "PUBLIC_READ_WRITE", "pages": [ { "name": "Datarobot Dashboard for Default Metrics", "description": null, "widgets": [ { "visualization": { "id": "viz.line" }, "layout": { "column": 1, "row": 1, "height": 3, "width": 4 }, "title": "Runtime Duration by Algorithm", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": YOUR_ACCOUNT_ID, "query": "SELECT average(algorithmia.duration_milliseconds) FROM Metric TIMESERIES FACET `algorithm_name` LIMIT 10 SINCE 1800 seconds ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 1, "height": 3, "width": 4 }, "title": "Throughput by Algorithm", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": YOUR_ACCOUNT_ID, "query": "SELECT count(algorithmia.duration_milliseconds) FROM Metric TIMESERIES FACET `algorithm_name` LIMIT 10 SINCE 1800 seconds ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null } ] } ]}Configure Datarobot Insights for New Relic: Use Datarobot's docs for how to configure Datarobot for New Relic.
Create the New Relic connector algorithm: Use Python 3.8 to create a connector algorithm. If you're new to writing code to generate algorithms, see Datarobot's getting started guide.
import Datarobotimport jsonfrom datetime import datetimefrom newrelic_telemetry_sdk import GaugeMetric, MetricClient
client = Datarobot.client()metric_client = MetricClient(os.environ["newrelic_api_key"])
def convert_str_timestamp_to_epoch(str_time): obj_time = datetime.strptime(str_time, "%Y-%m-%dT%H:%M:%S.%f") return int(obj_time.timestamp() * 1000)
def get_operational_metrics(payload): ALGORITHM_TAGS = { "algorithm_version", "request_id", "time", "algorithm_name", "session_id", "algorithm_owner", } inference_metrics = {key: payload[key] for key in payload.keys() ^ ALGORITHM_TAGS} return inference_metrics
def send_to_newrelic(inference_metrics, insights_payload): newrelic_metrics = [] for key, value in inference_metrics.items(): name = "algorithmia." + key epoch_time = convert_str_timestamp_to_epoch(insights_payload["time"]) tags = { "algorithm_name": insights_payload["algorithm_name"], "algorithm_version": insights_payload["algorithm_version"], "algorithm_owner": insights_payload["algorithm_owner"], "request_id": insights_payload["request_id"], "session_id": insights_payload["session_id"], }
newrelic_metrics.append( GaugeMetric(name=name, value=value, tags=tags, end_time_ms=epoch_time))
response = metric_client.send_batch(newrelic_metrics)response.raise_for_status()
def apply(input): insights_payload = input inference_metrics = get_operational_metrics(insights_payload) send_to_newrelic(inference_metrics, insights_payload) return NoneInclude these dependencies:
algorithmia>=1.0.0,<2.0newrelic_telemetry_sdk==0.4.2Once your algorithm build finishes, you can test it with this sample payload to make sure it runs successfully. Your output should look something like this.
{ "risk_score": 0.2, "duration_milliseconds": 8, "algorithm_version": "1.0.6", "session_id": "rses-f28bb94a-5556-4aeb-a6d2-89493626bf4f", "time": "2021-02-20T00:21:54.867231", "algorithm_name": "credit_card_approval", "request_id": "req-9f5345b4-a1cd-431c-a43a-bd2a06f4a6f4", "algorithm_owner": "asli"}Configure with your API key: Add your New Relic API key to the Datarobot secret store.
Set up Datarobot Event Flows with New Relic: See Datarobot's documentation on setting up your connector algorithm to send event-based machine learning flows to New Relic.
Did this doc help with your installation?
Monitor your machine learning models
Follow these steps to get the most of observing your machine-learning data in New Relic.
Get your API key: Go to one.newrelic.com > All capabilities > (user menu) > API keys. Create a user key for your account with a meaningful name. Make note of this name for later. For more on API keys, see our docs.
Create a dashboard: Go to one.newrelic.com > All capabilities > Dashboards, then click the Import dashboards button. Copy and paste the JSON code into the Paste your JSON field code.
Update the YOUR_ACCOUNT_ID values with your account ID.
{ "name": "Datarobot Dashboard for Default Metrics", "description": null, "permissions": "PUBLIC_READ_WRITE", "pages": [ { "name": "Datarobot Dashboard for Default Metrics", "description": null, "widgets": [ { "visualization": { "id": "viz.line" }, "layout": { "column": 1, "row": 1, "height": 3, "width": 4 }, "title": "Runtime Duration by Algorithm", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": YOUR_ACCOUNT_ID, "query": "SELECT average(algorithmia.duration_milliseconds) FROM Metric TIMESERIES FACET `algorithm_name` LIMIT 10 SINCE 1800 seconds ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 1, "height": 3, "width": 4 }, "title": "Throughput by Algorithm", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": YOUR_ACCOUNT_ID, "query": "SELECT count(algorithmia.duration_milliseconds) FROM Metric TIMESERIES FACET `algorithm_name` LIMIT 10 SINCE 1800 seconds ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null } ] } ]}Set up alerts notifications: Once you've created some dashboards, you can get alerted on your data. To create a NRQL condition from a chart, click the chart menu , then click Create alert condition. Once you've named and customized your condition, you can add it to an existing policy or create a new one.
Get notified: Once you've created an alerts condition, you can choose how you want to be notified. See our docs on how to set up notification channels.
Correlate your incidents: In addition to notifications, you can use incident intelligence to correlate your incidents. See our docs on how to correlate incidents using decisions.